aboutsummaryrefslogtreecommitdiff
path: root/paper.tex
AgeCommit message (Collapse)AuthorLines
2020-12-01IMPORTANT: organizational improvements in Maneage TeX sourcesMohammad Akhlaghi-170/+96
This only concerns the TeX sources in the default branch. In case you don't use them, there should only be a clean conflict in 'paper.tex' (that is obvious and easy to fix). Conflicts may only happen in some of the 'tex/src/preamble-*.tex' files if you have actually changed them for your project. But generally any conflict that does arise by this commit with your project branch should be very clear and easy to fix and test. In short, from now on things will even be easier: any LaTeX configuration that you want to do for your project can be done in 'tex/src/preamble-project.tex', so you don't have to worry about any other LaTeX preamble file. They are either templates (like the ones for PGFPlots and BibLaTeX) or low-level things directly related to Maneage. Until now, this distinction wasn't too clear. Here is a summary of the improvements: - Two new options to './project make': with '--highlight-new' and '--highlight-notes' it is now possible to activate highlighting on the command-line. Until now, there was a LaTeX macro for this at the start of 'paper.tex' (\highlightchanges). But changing that line would change the Git commit hash, making it hard for the readers to trust that this is the same PDF. With these two new run-time options, the printed commit hash will not changed. - paper.tex: the sentences are formatted as one sentence per line (and one line per sentence). This helps in version controlling narrative and following the changes per sentence. A description of this format (and its advantages) is also included in the default text. - The internal Maneage preambles have been modified: - 'tex/src/preamble-header.tex' and 'tex/src/preamble-style.tex' have been merged into one preamble file called 'tex/src/preamble-maneage-default-style.tex'. This helps a lot in simply removing it when you use a journal style file for example. - Things like the options to highlight parts of the text are now put in a special 'tex/src/preamble-maneage.tex'. This helps highlight that these are Maneage-specific features that are independent of the style used in the paper. - There is a new 'tex/src/preamble-project.tex' that is the place you can add your project-specific customizations.
2020-08-27Machine architecture and byte-order available as LaTeX macroMohammadreza Khellat-13/+14
Until now, no machine-related specifications were being documented in the workflow. This information can become helpful when observing differences in the outcome of both software and analysis segments of the workflow by others (some software may behave differently based on host machine). With this commit, the host machine's 'hardware class' and 'byte-order' are collected and now available as LaTeX macros for the authors to use in the paper. Currently it is placed in the acknowledgments, right after mentioning the Maneage commit. Furthermore, the project and configuration scripts are now capable of dealing with input directory names that have SPACE (and other special characters) by putting them inside double-quotes. However, having spaces and metacharacters in the address of the build directory could cause build/install failure for some software source files which are beyond the control of Maneage. So we now check the user's given build directory string, and if the string has any '@', '#', '$', '%', '^', '&', '*', '(', ')', '+', ';', and ' ' (SPACE), it will ask the user to provide a different directory.
2020-07-04Commit hash of Maneage branch used to build project as LaTeX macroMohammad Akhlaghi-12/+13
To help in the documentation, the Git hash of the Maneage branch commit that the project has most recently merged with (or branched from) is now also provided as a LaTeX macro ('\maneageversion'). It is calculated in 'reproduce/analysis/make/initialize.mk' (in the recipe to 'initialize.tex').
2020-07-04Citing Maneage paper in acknowledgmentsMohammad Akhlaghi-9/+9
In the previous commit, the modified abstract of the acknowledgments only included the URL of Maneage, but its more formal to cite the Maneage paper, the URL is already present in the paper.
2020-07-04Corrected Acknowledgments section in default paperMohammad Akhlaghi-14/+14
Until now, the acknowledgment section didn't contain the new name of Maneage and it also included an acknowledgment of Gnuastro (which is not appropriate for a general project which may not use Gnuastro). With this commit this is fixed.
2020-06-14Better comments for the top macros of paper.texMohammad Akhlaghi-26/+50
The default 'paper.tex' starts by defining some macros and comments describing them. Until now, the text was not too clear and could be confusing for someone that is not at all familiar with Maneage. With this commit, the comments have been edited to be more clear for a first-time reader. For example they all start with FULL CAPS summaries. Two other small things were corrected in 'tex/src/preamble-necessary.tex': - Until now 'project.tex' was included in this preamble. However, because of its importance in Maneage, and prominent place in the demonstration plot of the paper introducing Maneage, it is now included directly in 'paper.tex'. This also allows users to safely ignore/delete this preamble file if their LaTeX style is different. - I noticed that some macros for some astronomical software names from the very first commits in Maneage were still present here! They are no longer used, so they have been removed.
2020-06-10Updated text of default paper.tex, putting more recent examplesMohammad Akhlaghi-78/+87
The text of the default paper hadn't been changed for a very long time! In this time, three papers using Maneage have been published (which can be very good as an example), Maneage also now has a webpage! With these commit these examples and the webpage have been added and generally it was also polished a little to hopefully be more useful.
2020-06-06IMPORTANT: Added publication checklist, improved relevant infrastructureMohammad Akhlaghi-19/+26
Possible semantic conflicts (that may not show up as Git conflicts but may cause a crash in your project after the merge): 1) The project title (and other basic metadata) should be set in 'reproduce/analysis/conf/metadata.conf'. Please include this file in your merge (if it is ignored because of '.gitattributes'!). 2) Consider importing the changes in 'initialize.mk' and 'verify.mk' (if you have added all analysis Makefiles to the '.gitattributes' file (thus not merging any change in them with your branch). For example with this command: git diff master...maneage -- reproduce/analysis/make/initialize.mk 3) The old 'verify-txt-no-comments-leading-space' function has been replaced by 'verify-txt-no-comments-no-space'. The new function will also remove all white-space characters between the columns (not just white space characters at the start of the line). Thus the resulting check won't involve spacing between columns. A common set of steps are always necessary to prepare a project for publication. Until now, we would simply look at previous submissions and try to follow them, but that was prone to errors and could cause confusion. The internal infrastructure also didn't have some useful features to make good publication possible. Now that the submission of a paper fully devoted to the founding criteria of Maneage is complete (arXiv:2006.03018), it was time to formalize the necessary steps for easier submission of a project using Maneage and implement some low-level features that can make things easier. With this commit a first draft of the publication checklist has been added to 'README-hacking.md', it was tested in the submission of arXiv:2006.03018 and zenodo.3872248. To help guide users on implementing the good practices for output datasets, the outputs of the default project shown in the paper now use the new features). After reading the checklist, please inspect these. Some other relevant changes in this commit: - The publication involves a copy of the necessary software tarballs. Hence a new target ('dist-software') was also added to package all the project's software tarballs in one tarball for easy distribution. - A new 'dist-lzip' target has been defined for those who want to distribute an Lzip-compressed tarball. - The '\includetikz' LaTeX macro now has a second argument to allow configuring the '\includegraphics' call when the plot should not be built, but just imported.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-7/+10
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-04-25IMPORTANT: Primary Maneage repositories are now under maneage.orgMohammad Akhlaghi-15/+11
Until now, the primary Maneage URLs were under GitLab, but since we now have a dedicated URL and Git repository, its better to transfer to this as soon as possible. Therefore with this commit, throughout Maneage, any place that Maneage was referenced through GitLab has been corrected. Please correct your project's remote to point to the new repository at `git.maneage.org/project.git', and please make sure it follows the `maneage' branch. There is no more `master' branch on Maneage.
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-1/+1
Now that its 2020, its necessary to include this year in the copyright statements.
2019-11-04Acknowledged support from the RDA EU 4.0 grantMohammad Akhlaghi-1/+4
Since we got the RDA Adoption grant, it was necessary to add it in the acknowledgements.
2019-08-22Paper's title and author information moved to main paper.texMohammad Akhlaghi-0/+16
Until now, the paper's title and author information were set it `tex/src/preamble-header.tex'. But they are actually shown in the final PDF paper and a much better place to keep them is the top-level `paper.tex'. With this commit, the setting of the title and author names has been moved to `paper.tex', just after importing all the preambles. However, the basic package importation and low-level settings are still set in `tex/src/preamble-header.tex', because they are relatively low-level. This task was suggested by Deepak (Indian Institute of Astrophysics).
2019-08-18Updated README-hacking.md's checklist for better usabilityMohammad Akhlaghi-4/+10
After the checklist was applied in the 5th Indo-French Astronomy School, we found some cases in the checklist that were extra (and thus had to be removed), or were needed (and thus were added). Also the non-necessary steps for a first commit were moved to a separate/new section in the checklist for the people to add after doing their first commit. Also, the software part of the paper was moved to an appendix.
2019-05-21Source directory links to build directory all managed in configureMohammad Akhlaghi-2/+2
Until now, the `tex/build' symbolic link was put in the clone/source tree when the build-directory's `tex' directory was being built. Thanks to Roberto Baena, we just found a bug because of this behavior: when a second group member is trying to build the pipeline, since the build directory's `tex' directory already exists, no `tex/build' will be put in their clone/source directory. As a result, the PDF building will crash. To fix this (and keep things organized), the two `tex/build' and `tex/tikz' links (to the build directory) are now built in the configure step while it is building all the top-level directories. They are no longer built within the Makefiles. Also, a comment was added on top of every directory built during the configuration phase to be clear. This fixes bug #56362.
2019-04-15New architecture to separate software-building and analysis stepsMohammad Akhlaghi-4/+4
Until now, the software building and analysis steps of the pipeline were intertwined. However, these steps (of how to build a software, and how to use it) are logically completely independent. Therefore with this commit, the pipeline now has a new architecture (particularly in the `reproduce' directory) to emphasize this distinction: The `reproduce' directory now has the two `software' and `analysis' subdirectories and the respective parts of the previous architecture have been broken up between these two based on their function. There is also no more `src' directory. The `config' directory for software and analysis is now mixed with the language-specific directories. Also, some of the software versions were also updated after some checks with their webpages. This new architecture will allow much more focused work on each part of the pipeline (to install the software and to run them for an analysis).
2019-04-14Replaced all occurances of pipeline in textMohammad Akhlaghi-50/+50
All occurances of "pipeline" have been chanaged to "project" or "template" withint the text (comments, READMEs, and comments) of the template. The main template branch is now also named `template'. This was all because `pipeline' is too generic and couldn't be distinguished from the base, and customized project.
2019-04-13Corrected copyright notices and info about adding copyright infoMohammad Akhlaghi-1/+1
Until now, the files where the people were meant to change didn't have a proper copyright notice (for example `Copyright (C) YOUR NAME.'). This was wrong because the license does not convey copyright ownership. So the name of the file's original author must always be included and when people modify it (and add their own copyright-able modifications). With this commit, the file's original author (and email) are added to the copyright notice and when more than one person modified a file, both names have their individual copyright notice. Based on this, the description for adding a copyright notice in `README-hacking.md' has also been modified.
2019-04-12Gnuastro's citation included in its build targetMohammad Akhlaghi-2/+2
With this commit, we are applying the new style of citing software within the build rule of Gnuastro.
2019-04-05Software acknowledgement section is automatically generatedMohammad Akhlaghi-119/+6
Until now, management of the software names and versions in the paper was done manually (a macro had to be defined in `initialize.mk', then used in `paper.tex', so they had to be manually set in two places). Managing this was not easy. To fix this, with this commit, each software building rule's target is a text file that contains its human-readable name and its version. In the end, the configure script sorts them by their name and writes them into a LaTeX input file that we can easily import as a file into the main paper.
2019-04-04Dependency version LaTeX macros written at the end of configureMohammad Akhlaghi-43/+113
Until now, these versions were written in each run. This was mainly inherited from the old days of the pipeline, where we didn't know the software on the host. But now that we have almost everything under control, we can just write these LaTeX macros at the end of the configure script and make `initialize.mk' simpler and also (very slightly!) speed-up/simplify the processing.
2019-04-04Numpy and Scipy build on Mac imported into the main branchMohammad Akhlaghi-28/+41
We were developing the build of Numpy and Scipy on Mac in a parallel thread and things seems to be working relatively nice now. There were only two problems: 1) GCC still has some random building issues on Mac. 2) ATLAS shared libraries can't be built on Mac (so we used OpenBLAS to build Numpy and Scipy on both Mac and GNU/Linux). But for now, none of these problems are critical. So, we can progress in one branch. There were only very minor conflicts in the merge.
2019-04-02Copyright notice added to remaining filesMohammad Akhlaghi-3/+3
After doing a systematic search for files without a copyright notice, a few more were found that didn't have a notice. So a notice was added for them. I used this Bash command to find the files: for f in $(find ./ -type f); do \ if [[ $f != *.git* ]]; then \ n=$(grep -i copyright $f | wc -l); \ echo "$n $f"; \ fi; \ done | awk '$1==0'
2019-03-29Added Copyright to all TeX and README filesMohammad Akhlaghi-0/+18
In order to be more clear, a copyright statement was added to all the LaTeX and README files.
2019-03-22Acknowledged new software in paperRaul Infante-Sainz-12/+14
In the previous commit I already have included the latex macros. In this commit we use them in the paper source.
2019-03-22Imported recent work in the main pipeline, conflicts fixedRaul Infante-Sainz-1/+1
Conflicts in `gcc' build comments and in mentioning software used in paper fixed.
2019-03-22H5py added and setuptools as main Python module dependencyRaul Infante-Sainz-15/+25
In this commit we add `h5py' Python package. We also include `setuptools' as a main dependency of Python because with the previous commit it (as well as `pip') is no longer installed with Python. Numpy version also has been incremented.
2019-03-19Minor corrections: typo and adding file to .gitignoreMohammad Akhlaghi-16/+15
The LaTeX macro for libgit2 was not properly used in `paper.tex'. On Mac systems, after browsing the directory, a `.DS_Store' file was created. So to keep things clean on those systems, it is added to the files to be ignored by Git.
2019-02-13Astropy installed in the pipelineRaul Infante-Sainz-1/+3
Astropy was added and one very important thing is that we have to use the pypi tarball (https://pypi.org/) (which is bootstrapped) and not the github tarball.
2019-02-06Better management for .tex directories to build from tarballMohammad Akhlaghi-10/+10
In order to collaborate effectively in the project, even project members that don't necessarily want (or have the capacity) to do the whole analysis must be able to contribute to the project. Until now, the users of the distributed tarball could only modify the text and not the figures (built with PGFPlots) of the paper. With this commit, the management of TeX source files in the pipeline was slightly modified to allow this as cleanly as I could think of now! In short, the hand-written TeX files are now kept in `tex/src' and for the pipeline's generated TeX files (in particular the old `tex/pipeline.tex'), we now have a `tex/pipeline' symbolic-link/directory that points to the `tex' directory under the build directory. When packaging the project, `tex/pipeline' will be a full directory with a copy of all the necessary files. Therefore as far as LaTeX is concerned, having a build-directory is no longer relevant. Many other small changes were made to do this job cleanly which will just make this commit message too long! Also, the old `tarball' and `zip' targets are now `dist' and `dist-zip' (as in the standard GNU Build system).
2019-02-05Ability to package project into tarball or zip fileMohammad Akhlaghi-5/+6
With this commit, it is now possible to package the project into a tarball or zip file, ready to be distributed to collaborators who only want to modify the final paper (and not do the analysis technicalities), or for uploading to sites like arXiv, or online LaTeX sharing pages.
2019-01-22Updated to newly modified version of metastoreMohammad Akhlaghi-1/+1
In this version, too many extra notices (just regarding a change from branch to branch) are not printed with `-q'. Instead only a one line statement is printed that it is saved or applied.
2019-01-21Libbsd added as a dependency of MetastoreMohammad Akhlaghi-17/+18
Metastore depends on `bsd/string.h' to work properly (atleast on GNU/Linux systems). The first system I tried building with had that library, so I didn't notice! With this commit, we also build `libbsd' as part of the pipeline. Also, I couldn't find libbsd's version in any of its installed headers, so like Libjpeg, we can't actually check and will directly write our internal version into the paper.
2019-01-21Metastore package now installed to allow keeping file meta-dataMohammad Akhlaghi-16/+16
The pipeline heavily depends on file meta data (and in particular the modification dates), for example the configuration-Makefiles within the pipeline are set as prerequisites to the rules of the pipeline. However, when Git checks out a branch, it doesn't preserve the meta-data of the files unique to that branch (for example program source files or configuration-Makefiles). As a result, the rules that depend on them will be re-done. This is especially troublesome in the scenario of this reproducible paper project because we commonly need to switch between branches (for example to import recent work in the pipeline into the projects). After some searching, I think the Metastore program is the best solution. Metastore is now built as part of the pipeline and through two Git hooks, it is called by Git to store the original meta-data of files into a binary file that is version controlled (and managed by Metastore).
2019-01-17Added Gnuastro in paragraph with all programsMohammad Akhlaghi-14/+15
Until now, Gnuastro was only mentioned in the first acknowledgments section, but not in the paragraph with all the program names. But these two are not mutually exclusive. All the software should be mentioned in the last paragraph and those that need special mention can be mentioned before it.
2019-01-14GNU NCURSES and GNU Readline also built before GNU BashMohammad Akhlaghi-19/+20
Readline is a prerequisite of Bash and AWK, while NCURSES is a prerequisite of Readline. With the recent update of GNU Bash (and thus GNU Readline) on my host operating system, the pipeline crashed and I noticed this hole in the pipeline. In particular, AWK (which linked with Readline 7.0) would complain about not finding it and abort.
2018-12-31Updated Gzip and Gnuastro versions to standard buildsMohammad Akhlaghi-20/+20
Both Gzip and Gnuastro were being bootstrapped personally from their Git repository until now. But fortunately a new release of both came out last week and so to make things standard we are now using their standard tarballs. I also noticed that we weren't checking the version of Gzip or mentioning it in the acknowledgement section. This was also corrected.
2018-12-02Wget and OpenSSL now installed as a basic dependencyMohammad Akhlaghi-18/+19
The TeX Live installer needs Wget to operate smoothly, especially on recent Mac OS systems that don't have Wget pre-installed. Also, it would be good for the pipeline to have its own downloader. So with this commit, the pipeline also installs Wget and OpenSSL which is a dependency. Many other small changes/fixes were done in this process.
2018-11-28Better control of shared library linkingMohammad Akhlaghi-16/+16
Until now we weren't explicity writing the full path of the dynamic libraries necessary for linking a program. But now with `-Wl,-rpath=$(ildir)' we ensure that the linker keeps the address of the dynamic libraries necessary for linking at linking time, not running time. Also, `pkg-config' is also built when preparing the basics. Several other minor corrections were made thanks to the great help of Raúl Infante Sainz.
2018-11-26High-level dependencies build without system's PATHMohammad Akhlaghi-20/+22
The high-level dependencies are now built without having access to the system's PATH. To do this, all the necessary software that we aren't building ourselves are now brought into the installed `bin/' directory using a symbolic link to the corresponding software on the host. To do this, it was also necessary to increase the number of basic/low-level packages that we are building, and add several more (Diffutils and Findutils). With this process in place, we now have a list of the exact software packages that we are not building our selves, enabling easy building of all such dependencies in the future.
2018-11-25More generic naming conventionMohammad Akhlaghi-13/+11
Until now, we were keeping the input file within the reproduction pipeline's directories using the same name as the database/server. Now, we are using a short/summarized filename convention for the input dataset.
2018-11-25Pipeline now downloads and uses an input datasetMohammad Akhlaghi-29/+57
In most analysis situations (except for simulations), an input dataset is necessary, but that part of the pipeline was just left out and a general `SURVEY' variable was set and never used. So with this commit, we actually use a sample FITS file from the FITS standard webpage, show it (as well as its histogram) and do some basic calculations on it. This preparation of the input datasets is done in a generic way to enable easy addition of more datasets if necessary.
2018-11-21Modified reproducible paper statement in paperMohammad Akhlaghi-4/+2
Gnuastro is just one of the softwares used in the pipeline, so it is too much to include its version just under the abstract. However, the version of the pipeline is very important as well as the link to it. This change puts more emphasis on these two more important points.
2018-11-19Removed GNU Binutils, CMake's built with its own bootstrapMohammad Akhlaghi-5/+5
When the C compiler is not GNU GCC, linking with GNU Binutils is going to cause problems. So until the time that we can include GCC into this pipeline, its best to avoid Binutils also. Also, for building CMake, we were relying on an installed CMake, but now, we are using its own `./bootstrap' script, so it can be built even if the host system doesn't have CMake. Also, for TeX Live, we are now setting a custom file as main target to avoid complications with symbolic links as targets in Make. Finally, when the user says they don't want to re-write an existing configuration file, no extra notices will be printed and the configure script will immediately start building programs.
2018-11-18Pipeline also installs TeX live and necessary packagesMohammad Akhlaghi-12/+22
Since the final product of the pipeline is a LaTeX-created PDF file, it was necessary to also have LaTeX within the pipeline. With this commit, TeX Live is also built as part of the configuration and all the necessary packages to build the PDF are also installed and mentioned in the paper along with their versions.
2018-11-15Minor corrections in paper's textMohammad Akhlaghi-6/+7
Some minor corrections have been made in the paper's text to make things easier to read and be more formal.
2018-11-15Binutils and other compressors also included in pipelineMohammad Akhlaghi-11/+12
To have better control over the build, GNU Binutils, Bzip2, GNU Gzip, and XZ Utils have also been added to the pipeline. Some other minor cleanups and fixes were also implemented throughout the process.
2018-11-14Lzip and Tar also built as basic dependenciesMohammad Akhlaghi-4/+5
To ensure the easy unpacking and building of the programs, Lzip and Tar are now also build during the initial setup phase. Some minor corrections were also applied to make things cleaner and smoother.
2018-11-14All used software with versions, acknowledgedMohammad Akhlaghi-0/+30
All the used software are now acknowledged in the template paper along with their versions. This section is also mentioned in the check list, so users don't delete it by mistake.
2018-08-11Not using random distribution in demonstration plotMohammad Akhlaghi-7/+9
Different implementations of AWK may use different random number generators, so even setting the seed will not ensure a reproducible result. Because of this, the random plot may be different when the pipeline runs on different systems and this can confuse early users (its contrary to the exact reproducibility that is the whole purpose of this pipeline). The plot is just a simple X^2 plot, showing the squared value of the X axis on the Y axis. It is very simple, but atleast it will be identical on all systems. Also, there may be too many complicated things in the pipeline already for an early user, and its just a demonstration, so the easier/simpler, the better.