aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorLines
2022-05-10initialize.mk: Git call in variable works with LD_LIBRARY_PATHMohammad Akhlaghi-37/+43
Until now, the '$(project-commit-hash)' Make variable of 'initialize.mk' simply called 'git' to find the commit hash. However, due to one of the recent software updates, we noticed that this command is no longer working (and the project commit hash wasn't getting printed in the PDF)! The problem was that Maneage's Git, couldn't find the 'libiconv' library that it was built with. With this commit, the '$(shell' command that calls Git, first exports 'LD_LIBRARY_PATH' to Maneage's software build directory. As a result, the Git command can work and will report the commit as a LaTeX macro to be used in the paper. To avoid relying on PATH outside of Make recipes, we now also directly call the Git executable with Maneage. Some other minor issues have been found and fixed in this commit: - README-hacking.md: some minor edits and typo corrections. - initialize.mk: the '$(curdir)' variable is now used in several places that we were calling 'pwd'. - versions.conf: 'xlsxio-version' now included with other programs. Until now it was commented because GCC 11.1.0 had issues with it. However, GCC 11.2.0 doesn't have a problem any more, so it has been returned to the list of all high-level programs. - xorg.mk: used same format to comment recipe lines as the other Makefiles (a '#' followed by a TAB). - preamble-pgfplots.tex: lines to comment for building an EPS figure with PGFPlots have been re-formatted to be more human-readable.
2022-05-08./project: make clean removes extra tex files in top source directoryPedram Ashofteh Ardakani-0/+6
Until now, the './project make clean' command would only clean (remove) the PDF file from the top source directory. However, if a user would run LaTeX outside of Maneage, many extra latex output such as *.aux, *.log, *.synctex and etc would be produced in the top source directory. These files can interfere with './project make'. With this commit, when './project make clean' is run, any possibly existing LaTeX temporary files will also be deleted from the top source directory. This problem was first reported by Matin Torkian.
2022-04-20Updated Git, Coreutils and Emacs, new script to prepare tarballsPedram Ashofteh Ardakani-8/+192
Until now, one had to follow the instructions from [1] to prepare a standard software tarball before merging with the low-level tarballs-software repository [2]. The script only worked for '.tar.gz' suffix and was only available as a comment on Savannah (in [1]). With this commit, the script has been imported into Maneage as 'reproduce/software/shell/tarball-prepare.sh' to simplify future software updates. It work with all supported '.tar.*' suffixes (of the upstream tarball repository) and will convert the tarballs to Maneage's standard format. Also, this script has a minimal argument parser and can skip the tarballs that are already unpacked, allowing faster tests. This script was used to update the versions of: Coreutiles 9.0 --> 9.1 Git 2.34 --> 2.36 Emacs 27.2 --> 28.1 The main motive behind this update was Git which announced a vulnerability issue [3] and suggested an update to the latest version as soon as possible. More detail is described in this github blog [4], but in summary, it was a security issue on multi-user systems that has been found and fixed by Git developers. Since Maneage is often installed on such shared systems, it was important to make this update. GNU Coreutils and GNU Emacs were also updated because they are also commonly used. The following improvements have also done with this commit: - .gitignore: ignore emacs auto-save files (that end with a '#') - README-hacking.md: In the checklist for updating the Maneage branch, the no-longer-necessary '--decorate' option of Git was removed from the command to check the general branch history. [1] https://savannah.nongnu.org/task/?15699 [2] https://git.maneage.org/tarballs-software.git/ [3] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/ [4] https://github.blog/2022-04-12-git-security-vulnerability-announced/
2022-04-15./project: new --refresh-bib to force-build bibliographyPedram Ashofteh Ardakani-0/+2
Until now, the bibliography was only re-built when 'tex/src/references.tex' was modified. This is useful in many regular cases because building the bibliography can slow down the build and it is in-efficient to built it in every edit of the text of the paper. However, it can be inconvenient when a change in the paper's bibliography is necessary, without actually editing 'references.tex' (for example when you are removing a citation from the text). This happens because Make is only sensitive to file modification time. In this case, Make does not see the need to create a new 'bib' file because the 'tex/src/reference' is not changed, and only the 'paper.tex' is changed. Make is totally 'blind' to the new 'citation' defined in 'paper.tex'. As a workaround, until now users were forced to manually change the 'tex/src/references.tex' file modification date: either by altering the content, or using the 'touch' command. With this commit, the '--refresh-bib' is added to './project' arguments to address this issue. It will just 'touch' the 'tex/src/references.tex' file before calling Make. In effect, this will 'force' Make to create the bibliography file, even if 'tex/src/references.tex' hasn't been updated.
2022-04-15IMPORTANT: more generic, robust and secure INPUTS.conf and download.mkMohammad Akhlaghi-112/+126
SUMMARY: it is necessary to update your 'INPUTS.conf' and 'download.mk'. Until now, adding an input file involved several steps that needed manual (and inconvenient!) intervention: for every file, you needed to define four variables in 'INPUTS.conf', and in 'reproduce/analysis/make/download.mk' you had to use a (complex for large number of files) shell 'if/elif/else' condition to link the names of the input files to those variables. Besides inconvenience, this could cause bugs (typos!). Furthermore, a basic MD5 checksum was used for verifying the files. With this commit, a new structure has been defined for 'INPUTS.conf' that (thanks to some pretty useful GNU Make features), removes the need for users to manually edit 'reproduce/analysis/make/download.mk', and reduces the number of variables necessary for each file to three (from four). Furthermore, we now use the SHA256 checksum for input data validation. Regarding the trick used in 'INPUTS.conf' (form the newly added description in 'download.mk'): In GNU Make, '.VARIABLES' "... expands to a list of the names of all global variables defined so far" (from the "Other Special Variables" section of the GNU Make manual). Assuming that the pattern 'INPUT-%-sha256' is only used for input files, we find all the variables that contain the input file names (the '%' is the filename). Finally, using the pattern-substitution function ('patsubst'), we remove the fixed string at the start and end of the variable name. Steps you need to take: - INPUTS.conf: translate your old format to the new format (after carefully reading the description in the comments at the start of the file). After applying the new standards, you don't need to use the variables of 'INPUTS.conf' directly in your Makefiles! For example if one of your input datasets is called 'abc.fits', the checksum variable will be 'INPUT-abc.fits-sha256' and in your high-level Makefiles, you can simply set '$(indir)/abc.fits' as a prerequisite (like you probably did already). - reproduce/analysis/make/download.mk: for the definition and rule of 'inputdatasets', simply use the Maneage branch, and remove anything you had added in your project. In the process, I also noticed that 'README-hacking.md' still referred to 'master' as the main project branch, while we have used 'main' in the paper (and is the common convention with Git).
2022-03-10Bug fix: wrong definition of the prepare directory is correctedRaul Infante-Sainz-1/+1
Until now, the definition of the prepare directory was wrong (not in the 'analysis' directory of the build directory). I noticed this after an update of the Maneage branch of one project that requires the prepare step. With this commit, this problem has been fixed.
2022-03-07paper.tex: fix double dash that was not showing up in output pdfPedram Ashofteh Ardakani-5/+5
Until now, the 'double dash' (i.e. \texttt{--}) in the default 'paper.tex' would only print one (longer) dash in the output pdf. With this commit, the double dashes are replaced with '-{}-' in the LaTeX source as a workaround suggested by Stefan Kottwitz in [1]. [1] https://latex.org/forum/viewtopic.php?f=44&t=4670&start=0
2022-01-21IMPORTANT: Updates to almost all softwareMohammad Akhlaghi-1813/+2760
This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2
2021-10-01Configuration: GCC not linking to system libunwind (crashed GCC's build)Boud Roukema-1/+10
This commit provides a hack/correction to the unwrapped GCC source files that sym-links the generic file 'libgcc/unwind-generic.h' to the two directories in which a file includes "unwind.h" or <unwind.h>. The aim is that the gcc compilation system uses this header file from the internal gcc source files instead of searching for a system-level file 'unwind.h'. This commit also unaliases two 'ls' commands in some build recipes of 'basic.mk' in case the host system (normally at user level) has aliased the command to something like 'ls -F'. In the situation that sometimes occurs of library files being given executable status, the '-F' decorative option could lead to an asterisk being included in a string that is not expected to contain asterisks. If the system shell does not contain the 'alias' command at all, then a fallback of 'true' should provide safe behaviour. The notation of the 'sed' command is also clarified. This solves bug #61240: https://savannah.nongnu.org/bugs/index.php?61240
2021-07-12Configuration: fixed bugs in building of OpenSSL and GettextMohammad Akhlaghi-4/+5
Until now, the 'RPATH' variable (specifying where to look for shared libraries) wasn't being set in the 'libcrypto' library of OpenSSL (it was only set for the 'libssl' library). Also, Gettext used the host Emacs for some operations during installation that could cause the following crash (because we are giving priority to local libraries, which the host Emacs doesn't recognize): emacs: /BDIR/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libk5crypto.so.3) With this commit both these bugs have been fixed: 1) Patchelf is run on the 'libcrypto' library also and 2) we pass the '--without-emacs' configuration option to the configure script of Gettext. These bugs were found by Elham Saremi.
2021-06-25Configuration: New check to see if /dev/shm allows executionBoud Roukema-27/+61
On systems that allow it (like GNU/Linux systems), Maneage will build the necessary software in shared memory (a directory that is actually in the RAM, not on an SSD/HDD, on GNU/Linux systems, it is '/dev/shm'). This allows Maneage to operate faster and not harm the HDD/SSD with all the temporary writing of many small files. Until now, we would only check that this directory exists and that it has enough space. However, some systems also set the 'noexec' flag on shared memory for security reasons [1]. This causes Maneage to crash upon building of the software in later phases. With this commit, at the very start of the configuration step, and after all other shared-memory checks are done, a dummy executable script file is created there and its execution is tested. If it doesn't work, shared memory will not be used at all. In the process, the steps dealing with the software building directory in the configure script have been brought in one place and comments were added to further clarify every step. This commit was initially done by Boud Roukema and later edited by Mohammad Akhlaghi. [1] https://web.archive.org/web/20210624192819/https://serverfault.com/questions/72356/how-useful-is-mounting-tmp-noexec
2021-06-03Update publications: Peper+Roukema publishedBoud Roukema-18/+22
This commit updates some of the publication data in README-hacking.md : Peper+Roukema (2021) is now published in MNRAS and Akhlaghi+ (2021) is published online and very close to getting a conventional volume and page number. :) See task https://savannah.nongnu.org/task/?15736 for ideas of how to make a more systematic publication list instead one managed by prose text. There are already too many non-automated places for publication lists where we have to copy/paste our publication data again and again and again and ... This commit also adds the softwareheritage ID that we have in the content of Akhlaghi+2021 (without the extra context, because as a URL that's very long). There are plenty of arguments to be made each way for different versions of the swh IDS. One advantage of the 'rev' ID is that the hash is the original (full) git hash, which is what I've done for the elaphrocentre and subpoisson papers.
2021-06-03Configuration: improved warning when TeX Live couldn't be installedBoud Roukema-8/+12
Once a year, the texlive update system becomes incompatible with the version from the previous year. Since a texlive install failure is considered non-fatal by 'high-level.mk', so until now, the user could miss the printed message and mistakenly believe that the configure is valid. This commit explicitly adds a 10-second delay that should be enough for a user who does the 'configure --existing-conf' step alone to notice that there is a TeX Live problem. It also adds the explicit instruction of how to allow an update from an earlier year's texlive installer to the warning message (by deleting '.build/software/tarballs/install-tl-unx.tar.gz'). I had to rediscover this a few times for old Maneage installs. Also, a few lines in 'reproduce/software/shell/configure.sh' were indented with a TAB (that is not recommended because TAB is displayed with different widths on different browsers). So while doing this commit, those TABs were also converted to a space.
2021-04-25README.md: edited steps to only build software env in Docker imageMohammad Akhlaghi-76/+151
Until now, while the series of steps mentioned in 'README.md' were complete, they had some implicit thing in them that made it a little hard to run as a checklist (the commands to do some basic things weren't included). Also, it was recommending to run a long 'docker run ...' command, which wasn't too user friendly. With this commit, the series of steps is now a complete checklist, containing every step. Also, the checklist now recommends putting the long 'docker run' command inside a script called 'docker-run' that will also do a 'sudo' internally (thus making things very easy for a first-time user). Also, since the 'docker-run' script contains host OS-specific directory names, it should not be under control, so it has been added to the '.gitignore' file in case users decide to keep this same name (which is recommended).
2021-04-17IMPORTANT: print-general-metadata new name for print-copyrightMohammad Akhlaghi-77/+115
Summary: - Use the new name of this variable in your Makefiles. - In 'metadata.conf', remove fixed URL prefixes for DOIs ('https://doi.org/') or arXiv ('https://arxiv.org/abs'). Until now, the Make variable that would print the general metadata (of whole project) into each to-be-published dataset was called 'print-copyright'! But it now does much more than simply printing the copyright, it will also print a lot of metadata like arXiv ID, Zenodo DOI and etc into plain-text outputs. The out-dated name could thus be misleading and cause confusions. With this commit, the variable is therefore called 'print-general-metadata'. After merging your project with the Maneage branch, please replace any usage of 'print-copyright' to 'print-general-metadata'. Also with this commit, 'README-hacking.md' mentions 'metadata.conf' and 'print-general-metadata' in the "Publication checklist" section and reminds you to keep the first up to date, and use the second in your to-be-published datasets.
2021-03-28Configuration: corrected check of group nameMohammad Akhlaghi-11/+11
When built in 'group' mode, the write permissions of all created files will be activated for a certain group of users in the host operating system. The user specifies the name of the group with the '--group' option at configure time. At the very start, the './project' script checks to see if the given group name actually exists or not (to avoid hard-to-debug errors popping up later). Until now, the checking 'sg' command (that was used to build the project with group-writable permissions) would always fail due to the excessive number of redirections. Therefore, it would always print the error message and abort. With this commit, the output of 'sg' is no longer re-directed (which also helps users in debuggin). If the group does actually exist, it will just print a small statement saying so, and if it fails, the error message is printed. This fixed the problem, allowing maneage to be built in group-mode. I also noticed that the variable name keeping the group name ('reproducible_paper_group_name') used the old name for the project (which was "Reproducible paper template"! So it has been changed/corrected to 'maneage_group_name'.
2021-03-26Initialization: removed other Gnuastro-specific featuresMohammad Akhlaghi-8/+3
In the previous commit, some Gnuastro-specific initializations were removed but a few more cases remained that are removed with this commit.
2021-03-26./project: unused --minmapsize option is removedMohammad Akhlaghi-6/+0
Until now, the './project' script included an '--minmapsize' option which is an option to one of the original programs that was used in Maneage (Gnuastro). Such an option doesn't exist in many other programs, so it is not a suitable option for the generic Maneage project (and can just cause confusion). It was also not used in any part of Maneage any more! With this commit, this option is removed from the core Maneage './project' script and if any project uses it, they can implement it in their own branch.
2021-03-24Maneage installation: removed TCL as a dependency of SWIGMohammad Akhlaghi-1/+2
Until now the SWIG software would use the host operating system's packages to find the TCL configuraiton (which we don't install yet in Maneage). In particular, you can see the error during its configuration here: .... checking for pkg-config... pkg-config checking for Tcl configuration... found /usr/lib/tclConfig.sh /usr/lib/tclConfig.sh: line 2: dpkg-architecture: command not found /usr/lib//tcl8.6/tclConfig.sh: line 2: dpkg-architecture: com. not found With this commit, TCL has been disabled when building SWIG with the '--without-tcl' option. Later, when we add TCL in Maneage, we can remove this option.
2021-03-20Configuration: nullability-completeness warnings suppressedRaul Infante-Sainz-3/+13
With a recent update of macOS systems (macOS Big Sur 11.2.3 and Xcode 12.4), there are many warnings when building C programs (for example the simple program we compile to check the compiler, or some of the software like `gzip'). It prints hundreds of warning lines for every source file that are irrelevant for our builds, but really clutters the output. With this commit, these warnings are disabled by adding `-Wno-nullability-completeness' to the 'CPPFLAGS' environment variable. This has also been added to the very first check of the C compiler in the configure step.
2021-03-20Configuration: --debug option available in this phase alsoRaul Infante-Sainz-13/+40
Until now, each time there was a problem in the configuration of Maneage'd projects and debugging was necessary, we had to take the following changes: - Run the configuration on a single thread ('-j1') to see the building of only the problematic software. - Disable the Zenodo check manually by commenting those parts of 'reproduce/software/shell/configure.sh'. Because the internet connection wastes a few seconds and is thus very annoying during repeated runs! - Manually remove the '-k' option that was passed to Make (when building the software). With the '-k', Make keeps going with the execution of other targets if something crashes and this usually causes confusions during the debugging. Doing the manual changes within the code was both very annoying and prone to errors (forgetting to correct it!). With this commit, the existing '--debug' option has been generalized to the software configuration phase of Maneage also. Until now, it was only available in the analysis phase (and would directly be passed to the 'make' command that would run the analysis). When this option is used, and the project is in the software configuration phase, the Zenodo check won't be done, it will use one single thread ('-j1'), and it will stop the execution as soon as an error occurs (Make is not run with '-k').
2021-02-12Installation: minor correction in links to system librariesMohammad Akhlaghi-2/+6
Until now when making a link to the system's 'dl' and 'pthread' libraries we were simply linking the installed location on the system (in '/usr/lib'). However, in some systems, these may themselves be links to other locations and this could cause linking problems. With this commit, we now use 'realpath' to extract the absolute address of the final file that the libraries may link to, and directly link to them. A minor cosmetic correction was also made in the build rule for CFITSIO: the long line was broken into two!
2021-01-12Default LaTeX preamble: some packages moved to preamble-project.texMohammad Akhlaghi-70/+42
Until now, important LaTeX packages like 'caption' (for managing figure captions), 'hyperref' (for managing links) and 'xcolor' (for managing colors) were being loaded inside the optional 'tex/src/preamble-maneagge-defualt-style.tex' file. We recommend to remove this file from loading when you use custom journal sytels. However, these packages will often be necessary after loading special journal styles also. With this commit, these packages are now loaded into LaTeX as part of the 'tex/src/preamble-project.tex' file. This file is in charge of LaTeX settings that are custom to the project and independent of its style. Several other small corrections are made with this commit: - I noticed that './project make texclean' crashes if no PDF exists in the working directory! So a '-f' was added to the 'rm' command of the 'texclean' rule. - As part of the LaTeX Hyperref, we can set general metadata or properties for the PDF (that aren't written into the printable PDF, but into the file metadata). They can be viewed in many PDF viewers as PDF properties. Until now, we were only using the '\projecttitle' macro here to write the paper's title. However, thanks to the recently added 'reproduce/analysis/config/metadata.conf', we now have a lot of useful information that can also go here. So the 'metadata-copyright-owner' is now used to define the PDF author, and the project's 'metadata-git-repository' and commit hash are written into the PDF subject. But to import these, it was necessary to define them as LaTeX macros, hence the addition of these macros in 'initialize.mk'. - Some extra packages that aren't necessary to build the default PDF were removed in 'preamble-project.tex'.
2021-01-10make dist: removing temp files moved after project-specific filesMohammad Akhlaghi-6/+7
Until now, when you ran './project make dist', first it would delete the temporary files (like files ending in '~' or '.swp' created by some editors), then it had a place to add project-specific operations for the distribution. However, in the process of cleaning the temporary files, it would 'cd' into the directory that would later be packaged. So project-specific operations would first have to 'cd' back into the top source directory. This was prone to hard-to-find bugs. With this commit, to avoid the problem the project-specific operations are now placed before the cleaning phase. This is also technically good because in the project-specific operations there may also be temporary files that shouldn't go into the distribution tarball.
2021-01-09IMPORTANT: analysis outputs written in BDIR/analysisMohammad Akhlaghi-66/+211
Until now, the build directory contained a 'software/' directory (that hosted all the built software), a 'tex/' subdirectory for the final building of the paper, and many other directories containing intermediate/final data of the specific project. But this mixing of built software and data is against our modularity and minimal complexity principles: built software and built data are separate things and keeping them separate will enable many optimizations. With this commit, the build directory of the core Maneage branch will only contain two sub-directories: 'software/' and 'analysis/'. The 'software/' directory has the same contents as before and is not touched in this commit. However, the 'analysis/' directory is new and everything created in the './project make' phase of the project will be created inside of this directory. To facilitate easy access to these top-level built directories, two new variables are defined at the top of 'initialize.mk': 'badir', which is short for "built-analysis directory" and 'bsdir', which is short for "built-software directory". HOW TO IMPLEMENT THIS CHANGE IN YOUR PROJECT. It is easy: simply replace all occurances of '$(BDIR)' in your project's subMakefiles (except the ones below) to '$(badir)'. To confirm if everything is fine before building your project from scratch after merging, you can run the following command to see where 'BDIR' is used and confirm the only remaning cases. $ grep -r BDIR reproduce/analysis/* --> make/verify.mk: innobdir=$$(echo $$infile | sed -e's|$(BDIR)/||g'); \ --> make/initialize.mk:badir=$(BDIR)/analysis --> make/initialize.mk:bsdir=$(BDIR)/software --> make/initialize.mk: $$sys_rm -rf $(BDIR) --> make/top-prepare.mk:all: $(BDIR)/software/preparation-done.mk 'BDIR' should only be present in lines of the files above. If you see '$(BDIR)' used anywhere else, simply change it to '$(badir)'. Ofcourse, if your project assumes BDIR in other contexts, feel free to keep it, it will not conflict. If anything un-expected happens, please post a comment on the link below (you need to be registered on Savannah to post a comment): https://savannah.nongnu.org/task/?15855 One consequence of this change is that the 'analysis/' subdirectory can be optionally mounted on a separate partition. The need for this actually came up for some new users of Maneage in a Docker image. Docker can fix portability problems on systems that we haven't yet supported (even Windows!), or had a chance to fix low-level issues on. However, Docker doesn't have a GUI interface. So to see the built PDF or intermediate data, it was necessary to copy the built data to the host system after every change, which is annoying during working on a project. It would also need two copies of the source: one in the host, one in the container. All these frustrations can be fixed with this new feature. To describe this scenario, README.md now has a new section titled "Only software environment in the Docker image". It explains step-by-step how you can make a Docker image to only host the built software environment. While your project's source, software tarballs and 'BDIR/analysis' directories are on your host operating system. It has been tested before this commit and works very nicely.
2021-01-05Configuration: GNU Binutils linking bug on some systems fixedMohammad Akhlaghi-1/+1
Until now, when building GNU Binutils on GNU Linux operating systems, we would simply put a link to the host's core C library components (the '*crt*' files). However, the symbolic link wasn't "forced"! So if it already existed in the build directory, it would crash. With this commit a '-f' option has been added to the 'ln' command and this fixed the problem. This bug was reported by Zahra Sharbaf.
2021-01-04Building of Less program now uses patchelf to ensure good linkingMohammad Akhlaghi-0/+3
After correctly setting Less to depend on 'ncurses', I noticed its still not linking to Maneage's 'ncurses', but pointing to my host system's 'ncurses' (that happens to have the same version! So it would crash on a system with a different version). This shows that like some other software, we need to manually correct the RPATH inside Less. With this command, the necessary call to 'patchelf' has been added and with it, the installed 'less' command properly linked to Maneage's internal build of 'ncurses'.
2021-01-04README-hacking.md: edits and improvements to publication checklistMohammad Akhlaghi-23/+40
After going through the publication checklist, some edits were made to make things more clear. Also, an item was added to remind the project author that the commit hashes on the uploaded data files should be the same.
2021-01-04README.md: summary Dockerfile with all necessary lines in one stepMohammad Akhlaghi-0/+26
Until now, the description in 'README.md' to build the Dockerfile in 'README.md' had one item per line, thoroughly describing the reason behind that line. But in many cases, the user is already familiar with Docker (or has already read through the items) and just wants to have the Dockerfile ready fast. In these cases, all those extra explanations are annoying. With this commit, an item '0' has been added at the start of the item list for summary. It only contains the necessary Dockerfile contents with no extra explanation.
2021-01-04Building of less software depends on ncursesMohammad Akhlaghi-1/+1
Until now, the 'less' software package (used to view large files easily on the command-line and used by Git for things like 'git diff' or 'git log') only depended on 'patchelf' (which is a very low-level software). However, as Boud reported in bug #59811 [1], building less would crash with an error saying "Cannot find terminal libraries" in some systems (including the proposed Docker image of 'README.md' which I confirmed afterwards). Looking into the 'configure' script of 'less', I noticed that 'less' is actually just checking for some functions provided by the ncurses library! With this commit, 'less' depends on 'ncurses'. I was able to confirm that with this change, 'less' successfully builds within the Docker image. [1] https://savannah.nongnu.org/bugs/?59811
2021-01-02./project make: new texclean targetMohammad Akhlaghi-1/+7
Until now there was only a 'clean' (to delete all files created during the 'make' phase) and the 'distclean' (to delete all files during configuration and make). But sometimes we don't want to delete all the files created during the full 'make' phase, we only want to delete the files that were created by LaTeX for building the paper. Witht this commit, a new target has been added for this job. You can now run the following command for this job: ./project make texclean Only the files in '$(BDIR)/tex/build' will be deleted (and the 'tikz' directory under that location is recreated, ready for a future build).
2021-01-02Copyright year updated in all source filesMohammad Akhlaghi-87/+87
Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.
2020-12-14Better warnings when maneage branch not present and PDF not builtMohammad Akhlaghi-12/+31
Until now, there was no warning when the 'maneage' branch didn't exist in the Git history. This can happen when you forget to push the 'maneage' branch to a remote for your project, and you later clone your project from that remote (for example on another computer). We use the 'maneage' branch to report the latest commit hash and date in the final paper (which can greatly help future readers). Since we check the 'maneage' branch on every run of './project make' (in 'initialize.mk') this would result in a printed statement like this: fatal: Not a valid object name maneage Also until now, the description of what to do when TeXLive wasn't installed properly wasn't complete: it didn't mention that it is necessary to delete the TeXLive target files. This could confuse users (they would re-run './project configure -e', but with no effect). With this commit, for the 'maneage' branch issue a complete warning will be printed. Telling the user what to do to get the 'maneage' branch (and thus fix this warning). Also, the LaTeX macros that go in the paper are now red when the 'maneage' branch doesn't exist, telling the user to see the printed warning (thus encouraging the user to get the branch). For the TeXLive issue, the necessary commands to run are now also printed in the warning.
2020-12-09Configuration: not settting C_INCLUDE_PATH on macOSRaul Infante-Sainz-2/+9
Until now, when building the high-level (optional) software, we would give both 'CPPFLAGS' and 'C_INCLUDE_PATH' the same value/directory in 'high-level.mk'. But we recently found that on macOS's C compiler ('clang'), if a directory is included in both 'CPPFLAGS' and 'C_INCLUDE_PATH', then that directory is ignored in 'CPPFLAGS' (which has higher priority). This caused linking problems when the version of a software on the host was different from the Maneage version. With this commit, 'C_INCLUDE_PATH' is not set on macOS any more and this fixed the problem on the reported systems. This bug was fixed with the help of Mohammad Akhlaghi and Mahdieh Navabi.
2020-12-02Less is now built as a basic softwareMohammad Akhlaghi-0/+13
Less is rarely used in non-interactive mode and is primarily intended for interactively viewing large files. So its need within Maneage (for batch processing) wasn't often felt until now. However, when running './project shell' (which completely closes-off the outside environment), or building a Maneage'd project within a minimal container that doesn't have less, it becomes hard to use Git (and in particular its 'diff' output which depends on 'less'). With this commit, Less has been added as a dependency of Git in 'basic.mk'. In total its built product is roughly 800KB and builds within a second or two. So it isn't a burden on any project. But it can be very useful when the projects are being developed within the Maneage environment itself.
2020-12-01Installation: m4 no longer depends on TexinfoMohammad Akhlaghi-29/+67
In a recent build on a macOS, we recognized that Texinfo needs the 'libintl.h' headers of Gettext. However, Gettext depends on M4, and until now we had set M4 to depend on Texinfo. Therefore adding Gettext as a dependency of Texinfo would cause a circular dependency. On the macOS, we temporarily disabled M4's Texinfo dependency, and the build went through. I also checked on my GNU/Linux system: temporarily renamed all Texinfo built files from my system and done a clean build of M4 and it succeeded. To be further safe, I built Maneage from this commit (where M4 doesn't depend on Texinfo) in a Docker container, and it went through with no problems. So the current M4 version indeed doesn't need Texinfo. I think adding Texinfo as a dependency of M4 was a historic issue from the early days. In the process, I also cleaned 'basic.mk' a little: - A "# Level N" comment was added on top of each group of software that can be built in parallel (generally). - GNU Nano was moved to the end of the file (to be "Level 6"). - Some comments were edited in some places.
2020-12-01README-hacking.md: recommended to push maneage after mergingMohammad Akhlaghi-4/+11
Until now at the end of the updating process, we hadn't explicity talked about pushing the branches. So people would usually only push their 'master' branch to their remote. While the merged 'master' branch does contain the commits from the core Maneage branch, having a no-updated 'maneage' branch reference on their remote can be confusing. With this commit, at the end of the process to merge with the 'maneage' branch we explicitly recommend to push both the 'master' and 'maneage' branches.
2020-12-01Default paper: macros available for date of commits citedMohammad Akhlaghi-5/+12
Until now, Maneage only provided the commit hashes (of the project and Maneage) as LaTeX macros to use in your paper. However, they are too cryptic and not really human friendly (unless you have access to the Git history on a computer). With this commit, to make things easier for the readers, the date of both commits are also available as LaTeX macros for use in the paper. The date of the Maneage commit is also included in the acknowledgements. Also, the paragraph above the acknowledgements has been updated with better explanation on why adding this acknowledgement in the science papers is good/necessary.
2020-12-01IMPORTANT: organizational improvements in Maneage TeX sourcesMohammad Akhlaghi-334/+298
This only concerns the TeX sources in the default branch. In case you don't use them, there should only be a clean conflict in 'paper.tex' (that is obvious and easy to fix). Conflicts may only happen in some of the 'tex/src/preamble-*.tex' files if you have actually changed them for your project. But generally any conflict that does arise by this commit with your project branch should be very clear and easy to fix and test. In short, from now on things will even be easier: any LaTeX configuration that you want to do for your project can be done in 'tex/src/preamble-project.tex', so you don't have to worry about any other LaTeX preamble file. They are either templates (like the ones for PGFPlots and BibLaTeX) or low-level things directly related to Maneage. Until now, this distinction wasn't too clear. Here is a summary of the improvements: - Two new options to './project make': with '--highlight-new' and '--highlight-notes' it is now possible to activate highlighting on the command-line. Until now, there was a LaTeX macro for this at the start of 'paper.tex' (\highlightchanges). But changing that line would change the Git commit hash, making it hard for the readers to trust that this is the same PDF. With these two new run-time options, the printed commit hash will not changed. - paper.tex: the sentences are formatted as one sentence per line (and one line per sentence). This helps in version controlling narrative and following the changes per sentence. A description of this format (and its advantages) is also included in the default text. - The internal Maneage preambles have been modified: - 'tex/src/preamble-header.tex' and 'tex/src/preamble-style.tex' have been merged into one preamble file called 'tex/src/preamble-maneage-default-style.tex'. This helps a lot in simply removing it when you use a journal style file for example. - Things like the options to highlight parts of the text are now put in a special 'tex/src/preamble-maneage.tex'. This helps highlight that these are Maneage-specific features that are independent of the style used in the paper. - There is a new 'tex/src/preamble-project.tex' that is the place you can add your project-specific customizations.
2020-11-28README-hacking.md: updated paper to cite for using ManeageMohammad Akhlaghi-4/+3
Until now, we were asking the users of Maneage to cite the first paper that used its primoridal version (arXiv:1505:01664). But there is now a paper that fully describes the concept (arXiv:2006.03018). With this commit, in the 'citation' section of 'README-hacking.md' we now ask to cite the new paper.
2020-11-15Building final PDF: pdf-build-final has to be given an explicit yesMohammad Akhlaghi-15/+15
Until now, when the 'pdf-build-final' configuration variable (defined in 'reproduce/analysis/config/pdf-build.conf') was given any string a PDF would be built. This was very confusing, because people could put a 'no' and the PDF would still be built! With this commit, only when this variable has a value of 'yes' will the PDF be built. If given any other string (or no string at all), it will not produce a PDF. This issue was reported by Zahra Sharbaf.
2020-11-13README.md: added commands to delete all Docker imagesMohammad Akhlaghi-0/+12
Until now we had described the basic commands on how to create and use Docker images, but we hadn't mentioned how you can delete them. With this commit the commands necessary for deleting Docker images have also been added at the bottom of the section on Docker.
2020-10-18Recipes for final initialize and verify targets not on stdoutMohammad Akhlaghi-4/+11
The LaTeX macro files for these two subMakefiles are created on every run of './project make'. So their commands are also printed every time and hardly ever will a normal user want to modify or change these. So to avoid populating the standard output of a Maneaged project with all these extra lines every time (possibly getting mixed with the important analysis or LaTeX outputs), an '@' has been placed at the start of the recipes. With an '@' at the start of the recipe, Make is instructed to not print the commands it wants to run in the standard output.
2020-10-09Update README-hacking.md with elaphrocentre ArXiv:2010.03742Boud Roukema-2/+11
This commit updates README-hacking.md with the URIs for the 'elaphrocentre' galaxy formation pipeline paper arXiv:2010.03742. This makes three papers currently in the peer review pipeline: arXiv:2006.03018, arXiv:2007.11779, and arXiv:2010.03742, each chronologically corresponding to various stages of the review process.
2020-10-02TexLive's xstring package is now necessaryMohammad Akhlaghi-4/+4
After a fresh build of Maneage with a newly downloaded TeXLive, I noticed that it is complaining about not finding 'xstring.sty', apparently some package that depeneded on it is no longer including it itself! It is thus now added to the packages that are built by Maneage's TeXLive.
2020-09-24Gnuastro's analysis configuration files removedMohammad Akhlaghi-141/+7
Until now, the core Maneage branch included some configuration files for Gnuastro's programs. This was actually a remnant of the distant past when Maneage didn't actually build its own software and we had to rely on the host's software versions. This file contained the configuration files specific to Gnuastro for this project and also had a feature to avoid checking the host's own configuration files. However, we now build all our software ourselves with fixed configuration files (for the version that is being installed and its version is stored). So those extra configuration files were just extra and caused confusion and problems in some scenarios. With this commit, those extra files are now removed. Also, two small issues are also addressed in parallel with this commit: - When running './project make clean', the 'hardware-parameters.tex' macro file (which is created by './project configure' is not deleted. - The project title is now written into the default output's PDF's properties (through 'hypersetup' in 'tex/src/preamble-header.tex') through the LaTeX macro. All these issues were found and fixed with the help of Samane Raji.
2020-09-15Checking Xcode installation for macOS systemsRaul Infante-Sainz-1/+60
Until now, during the configure step it was checked if the host Operative System were GNU/Linux, and if not, we assumed it is macOS. However, it can be any other different OS! With this commit, now we explicity check if the system is GNU/Linux or Darwin (macOS). If it is not any of them, a warning message says to the user that the host system is different from which we have checked so far (and invite to contact us if there is any problem). In addition to this, if the system is macOS, now it checks if Xcode is already installed in the host system. If it is not installed, a warning message informs the user to do that in case a problem/crash in the configure step occurs. We have found that it is convenient to have Xcode installed in order to avoid some problems.
2020-09-09R is built without tcl/tk (for GUI) dependenceBoud Roukema-0/+19
Tcl/Tk are a set of tools to provide Graphic User Interface (GUI) support in some software. But they are not yet natively built within Maneage, primarily because we have higher-priority work right now. GUI tools in general aren't high on our priority list right now because GUI tools are generally good for human interaction (which is contrary to the reproducible philosophy), not automatic analysis (a core concept in reproducibility). So even later, when we do include Tcl/Tk in Maneage, their direct usage will be discouraged. Until this commit, because we don't yet build Tcl/Tk, the default maneage install of the statistical package R failed on a Debian Stretch, with 6227 repeats of the line: '/usr/lib//tcl8.5/tclConfig.sh: line 2: dpkg-architecture: command not found' To fix this problem (atleast until Tcl/Tk is installed within Maneage), R is now configured with the '--without-tcltk' option which fixed the problem. Please see the description above the R installation instructions in 'reproduce/software/make/high-level.mk' for more.
2020-09-08Removed all occurances of IFS in low-level scriptsMohammad Akhlaghi-11/+11
Following the previous commit, we recognized that the 'IFS' terms are not necessary and can be even cause problems. So all their occurances in the scripts of Maneage have been removed with this commit.
2020-09-07Software installation: removed IFS statements in pre-make-build.shMohammad Akhlaghi-4/+4
Until a recent commit, the IFS='"' was added at the start of the variables in this shell script and as a result, the SPACE character wasn't being used as a delimiter. This caused a major problem when downloading the tarballs (all the backup servers were considered as the top link). With this commit we removed these 'IFS' statements). Because we now check for the existance of meta-characters in the build directory name, there is no more problem, and also generally both the calling command and internally, we have double-qutations around the variable names. So removal of IFS will not affect the result in this scenario. This bug was found by Mohammadreza Khellat.