project.git - Core Maneage branch (where all projects derive from)

Age	Commit message (Collapse)	Author	Lines
2023-05-07	Copyright years: updated to 2023, accompanied by some minor fixes	Mohammad Akhlaghi	-2/+2
	SUMMARY: just house-cleaning, no need to do anything major in your branch. Just update the copyright years in files that you have added. Until now, the latest copyright years of the whole Maneage source code was 2022! As of this commit, we have already moved to 2023 for 5 months! Furthermore, there were a few other minor issues that needed correction: - The URL to download input datasets wasn't quoted in 'initialize.mk' or the download script! As a result, when the input URL had characters that are meaningful to the shell (like '&'), the download command would not work. - The only program that had 'make check' in the 'basic.mk' programs was MPFR. At that stage, we still haven't built our own compiler at this stage, this is not accurate. - The 'pyerfa' and 'extension-helpers' packages in Python need 'setuptools_scm' on some systems. But until now, it was not in the list of their prerequisites. With this commit, all the issues above have been corrected.
2022-01-21	IMPORTANT: Updates to almost all software	Mohammad Akhlaghi	-2/+2
	This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2
2021-01-02	Copyright year updated in all source files	Mohammad Akhlaghi	-2/+2
	Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.
2020-06-27	IMPORTANT: many improvements to low-level software building phase	Mohammad Akhlaghi	-0/+71
	POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause conflicts to your project if you have changed the software building Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and 'python.mk'). If your project has only added analysis, it shouldn't be affected. This is a large commit, involving a long series of corrections in a differnt branch which is now finally being merged into the core Maneage branch. All changes were related and came up naturally as the low-level infrastructure was improved. So separating them in the end for the final merge would have been very time consuming and we are merging them as one commit. In general, the software building Makefiles are now much more easier to read, modify and use, along with several new features that have been added. See below for the full list. - Until now, Maneage needed the host to have a 'make' implementation because Make was necessary to build Lzip. Lzip is then used to uncompress the source of our own GNU Make. However, in the minimalist/slim versions of operating systems (for example used to build Docker images) Make isn't included by default. Since Lzip was the only program before our own GNU Make was installed, we consulting Antonio Diaz Diaz (creator of Lzip) and he kindly added the necessary functionality to a new version of Lzip, which we are using now. Hence we don't need to assume a Make implementation on the host any more. With this commit, Lzip and GNU Make are built without Make, allowing everything else to be safely built with our own custom version of GNU Make and not using the host's 'make' at all. - Until recently (Commit 3d8aa5953c4) GNU Make was built in 'basic.mk'. Therefore 'basic.mk' was written in a way that it can be used with other 'make' implementations also (i.e., important shell commands starting with '&&' and ending in '\' without any comments between them!). Furthermore, to help in style uniformity, the rules in 'high-level.mk' and 'python.mk' also followed a similar structure. But due to the point above, we can now guarantee that GNU Make is used from the very first Makefile, so this hard-to-read structure has been removed in the software build recipes and they are much more readable and edit-friendly now. - Until now, the default backup servers where at some fixed URLs, on our own pages or on Gitlab. But recently we uploaded all the necessary software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is more suitable for this task (it promises longevity, has a fixed DOI, while allowing us to add new content, or new software tarball versions). With this commit, a small script has been written to extract the most recent Zenodo upload link from the Zenodo DOI and use it for downloading the software source codes. - Until now, we primarily used the webpage of each software for downloading its tarball. But this caused many problems: 1) Some of them needed Javascript before the download, 2) Some URLs had a complex dependency on the version number, 3) some servers would be randomly down for maintenance and etc. So thanks to the point above, we now use the Zenodo server as the primary download location. However, if a user wants to use a custom software that is not (yet!) in Zenodo, the download script gives priority to a custom URL that the users can give as Make variables. If that variable is defined, then the script will use that URL before going onto Zenodo. We now have a special place for such URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a good documentation themselves) are preserved here, but are commented by default. - The software source code downloading and checksum verification step has been moved into a Make function called 'import-source' (defined in the 'build-rules.mk' and loaded in all software Makefiles). Having taken all the low-level steps there, I noticed that there is no more need for having the tarball as a separate target! So with this commit, a single rule is the only place that needs to be edited/added (greatly simplifying the software building Makefiles). - Following task #15272, A new option has been added to the './project' script called '--all-highlevel'. When this option is given, the contents of 'TARGETS.conf' are ignored and all the software in Maneage are built (selected by parsing the 'versions.conf' file). This new option was added to confirm the extensive changes made in all the software building recipes and is great for development/testing purposes. - Many of the software hadn't been tested for a long time! So after using the newly added '--all-highlevel', we noticed that some need to be updated. In general, with this commit, 'libpaper' and 'pcre' were added as new software, and the versions of the following software was updated: 'boost', 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in' shell script was added in 'reproduce/software/shell/' which is installed with 'libpaper'. - Even though we intentionally add the necessary flags to add RPATH inside the built executable at compilation time, some software don't do it (different software on different operating systems!). Until now, for historical reasons this check was done in different ways for different software on GNU/Linux sytems. But now it is unified: if 'patchelf' is present we apply it. Because of this, 'patchelf' has been put as a top-level prerequisite, right after Tar and is installed before anything else. - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in 'basic.mk', it was 'glibtool'! This caused many confusions and is corrected with this commit (in 'basic.mk', it is also 'libtool'). - A new argument is added to the './project' script to allow easy loading of the project's shell and environment for fast/temporary testing of things in the same environment as the project. Before activating the project's shell, we completely remove all host environment variables to simulate the project's environment. It can be called with this command: './project shell'. A simple prompt has also been added to highlight that the user is using the Maneage shell!

SUMMARY: just house-cleaning, no need to do anything major in your branch. Just update the copyright years in files that you have added. Until now, the latest copyright years of the whole Maneage source code was 2022! As of this commit, we have already moved to 2023 for 5 months! Furthermore, there were a few other minor issues that needed correction: - The URL to download input datasets wasn't quoted in 'initialize.mk' or the download script! As a result, when the input URL had characters that are meaningful to the shell (like '&'), the download command would not work. - The only program that had 'make check' in the 'basic.mk' programs was MPFR. At that stage, we still haven't built our own compiler at this stage, this is not accurate. - The 'pyerfa' and 'extension-helpers' packages in Python need 'setuptools_scm' on some systems. But until now, it was not in the list of their prerequisites. With this commit, all the issues above have been corrected.

This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2

Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.

POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause conflicts to your project if you have changed the software building Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and 'python.mk'). If your project has only added analysis, it shouldn't be affected. This is a large commit, involving a long series of corrections in a differnt branch which is now finally being merged into the core Maneage branch. All changes were related and came up naturally as the low-level infrastructure was improved. So separating them in the end for the final merge would have been very time consuming and we are merging them as one commit. In general, the software building Makefiles are now much more easier to read, modify and use, along with several new features that have been added. See below for the full list. - Until now, Maneage needed the host to have a 'make' implementation because Make was necessary to build Lzip. Lzip is then used to uncompress the source of our own GNU Make. However, in the minimalist/slim versions of operating systems (for example used to build Docker images) Make isn't included by default. Since Lzip was the only program before our own GNU Make was installed, we consulting Antonio Diaz Diaz (creator of Lzip) and he kindly added the necessary functionality to a new version of Lzip, which we are using now. Hence we don't need to assume a Make implementation on the host any more. With this commit, Lzip and GNU Make are built without Make, allowing everything else to be safely built with our own custom version of GNU Make and not using the host's 'make' at all. - Until recently (Commit 3d8aa5953c4) GNU Make was built in 'basic.mk'. Therefore 'basic.mk' was written in a way that it can be used with other 'make' implementations also (i.e., important shell commands starting with '&&' and ending in '\' without any comments between them!). Furthermore, to help in style uniformity, the rules in 'high-level.mk' and 'python.mk' also followed a similar structure. But due to the point above, we can now guarantee that GNU Make is used from the very first Makefile, so this hard-to-read structure has been removed in the software build recipes and they are much more readable and edit-friendly now. - Until now, the default backup servers where at some fixed URLs, on our own pages or on Gitlab. But recently we uploaded all the necessary software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is more suitable for this task (it promises longevity, has a fixed DOI, while allowing us to add new content, or new software tarball versions). With this commit, a small script has been written to extract the most recent Zenodo upload link from the Zenodo DOI and use it for downloading the software source codes. - Until now, we primarily used the webpage of each software for downloading its tarball. But this caused many problems: 1) Some of them needed Javascript before the download, 2) Some URLs had a complex dependency on the version number, 3) some servers would be randomly down for maintenance and etc. So thanks to the point above, we now use the Zenodo server as the primary download location. However, if a user wants to use a custom software that is not (yet!) in Zenodo, the download script gives priority to a custom URL that the users can give as Make variables. If that variable is defined, then the script will use that URL before going onto Zenodo. We now have a special place for such URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a good documentation themselves) are preserved here, but are commented by default. - The software source code downloading and checksum verification step has been moved into a Make function called 'import-source' (defined in the 'build-rules.mk' and loaded in all software Makefiles). Having taken all the low-level steps there, I noticed that there is no more need for having the tarball as a separate target! So with this commit, a single rule is the only place that needs to be edited/added (greatly simplifying the software building Makefiles). - Following task #15272, A new option has been added to the './project' script called '--all-highlevel'. When this option is given, the contents of 'TARGETS.conf' are ignored and all the software in Maneage are built (selected by parsing the 'versions.conf' file). This new option was added to confirm the extensive changes made in all the software building recipes and is great for development/testing purposes. - Many of the software hadn't been tested for a long time! So after using the newly added '--all-highlevel', we noticed that some need to be updated. In general, with this commit, 'libpaper' and 'pcre' were added as new software, and the versions of the following software was updated: 'boost', 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in' shell script was added in 'reproduce/software/shell/' which is installed with 'libpaper'. - Even though we intentionally add the necessary flags to add RPATH inside the built executable at compilation time, some software don't do it (different software on different operating systems!). Until now, for historical reasons this check was done in different ways for different software on GNU/Linux sytems. But now it is unified: if 'patchelf' is present we apply it. Because of this, 'patchelf' has been put as a top-level prerequisite, right after Tar and is installed before anything else. - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in 'basic.mk', it was 'glibtool'! This caused many confusions and is corrected with this commit (in 'basic.mk', it is also 'libtool'). - A new argument is added to the './project' script to allow easy loading of the project's shell and environment for fast/temporary testing of things in the same environment as the project. Before activating the project's shell, we completely remove all host environment variables to simulate the project's environment. It can be called with this command: './project shell'. A simple prompt has also been added to highlight that the user is using the Maneage shell!