Until now, one had to follow the instructions from [1] to prepare a
standard software tarball before merging with the low-level
tarballs-software repository [2]. The script only worked for '.tar.gz'
suffix and was only available as a comment on Savannah (in [1]).
With this commit, the script has been imported into Maneage as
'reproduce/software/shell/tarball-prepare.sh' to simplify future software
updates. It work with all supported '.tar.*' suffixes (of the upstream
tarball repository) and will convert the tarballs to Maneage's standard
format. Also, this script has a minimal argument parser and can skip the
tarballs that are already unpacked, allowing faster tests.
This script was used to update the versions of:
Coreutiles 9.0 --> 9.1
Git 2.34 --> 2.36
Emacs 27.2 --> 28.1
The main motive behind this update was Git which announced a vulnerability
issue [3] and suggested an update to the latest version as soon as
possible. More detail is described in this github blog [4], but in summary,
it was a security issue on multi-user systems that has been found and fixed
by Git developers. Since Maneage is often installed on such shared systems,
it was important to make this update. GNU Coreutils and GNU Emacs were also
updated because they are also commonly used.
The following improvements have also done with this commit:
- .gitignore: ignore emacs auto-save files (that end with a '#')
- README-hacking.md: In the checklist for updating the Maneage branch, the
no-longer-necessary '--decorate' option of Git was removed from the
command to check the general branch history.
[1] https://savannah.nongnu.org/task/?15699
[2] https://git.maneage.org/tarballs-software.git/
[3] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/
[4] https://github.blog/2022-04-12-git-security-vulnerability-announced/
This commit primarily affects the configuration step of Maneage'd projects,
and in particular, updated versions of the many of the software (see
P.S.). So it shouldn't affect your high-level analysis other than the
version bumps of the software you use (and the software's possibly
improve/changed behavior).
The following software (and thus their dependencies) couldn't be updated as
described below:
- Cryptography: isn't building because it depends on a new
setuptools-rust package that has problems
(https://savannah.nongnu.org/bugs/index.php?61731), so it has been
commented in 'versions.conf'.
- SecretStorage: because it depends on Cryptography.
- Keyring: because it depends on SecretStorage.
- Astroquery: because it depends on Keyring.
This is a "squashed" commit after rebasing a development branch of 60
commits corresponding to a roughly two-month time interval. The following
people contributed to this branch.
- Boudewijn Roukema added all the R software infrastructure and the R
packages, as well as greatly helping in fixing many bugs during the
update.
- Raul Infante-Sainz helped in testing and debugging the build.
- Pedram Ashofteh Ardakani found and fixed a bug.
- Zahra Sharbaf helped in testing and found several bugs.
Below a description of the most noteworthy points is given.
- Software tarballs: all updated software now have a unified format
tarball (ustar; if not possible, pax) and unified compression (Lzip) in
Maneage's software repository in Zenodo
(https://doi.org/10.5281/zenodo.3883409). For more on this See
https://savannah.nongnu.org/task/?15699 . This won't affect any extra
software you would like to add; you can use any format recognized by
GNU Tar, and all common compression algorithms. This new requirement is
only for software that get merged to the core Maneage branch.
- Metastore (and thus libbsd and libmd) moved to highlevel: Metastore
(and the packages it depends on) is a high-level product that is only
relevant during the project development (like Emacs!): when the user
wants the file meta data (like dates) to be unchanged after checking
out branches. So it should be considered a high-level software, not
basic. Metastore also usually causes many more headaches and error
messages, so personally, I have stopped using it! Instead I simply
merge my branches in a separate clone, then pull the merge commit: in
this way, the files of my project aren't re-written during the checkout
phase and therefore their dates are untouched (which can conflict with
Make's dates on configuration files).
- The un-official cloned version of Flex (2.6.4-91 until this commit) was
causing problems in the building of Netpbm, so with this commit, it has
been moved back to version 2.6.4.
- Netpbm's official page had version 10.73.38 as the latest stable
tarball that was just released in late 2021. But I couldn't find our
previously-used version 10.86.99 anywhere (to see when it was released
and why we used it! Its at last more than one year old!). So the
official stable version is being used now.
- Improved instructions in 'README.md' for building software environment
in a Docker container (while having project source and output data
products on the local system; including the usage of the host's
'/dev/shm' to speed up temporary operations).
- Until now, the convention in Maneage was to put eight SPACE characters
before the comment lines within recipes. This was done because by
default GNU Emacs (also many other editors) show a TAB as eight
characters. However, in other text editors, online browsers, or even
the Git diff, a TAB can correspond to a different number of
characters. In such cases, the Maneage recipes wouldn't look too
interesting (the comments and the recipe commands would show a
different indentation!).
With this commit, all the comment lines in the Makefiles within the
core Maneage branch have a hash ('#') as their first character and a
TAB as the second. This allows the comment lines in recipes to have the
same indentation as code; making the code much more easier to read in a
general scenario including a 'git diff' (editor agnostic!).
P.S. List of updated software with their old and new versions
- Software with no version update are not mentioned.
- The old version of newly added software are shown with '--'.
Name (Basic) Old version New version
------------ ----------- -----------
Bzip2 1.0.6 1.0.8
CURL 7.71.1 7.79.1
Dash 0.5.10.2 0.5.11.5
File 5.39 5.41
Flock 0.2.3 0.4.0
GNU Bash 5.0.18 5.1.8
GNU Binutils 2.35 2.37
GNU Coreutils 8.32 9.0
GNU GCC 10.2.0 11.2.0
GNU M4 1.4.18 1.4.19
GNU Readline 8.0 8.1.1
GNU Tar 1.32 1.34
GNU Texinfo 6.7 6.8
GNU diffutils 3.7 3.8
GNU findutils 4.7.0 4.8.0
GNU gmp 6.2.0 6.2.1
GNU grep 3.4 3.7
GNU gzip 1.10 1.11
GNU libunistring 0.9.10 1.0
GNU mpc 1.1.0 1.2.1
GNU mpfr 4.0.2 4.1.0
GNU nano 5.2 6.0
GNU ncurses 6.2 6.3
GNU wget 1.20.3 1.21.2
Git 2.28.0 2.34.0
Less 563 590
Libxml2 2.9.9 2.9.12
Lzip 1.22-rc2 1.22
OpenSLL 1.1.1a 3.0.0
Patchelf 0.10 0.13
Perl 5.32.0 5.34.0
Podlators -- 4.14
Name (Highlevel) Old version New version
---------------- ----------- -----------
Apachelog4cxx 0.10.0-603 0.12.1
Astrometry.net 0.80 0.85
Boost 1.73.0 1.77.0
CFITSIO 3.48 4.0.0
Cmake 3.18.1 3.21.4
Eigen 3.3.7 3.4.0
Expat 2.2.9 2.4.1
FFTW 3.3.8 3.3.10
Flex 2.6.4-91 2.6.4
Fontconfig 2.13.1 2.13.94
Freetype 2.10.2 2.11.0
GNU Astronomy Utilities 0.12 0.16.1-e0f1
GNU Autoconf 2.69.200-babc 2.71
GNU Automake 1.16.2 1.16.5
GNU Bison 3.7 3.8.2
GNU Emacs 27.1 27.2
GNU GDB 9.2 11.1
GNU GSL 2.6 2.7
GNU Help2man 1.47.11 1.48.5
Ghostscript 9.52 9.55.0
ICU -- 70.1
ImageMagick 7.0.8-67 7.1.0-13
Libbsd 0.10.0 0.11.3
Libffi 3.2.1 3.4.2
Libgit2 1.0.1 1.3.0
Libidn 1.36 1.38
Libjpeg 9b 9d
Libmd -- 1.0.4
Libtiff 4.0.10 4.3.0
Libx11 1.6.9 1.7.2
Libxt 1.2.0 1.2.1
Netpbm 10.86.99 10.73.38
OpenBLAS 0.3.10 0.3.18
OpenMPI 4.0.4 4.1.1
Pixman 0.38.0 0.40.0
Python 3.8.5 3.10.0
R 4.0.2 4.1.2
SWIG 3.0.12 4.0.2
Util-linux 2.35 2.37.2
Util-macros 1.19.2 1.19.3
Valgrind 3.15.0 3.18.1
WCSLIB 7.3 7.7
Xcb-proto 1.14 1.14.1
Xorgproto 2020.1 2021.5
Name (Python) Old version New version
------------- ----------- -----------
Astropy 4.0 5.0
Beautifulsoup4 4.7.1 4.10.0
Beniget -- 0.4.1
Cffi 1.12.2 1.15.0
Cryptography 2.6.1 36.0.1
Cycler 0.10.0 0.11.0+}
Cython 0.29.21 0.29.24
Esutil 0.6.4 0.6.9
Extension-helpers -- 0.1
Galsim 2.2.1 2.3.3
Gast -- 0.5.3
Jinja2 -- 3.0.3
MPI4py 3.0.3 3.1.3
Markupsafe -- 2.0.1
Numpy 1.19.1 1.21.3
Packaging -- 21.3
Pillow -- 8.4.0
Ply -- 3.11
Pyerfa -- 2.0.0.1
Pyparsing 2.3.1 3.0.4
Pythran -- 0.11.0
Scipy 1.5.2 1.7.3
Setuptools 41.6.0 58.3.0
Six 1.12.0 1.16.0
Uncertainties 3.1.2 3.1.6
Wheel -- 0.37.0
Name (R) Old version New version
-------- ----------- -----------
Cli -- 2.5.0
Colorspace -- 2.0-1
Cowplot -- 1.1.1
Crayon -- 1.4.1
Digest -- 0.6.27
Ellipsis -- 0.3.2
Fansi -- 0.5.0
Farver -- 2.1.0
Ggplot2 -- 3.3.4
Glue -- 1.4.2
GridExtra -- 2.3
Gtable -- 0.3.0
Isoband -- 0.2.4
Labeling -- 0.4.2
Lifecycle -- 1.0.0
Magrittr -- 2.0.1
MASS -- 7.3-54
Mgcv -- 1.8-36
Munsell -- 0.5.0
Pillar -- 1.6.1
R-Pkgconfig -- 2.0.3
R6 -- 2.5.0
RColorBrewer -- 1.1-2
Rlang -- 0.4.11
Scales -- 1.1.1
Tibble -- 3.1.2
Utf8 -- 1.2.1
Vctrs -- 0.3.8
ViridisLite -- 0.4.0
Withr -- 2.4.2
This commit provides a hack/correction to the unwrapped GCC source files that sym-links the generic file 'libgcc/unwind-generic.h' to the two directories in which a file includes "unwind.h" or <unwind.h>. The aim is that the gcc compilation system uses this header file from the internal gcc source files instead of searching for a system-level file 'unwind.h'. This commit also unaliases two 'ls' commands in some build recipes of 'basic.mk' in case the host system (normally at user level) has aliased the command to something like 'ls -F'. In the situation that sometimes occurs of library files being given executable status, the '-F' decorative option could lead to an asterisk being included in a string that is not expected to contain asterisks. If the system shell does not contain the 'alias' command at all, then a fallback of 'true' should provide safe behaviour. The notation of the 'sed' command is also clarified. This solves bug #61240: https://savannah.nongnu.org/bugs/index.php?61240
Until now, the 'RPATH' variable (specifying where to look for shared libraries) wasn't being set in the 'libcrypto' library of OpenSSL (it was only set for the 'libssl' library). Also, Gettext used the host Emacs for some operations during installation that could cause the following crash (because we are giving priority to local libraries, which the host Emacs doesn't recognize): emacs: /BDIR/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libk5crypto.so.3) With this commit both these bugs have been fixed: 1) Patchelf is run on the 'libcrypto' library also and 2) we pass the '--without-emacs' configuration option to the configure script of Gettext. These bugs were found by Elham Saremi.
On systems that allow it (like GNU/Linux systems), Maneage will build the necessary software in shared memory (a directory that is actually in the RAM, not on an SSD/HDD, on GNU/Linux systems, it is '/dev/shm'). This allows Maneage to operate faster and not harm the HDD/SSD with all the temporary writing of many small files. Until now, we would only check that this directory exists and that it has enough space. However, some systems also set the 'noexec' flag on shared memory for security reasons [1]. This causes Maneage to crash upon building of the software in later phases. With this commit, at the very start of the configuration step, and after all other shared-memory checks are done, a dummy executable script file is created there and its execution is tested. If it doesn't work, shared memory will not be used at all. In the process, the steps dealing with the software building directory in the configure script have been brought in one place and comments were added to further clarify every step. This commit was initially done by Boud Roukema and later edited by Mohammad Akhlaghi. [1] https://web.archive.org/web/20210624192819/https://serverfault.com/questions/72356/how-useful-is-mounting-tmp-noexec
Once a year, the texlive update system becomes incompatible with the version from the previous year. Since a texlive install failure is considered non-fatal by 'high-level.mk', so until now, the user could miss the printed message and mistakenly believe that the configure is valid. This commit explicitly adds a 10-second delay that should be enough for a user who does the 'configure --existing-conf' step alone to notice that there is a TeX Live problem. It also adds the explicit instruction of how to allow an update from an earlier year's texlive installer to the warning message (by deleting '.build/software/tarballs/install-tl-unx.tar.gz'). I had to rediscover this a few times for old Maneage installs. Also, a few lines in 'reproduce/software/shell/configure.sh' were indented with a TAB (that is not recommended because TAB is displayed with different widths on different browsers). So while doing this commit, those TABs were also converted to a space.
When built in 'group' mode, the write permissions of all created files will
be activated for a certain group of users in the host operating system. The
user specifies the name of the group with the '--group' option at configure
time. At the very start, the './project' script checks to see if the given
group name actually exists or not (to avoid hard-to-debug errors popping up
later).
Until now, the checking 'sg' command (that was used to build the project
with group-writable permissions) would always fail due to the excessive
number of redirections. Therefore, it would always print the error message
and abort.
With this commit, the output of 'sg' is no longer re-directed (which also
helps users in debuggin). If the group does actually exist, it will just
print a small statement saying so, and if it fails, the error message is
printed. This fixed the problem, allowing maneage to be built in
group-mode.
I also noticed that the variable name keeping the group name
('reproducible_paper_group_name') used the old name for the project (which
was "Reproducible paper template"! So it has been changed/corrected to
'maneage_group_name'.
Until now the SWIG software would use the host operating system's packages to find the TCL configuraiton (which we don't install yet in Maneage). In particular, you can see the error during its configuration here: .... checking for pkg-config... pkg-config checking for Tcl configuration... found /usr/lib/tclConfig.sh /usr/lib/tclConfig.sh: line 2: dpkg-architecture: command not found /usr/lib//tcl8.6/tclConfig.sh: line 2: dpkg-architecture: com. not found With this commit, TCL has been disabled when building SWIG with the '--without-tcl' option. Later, when we add TCL in Maneage, we can remove this option.
With a recent update of macOS systems (macOS Big Sur 11.2.3 and Xcode 12.4), there are many warnings when building C programs (for example the simple program we compile to check the compiler, or some of the software like `gzip'). It prints hundreds of warning lines for every source file that are irrelevant for our builds, but really clutters the output. With this commit, these warnings are disabled by adding `-Wno-nullability-completeness' to the 'CPPFLAGS' environment variable. This has also been added to the very first check of the C compiler in the configure step.
Until now, each time there was a problem in the configuration of Maneage'd
projects and debugging was necessary, we had to take the following changes:
- Run the configuration on a single thread ('-j1') to see the building of
only the problematic software.
- Disable the Zenodo check manually by commenting those parts of
'reproduce/software/shell/configure.sh'. Because the internet connection
wastes a few seconds and is thus very annoying during repeated runs!
- Manually remove the '-k' option that was passed to Make (when building
the software). With the '-k', Make keeps going with the execution of
other targets if something crashes and this usually causes confusions
during the debugging.
Doing the manual changes within the code was both very annoying and prone
to errors (forgetting to correct it!).
With this commit, the existing '--debug' option has been generalized to the
software configuration phase of Maneage also. Until now, it was only
available in the analysis phase (and would directly be passed to the 'make'
command that would run the analysis). When this option is used, and the
project is in the software configuration phase, the Zenodo check won't be
done, it will use one single thread ('-j1'), and it will stop the execution
as soon as an error occurs (Make is not run with '-k').