| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, Maneage had undergone some updates.
With this commit, those updates have been imported and the conflicts that
resulted were fixed. They were all cosmetic and had no effect on the
analysis. The most significant one was about the change in the format of
'INPUTS.conf'.
In the process, I also noticed that the IEEEtran LaTeX package is now
called 'ieeetran' (the 'tlmgr' of TeXLive 2022 was failing).
|
|
Until now, one had to follow the instructions from [1] to prepare a
standard software tarball before merging with the low-level
tarballs-software repository [2]. The script only worked for '.tar.gz'
suffix and was only available as a comment on Savannah (in [1]).
With this commit, the script has been imported into Maneage as
'reproduce/software/shell/tarball-prepare.sh' to simplify future software
updates. It work with all supported '.tar.*' suffixes (of the upstream
tarball repository) and will convert the tarballs to Maneage's standard
format. Also, this script has a minimal argument parser and can skip the
tarballs that are already unpacked, allowing faster tests.
This script was used to update the versions of:
Coreutiles 9.0 --> 9.1
Git 2.34 --> 2.36
Emacs 27.2 --> 28.1
The main motive behind this update was Git which announced a vulnerability
issue [3] and suggested an update to the latest version as soon as
possible. More detail is described in this github blog [4], but in summary,
it was a security issue on multi-user systems that has been found and fixed
by Git developers. Since Maneage is often installed on such shared systems,
it was important to make this update. GNU Coreutils and GNU Emacs were also
updated because they are also commonly used.
The following improvements have also done with this commit:
- .gitignore: ignore emacs auto-save files (that end with a '#')
- README-hacking.md: In the checklist for updating the Maneage branch, the
no-longer-necessary '--decorate' option of Git was removed from the
command to check the general branch history.
[1] https://savannah.nongnu.org/task/?15699
[2] https://git.maneage.org/tarballs-software.git/
[3] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/
[4] https://github.blog/2022-04-12-git-security-vulnerability-announced/
|
|
This commit primarily affects the configuration step of Maneage'd projects,
and in particular, updated versions of the many of the software (see
P.S.). So it shouldn't affect your high-level analysis other than the
version bumps of the software you use (and the software's possibly
improve/changed behavior).
The following software (and thus their dependencies) couldn't be updated as
described below:
- Cryptography: isn't building because it depends on a new
setuptools-rust package that has problems
(https://savannah.nongnu.org/bugs/index.php?61731), so it has been
commented in 'versions.conf'.
- SecretStorage: because it depends on Cryptography.
- Keyring: because it depends on SecretStorage.
- Astroquery: because it depends on Keyring.
This is a "squashed" commit after rebasing a development branch of 60
commits corresponding to a roughly two-month time interval. The following
people contributed to this branch.
- Boudewijn Roukema added all the R software infrastructure and the R
packages, as well as greatly helping in fixing many bugs during the
update.
- Raul Infante-Sainz helped in testing and debugging the build.
- Pedram Ashofteh Ardakani found and fixed a bug.
- Zahra Sharbaf helped in testing and found several bugs.
Below a description of the most noteworthy points is given.
- Software tarballs: all updated software now have a unified format
tarball (ustar; if not possible, pax) and unified compression (Lzip) in
Maneage's software repository in Zenodo
(https://doi.org/10.5281/zenodo.3883409). For more on this See
https://savannah.nongnu.org/task/?15699 . This won't affect any extra
software you would like to add; you can use any format recognized by
GNU Tar, and all common compression algorithms. This new requirement is
only for software that get merged to the core Maneage branch.
- Metastore (and thus libbsd and libmd) moved to highlevel: Metastore
(and the packages it depends on) is a high-level product that is only
relevant during the project development (like Emacs!): when the user
wants the file meta data (like dates) to be unchanged after checking
out branches. So it should be considered a high-level software, not
basic. Metastore also usually causes many more headaches and error
messages, so personally, I have stopped using it! Instead I simply
merge my branches in a separate clone, then pull the merge commit: in
this way, the files of my project aren't re-written during the checkout
phase and therefore their dates are untouched (which can conflict with
Make's dates on configuration files).
- The un-official cloned version of Flex (2.6.4-91 until this commit) was
causing problems in the building of Netpbm, so with this commit, it has
been moved back to version 2.6.4.
- Netpbm's official page had version 10.73.38 as the latest stable
tarball that was just released in late 2021. But I couldn't find our
previously-used version 10.86.99 anywhere (to see when it was released
and why we used it! Its at last more than one year old!). So the
official stable version is being used now.
- Improved instructions in 'README.md' for building software environment
in a Docker container (while having project source and output data
products on the local system; including the usage of the host's
'/dev/shm' to speed up temporary operations).
- Until now, the convention in Maneage was to put eight SPACE characters
before the comment lines within recipes. This was done because by
default GNU Emacs (also many other editors) show a TAB as eight
characters. However, in other text editors, online browsers, or even
the Git diff, a TAB can correspond to a different number of
characters. In such cases, the Maneage recipes wouldn't look too
interesting (the comments and the recipe commands would show a
different indentation!).
With this commit, all the comment lines in the Makefiles within the
core Maneage branch have a hash ('#') as their first character and a
TAB as the second. This allows the comment lines in recipes to have the
same indentation as code; making the code much more easier to read in a
general scenario including a 'git diff' (editor agnostic!).
P.S. List of updated software with their old and new versions
- Software with no version update are not mentioned.
- The old version of newly added software are shown with '--'.
Name (Basic) Old version New version
------------ ----------- -----------
Bzip2 1.0.6 1.0.8
CURL 7.71.1 7.79.1
Dash 0.5.10.2 0.5.11.5
File 5.39 5.41
Flock 0.2.3 0.4.0
GNU Bash 5.0.18 5.1.8
GNU Binutils 2.35 2.37
GNU Coreutils 8.32 9.0
GNU GCC 10.2.0 11.2.0
GNU M4 1.4.18 1.4.19
GNU Readline 8.0 8.1.1
GNU Tar 1.32 1.34
GNU Texinfo 6.7 6.8
GNU diffutils 3.7 3.8
GNU findutils 4.7.0 4.8.0
GNU gmp 6.2.0 6.2.1
GNU grep 3.4 3.7
GNU gzip 1.10 1.11
GNU libunistring 0.9.10 1.0
GNU mpc 1.1.0 1.2.1
GNU mpfr 4.0.2 4.1.0
GNU nano 5.2 6.0
GNU ncurses 6.2 6.3
GNU wget 1.20.3 1.21.2
Git 2.28.0 2.34.0
Less 563 590
Libxml2 2.9.9 2.9.12
Lzip 1.22-rc2 1.22
OpenSLL 1.1.1a 3.0.0
Patchelf 0.10 0.13
Perl 5.32.0 5.34.0
Podlators -- 4.14
Name (Highlevel) Old version New version
---------------- ----------- -----------
Apachelog4cxx 0.10.0-603 0.12.1
Astrometry.net 0.80 0.85
Boost 1.73.0 1.77.0
CFITSIO 3.48 4.0.0
Cmake 3.18.1 3.21.4
Eigen 3.3.7 3.4.0
Expat 2.2.9 2.4.1
FFTW 3.3.8 3.3.10
Flex 2.6.4-91 2.6.4
Fontconfig 2.13.1 2.13.94
Freetype 2.10.2 2.11.0
GNU Astronomy Utilities 0.12 0.16.1-e0f1
GNU Autoconf 2.69.200-babc 2.71
GNU Automake 1.16.2 1.16.5
GNU Bison 3.7 3.8.2
GNU Emacs 27.1 27.2
GNU GDB 9.2 11.1
GNU GSL 2.6 2.7
GNU Help2man 1.47.11 1.48.5
Ghostscript 9.52 9.55.0
ICU -- 70.1
ImageMagick 7.0.8-67 7.1.0-13
Libbsd 0.10.0 0.11.3
Libffi 3.2.1 3.4.2
Libgit2 1.0.1 1.3.0
Libidn 1.36 1.38
Libjpeg 9b 9d
Libmd -- 1.0.4
Libtiff 4.0.10 4.3.0
Libx11 1.6.9 1.7.2
Libxt 1.2.0 1.2.1
Netpbm 10.86.99 10.73.38
OpenBLAS 0.3.10 0.3.18
OpenMPI 4.0.4 4.1.1
Pixman 0.38.0 0.40.0
Python 3.8.5 3.10.0
R 4.0.2 4.1.2
SWIG 3.0.12 4.0.2
Util-linux 2.35 2.37.2
Util-macros 1.19.2 1.19.3
Valgrind 3.15.0 3.18.1
WCSLIB 7.3 7.7
Xcb-proto 1.14 1.14.1
Xorgproto 2020.1 2021.5
Name (Python) Old version New version
------------- ----------- -----------
Astropy 4.0 5.0
Beautifulsoup4 4.7.1 4.10.0
Beniget -- 0.4.1
Cffi 1.12.2 1.15.0
Cryptography 2.6.1 36.0.1
Cycler 0.10.0 0.11.0+}
Cython 0.29.21 0.29.24
Esutil 0.6.4 0.6.9
Extension-helpers -- 0.1
Galsim 2.2.1 2.3.3
Gast -- 0.5.3
Jinja2 -- 3.0.3
MPI4py 3.0.3 3.1.3
Markupsafe -- 2.0.1
Numpy 1.19.1 1.21.3
Packaging -- 21.3
Pillow -- 8.4.0
Ply -- 3.11
Pyerfa -- 2.0.0.1
Pyparsing 2.3.1 3.0.4
Pythran -- 0.11.0
Scipy 1.5.2 1.7.3
Setuptools 41.6.0 58.3.0
Six 1.12.0 1.16.0
Uncertainties 3.1.2 3.1.6
Wheel -- 0.37.0
Name (R) Old version New version
-------- ----------- -----------
Cli -- 2.5.0
Colorspace -- 2.0-1
Cowplot -- 1.1.1
Crayon -- 1.4.1
Digest -- 0.6.27
Ellipsis -- 0.3.2
Fansi -- 0.5.0
Farver -- 2.1.0
Ggplot2 -- 3.3.4
Glue -- 1.4.2
GridExtra -- 2.3
Gtable -- 0.3.0
Isoband -- 0.2.4
Labeling -- 0.4.2
Lifecycle -- 1.0.0
Magrittr -- 2.0.1
MASS -- 7.3-54
Mgcv -- 1.8-36
Munsell -- 0.5.0
Pillar -- 1.6.1
R-Pkgconfig -- 2.0.3
R6 -- 2.5.0
RColorBrewer -- 1.1-2
Rlang -- 0.4.11
Scales -- 1.1.1
Tibble -- 3.1.2
Utf8 -- 1.2.1
Vctrs -- 0.3.8
ViridisLite -- 0.4.0
Withr -- 2.4.2
|
|
This commit provides a hack/correction to the unwrapped GCC source files
that sym-links the generic file 'libgcc/unwind-generic.h' to the two
directories in which a file includes "unwind.h" or <unwind.h>. The aim is
that the gcc compilation system uses this header file from the internal gcc
source files instead of searching for a system-level file 'unwind.h'.
This commit also unaliases two 'ls' commands in some build recipes of
'basic.mk' in case the host system (normally at user level) has aliased the
command to something like 'ls -F'. In the situation that sometimes occurs
of library files being given executable status, the '-F' decorative option
could lead to an asterisk being included in a string that is not expected
to contain asterisks. If the system shell does not contain the 'alias'
command at all, then a fallback of 'true' should provide safe
behaviour. The notation of the 'sed' command is also clarified.
This solves bug #61240: https://savannah.nongnu.org/bugs/index.php?61240
|
|
Until now, the 'RPATH' variable (specifying where to look for shared
libraries) wasn't being set in the 'libcrypto' library of OpenSSL (it was
only set for the 'libssl' library).
Also, Gettext used the host Emacs for some operations during installation
that could cause the following crash (because we are giving priority to
local libraries, which the host Emacs doesn't recognize):
emacs: /BDIR/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found
(required by /lib64/libk5crypto.so.3)
With this commit both these bugs have been fixed: 1) Patchelf is run on the
'libcrypto' library also and 2) we pass the '--without-emacs' configuration
option to the configure script of Gettext.
These bugs were found by Elham Saremi.
|
|
On systems that allow it (like GNU/Linux systems), Maneage will build the
necessary software in shared memory (a directory that is actually in the
RAM, not on an SSD/HDD, on GNU/Linux systems, it is '/dev/shm'). This
allows Maneage to operate faster and not harm the HDD/SSD with all the
temporary writing of many small files.
Until now, we would only check that this directory exists and that it has
enough space. However, some systems also set the 'noexec' flag on shared
memory for security reasons [1]. This causes Maneage to crash upon building
of the software in later phases.
With this commit, at the very start of the configuration step, and after
all other shared-memory checks are done, a dummy executable script file is
created there and its execution is tested. If it doesn't work, shared
memory will not be used at all.
In the process, the steps dealing with the software building directory in
the configure script have been brought in one place and comments were added
to further clarify every step.
This commit was initially done by Boud Roukema and later edited by Mohammad
Akhlaghi.
[1] https://web.archive.org/web/20210624192819/https://serverfault.com/questions/72356/how-useful-is-mounting-tmp-noexec
|
|
Once a year, the texlive update system becomes incompatible with the
version from the previous year. Since a texlive install failure is
considered non-fatal by 'high-level.mk', so until now, the user could miss
the printed message and mistakenly believe that the configure is valid.
This commit explicitly adds a 10-second delay that should be enough for a
user who does the 'configure --existing-conf' step alone to notice that
there is a TeX Live problem. It also adds the explicit instruction of how
to allow an update from an earlier year's texlive installer to the warning
message (by deleting '.build/software/tarballs/install-tl-unx.tar.gz'). I
had to rediscover this a few times for old Maneage installs.
Also, a few lines in 'reproduce/software/shell/configure.sh' were indented
with a TAB (that is not recommended because TAB is displayed with different
widths on different browsers). So while doing this commit, those TABs were
also converted to a space.
|
|
Some minor conflicts (all expected from the commit messages in the Maneage
branch) occurred but were easily fixed.
|
|
When built in 'group' mode, the write permissions of all created files will
be activated for a certain group of users in the host operating system. The
user specifies the name of the group with the '--group' option at configure
time. At the very start, the './project' script checks to see if the given
group name actually exists or not (to avoid hard-to-debug errors popping up
later).
Until now, the checking 'sg' command (that was used to build the project
with group-writable permissions) would always fail due to the excessive
number of redirections. Therefore, it would always print the error message
and abort.
With this commit, the output of 'sg' is no longer re-directed (which also
helps users in debuggin). If the group does actually exist, it will just
print a small statement saying so, and if it fails, the error message is
printed. This fixed the problem, allowing maneage to be built in
group-mode.
I also noticed that the variable name keeping the group name
('reproducible_paper_group_name') used the old name for the project (which
was "Reproducible paper template"! So it has been changed/corrected to
'maneage_group_name'.
|
|
Until now the SWIG software would use the host operating system's packages
to find the TCL configuraiton (which we don't install yet in Maneage). In
particular, you can see the error during its configuration here:
....
checking for pkg-config... pkg-config
checking for Tcl configuration... found /usr/lib/tclConfig.sh
/usr/lib/tclConfig.sh: line 2: dpkg-architecture: command not found
/usr/lib//tcl8.6/tclConfig.sh: line 2: dpkg-architecture: com. not found
With this commit, TCL has been disabled when building SWIG with the
'--without-tcl' option. Later, when we add TCL in Maneage, we can remove
this option.
|
|
With a recent update of macOS systems (macOS Big Sur 11.2.3 and Xcode
12.4), there are many warnings when building C programs (for example the
simple program we compile to check the compiler, or some of the software
like `gzip'). It prints hundreds of warning lines for every source file
that are irrelevant for our builds, but really clutters the output.
With this commit, these warnings are disabled by adding
`-Wno-nullability-completeness' to the 'CPPFLAGS' environment
variable. This has also been added to the very first check of the C
compiler in the configure step.
|
|
Until now, each time there was a problem in the configuration of Maneage'd
projects and debugging was necessary, we had to take the following changes:
- Run the configuration on a single thread ('-j1') to see the building of
only the problematic software.
- Disable the Zenodo check manually by commenting those parts of
'reproduce/software/shell/configure.sh'. Because the internet connection
wastes a few seconds and is thus very annoying during repeated runs!
- Manually remove the '-k' option that was passed to Make (when building
the software). With the '-k', Make keeps going with the execution of
other targets if something crashes and this usually causes confusions
during the debugging.
Doing the manual changes within the code was both very annoying and prone
to errors (forgetting to correct it!).
With this commit, the existing '--debug' option has been generalized to the
software configuration phase of Maneage also. Until now, it was only
available in the analysis phase (and would directly be passed to the 'make'
command that would run the analysis). When this option is used, and the
project is in the software configuration phase, the Zenodo check won't be
done, it will use one single thread ('-j1'), and it will stop the execution
as soon as an error occurs (Make is not run with '-k').
|
|
Until now when making a link to the system's 'dl' and 'pthread' libraries
we were simply linking the installed location on the system (in
'/usr/lib'). However, in some systems, these may themselves be links to
other locations and this could cause linking problems.
With this commit, we now use 'realpath' to extract the absolute address of
the final file that the libraries may link to, and directly link to them.
A minor cosmetic correction was also made in the build rule for CFITSIO:
the long line was broken into two!
|
|
There was a single conflict in the comments of one part of 'configure.sh'
that has been fixed.
There was also a single place that needed to convert 'BDIR' to 'badir' in
this project (so after the merge, it also built easily).
|
|
Until now, the build directory contained a 'software/' directory (that
hosted all the built software), a 'tex/' subdirectory for the final
building of the paper, and many other directories containing
intermediate/final data of the specific project. But this mixing of built
software and data is against our modularity and minimal complexity
principles: built software and built data are separate things and keeping
them separate will enable many optimizations.
With this commit, the build directory of the core Maneage branch will only
contain two sub-directories: 'software/' and 'analysis/'. The 'software/'
directory has the same contents as before and is not touched in this
commit. However, the 'analysis/' directory is new and everything created in
the './project make' phase of the project will be created inside of this
directory.
To facilitate easy access to these top-level built directories, two new
variables are defined at the top of 'initialize.mk': 'badir', which is
short for "built-analysis directory" and 'bsdir', which is short for
"built-software directory".
HOW TO IMPLEMENT THIS CHANGE IN YOUR PROJECT. It is easy: simply replace
all occurances of '$(BDIR)' in your project's subMakefiles (except the ones
below) to '$(badir)'. To confirm if everything is fine before building your
project from scratch after merging, you can run the following command to
see where 'BDIR' is used and confirm the only remaning cases.
$ grep -r BDIR reproduce/analysis/*
--> make/verify.mk: innobdir=$$(echo $$infile | sed -e's|$(BDIR)/||g'); \
--> make/initialize.mk:badir=$(BDIR)/analysis
--> make/initialize.mk:bsdir=$(BDIR)/software
--> make/initialize.mk: $$sys_rm -rf $(BDIR)
--> make/top-prepare.mk:all: $(BDIR)/software/preparation-done.mk
'BDIR' should only be present in lines of the files above. If you see
'$(BDIR)' used anywhere else, simply change it to '$(badir)'. Ofcourse, if
your project assumes BDIR in other contexts, feel free to keep it, it will
not conflict. If anything un-expected happens, please post a comment on the
link below (you need to be registered on Savannah to post a comment):
https://savannah.nongnu.org/task/?15855
One consequence of this change is that the 'analysis/' subdirectory can be
optionally mounted on a separate partition. The need for this actually came
up for some new users of Maneage in a Docker image. Docker can fix
portability problems on systems that we haven't yet supported (even
Windows!), or had a chance to fix low-level issues on. However, Docker
doesn't have a GUI interface. So to see the built PDF or intermediate data,
it was necessary to copy the built data to the host system after every
change, which is annoying during working on a project. It would also need
two copies of the source: one in the host, one in the container. All these
frustrations can be fixed with this new feature.
To describe this scenario, README.md now has a new section titled "Only
software environment in the Docker image". It explains step-by-step how you
can make a Docker image to only host the built software environment. While
your project's source, software tarballs and 'BDIR/analysis' directories
are on your host operating system. It has been tested before this commit
and works very nicely.
|
|
Until now, when building GNU Binutils on GNU Linux operating systems, we
would simply put a link to the host's core C library components (the
'*crt*' files). However, the symbolic link wasn't "forced"! So if it
already existed in the build directory, it would crash.
With this commit a '-f' option has been added to the 'ln' command and this
fixed the problem.
This bug was reported by Zahra Sharbaf.
|
|
There weren't any conflicts in this merge; either technical conflicts that
can be found by Git, or logical conflicts (that will cause a crash in the
project).
|
|
After correctly setting Less to depend on 'ncurses', I noticed its still
not linking to Maneage's 'ncurses', but pointing to my host system's
'ncurses' (that happens to have the same version! So it would crash on a
system with a different version). This shows that like some other software,
we need to manually correct the RPATH inside Less.
With this command, the necessary call to 'patchelf' has been added and with
it, the installed 'less' command properly linked to Maneage's internal
build of 'ncurses'.
|
|
Until now, the 'less' software package (used to view large files easily on
the command-line and used by Git for things like 'git diff' or 'git log')
only depended on 'patchelf' (which is a very low-level software).
However, as Boud reported in bug #59811 [1], building less would crash with
an error saying "Cannot find terminal libraries" in some systems (including
the proposed Docker image of 'README.md' which I confirmed
afterwards). Looking into the 'configure' script of 'less', I noticed that
'less' is actually just checking for some functions provided by the ncurses
library!
With this commit, 'less' depends on 'ncurses'. I was able to confirm that
with this change, 'less' successfully builds within the Docker image.
[1] https://savannah.nongnu.org/bugs/?59811
|
|
Having entered 2021, it was necessary to update the years of all the
copyright statements.
|
|
There were only three very small conflicts that have been fixed.
|
|
Having entered 2021, it was necessary to update the copyright years at the
top of the source files. We recommend that you do this for all your
project-specific source files also.
|
|
Until now, when building the high-level (optional) software, we would give
both 'CPPFLAGS' and 'C_INCLUDE_PATH' the same value/directory in
'high-level.mk'. But we recently found that on macOS's C compiler
('clang'), if a directory is included in both 'CPPFLAGS' and
'C_INCLUDE_PATH', then that directory is ignored in 'CPPFLAGS' (which has
higher priority). This caused linking problems when the version of a
software on the host was different from the Maneage version.
With this commit, 'C_INCLUDE_PATH' is not set on macOS any more and this
fixed the problem on the reported systems.
This bug was fixed with the help of Mohammad Akhlaghi and Mahdieh Navabi.
|
|
Less is rarely used in non-interactive mode and is primarily intended for
interactively viewing large files. So its need within Maneage (for batch
processing) wasn't often felt until now. However, when running './project
shell' (which completely closes-off the outside environment), or building a
Maneage'd project within a minimal container that doesn't have less, it
becomes hard to use Git (and in particular its 'diff' output which depends
on 'less').
With this commit, Less has been added as a dependency of Git in
'basic.mk'. In total its built product is roughly 800KB and builds within a
second or two. So it isn't a burden on any project. But it can be very
useful when the projects are being developed within the Maneage environment
itself.
|
|
In a recent build on a macOS, we recognized that Texinfo needs the
'libintl.h' headers of Gettext. However, Gettext depends on M4, and until
now we had set M4 to depend on Texinfo. Therefore adding Gettext as a
dependency of Texinfo would cause a circular dependency.
On the macOS, we temporarily disabled M4's Texinfo dependency, and the
build went through. I also checked on my GNU/Linux system: temporarily
renamed all Texinfo built files from my system and done a clean build of M4
and it succeeded. To be further safe, I built Maneage from this commit
(where M4 doesn't depend on Texinfo) in a Docker container, and it went
through with no problems. So the current M4 version indeed doesn't need
Texinfo. I think adding Texinfo as a dependency of M4 was a historic issue
from the early days.
In the process, I also cleaned 'basic.mk' a little:
- A "# Level N" comment was added on top of each group of software that
can be built in parallel (generally).
- GNU Nano was moved to the end of the file (to be "Level 6").
- Some comments were edited in some places.
|
|
Some minor conflicts that came up during the merge were fixed.
|
|
Given the referee reports, after discussing with the editors of CiSE, we
decided that it is important to include the complete appendix we had before
that included a thorough review of existing tools and methods. However, the
appendix will not be published in the paper (due to the strict word-count
limit). It will only be used in the arXiv/Zenodo versions of the paper.
This actually created a technical problem: we want the commit hash of the
project source to remain the same when the paper is built with an appendix
or without it.
To fix this problem the choice of including an appendix has gone into the
'project' script as a run-time option called '--no-appendix'. So by default
(when someone just runs './project make'), the PDF will have an appendix,
but when we want to submit to the journal, or when the appendix isn't
needed for a certain reason, we can use this new option. The appendix also
has its own separate bibliography.
Some other corrections made in this commit:
1. Some new references were added that had an '_' in their source, they
were corrected in 'references.tex'.
2. I noticed that 'preamble-style.tex' is not actually used in this paper,
so it has been deleted.
|
|
After a fresh build of Maneage with a newly downloaded TeXLive, I noticed
that it is complaining about not finding 'xstring.sty', apparently some
package that depeneded on it is no longer including it itself!
It is thus now added to the packages that are built by Maneage's TeXLive.
|
|
Until now, the core Maneage branch included some configuration files for
Gnuastro's programs. This was actually a remnant of the distant past when
Maneage didn't actually build its own software and we had to rely on the
host's software versions. This file contained the configuration files
specific to Gnuastro for this project and also had a feature to avoid
checking the host's own configuration files.
However, we now build all our software ourselves with fixed configuration
files (for the version that is being installed and its version is
stored). So those extra configuration files were just extra and caused
confusion and problems in some scenarios. With this commit, those extra
files are now removed.
Also, two small issues are also addressed in parallel with this commit:
- When running './project make clean', the 'hardware-parameters.tex' macro
file (which is created by './project configure' is not deleted.
- The project title is now written into the default output's PDF's
properties (through 'hypersetup' in 'tex/src/preamble-header.tex')
through the LaTeX macro.
All these issues were found and fixed with the help of Samane Raji.
|
|
Until now, during the configure step it was checked if the host Operative
System were GNU/Linux, and if not, we assumed it is macOS. However, it can
be any other different OS! With this commit, now we explicity check if the
system is GNU/Linux or Darwin (macOS). If it is not any of them, a warning
message says to the user that the host system is different from which we
have checked so far (and invite to contact us if there is any problem).
In addition to this, if the system is macOS, now it checks if Xcode is
already installed in the host system. If it is not installed, a warning
message informs the user to do that in case a problem/crash in the
configure step occurs. We have found that it is convenient to have Xcode
installed in order to avoid some problems.
|
|
There weren't any conflicts in this merge.
|
|
Tcl/Tk are a set of tools to provide Graphic User Interface (GUI) support
in some software. But they are not yet natively built within Maneage,
primarily because we have higher-priority work right now. GUI tools in
general aren't high on our priority list right now because GUI tools are
generally good for human interaction (which is contrary to the reproducible
philosophy), not automatic analysis (a core concept in reproducibility). So
even later, when we do include Tcl/Tk in Maneage, their direct usage will
be discouraged.
Until this commit, because we don't yet build Tcl/Tk, the default maneage
install of the statistical package R failed on a Debian Stretch, with 6227
repeats of the line:
'/usr/lib//tcl8.5/tclConfig.sh: line 2: dpkg-architecture:
command not found'
To fix this problem (atleast until Tcl/Tk is installed within Maneage), R
is now configured with the '--without-tcltk' option which fixed the
problem. Please see the description above the R installation instructions
in 'reproduce/software/make/high-level.mk' for more.
|
|
Following the previous commit, we recognized that the 'IFS' terms are not
necessary and can be even cause problems. So all their occurances in the
scripts of Maneage have been removed with this commit.
|
|
Until a recent commit, the IFS='"' was added at the start of the variables
in this shell script and as a result, the SPACE character wasn't being used
as a delimiter. This caused a major problem when downloading the tarballs
(all the backup servers were considered as the top link).
With this commit we removed these 'IFS' statements). Because we now check
for the existance of meta-characters in the build directory name, there is
no more problem, and also generally both the calling command and
internally, we have double-qutations around the variable names. So removal
of IFS will not affect the result in this scenario.
This bug was found by Mohammadreza Khellat.
|
|
Only two small conflicts came up:
* The addition of the hardware architecture macro in 'paper.tex' (which
was removed for now, but will be added as the referee has requested
within the text).
* The usage of "" around directory variables in 'paper.mk'.
|
|
While a project is under development, the raw analysis software are not the
only necessary software in a project. We also need tools to all the edit
plain-text files within the Maneaged project. Usually people use their
operating system's plain-text editor. However, when working on the project
on a new computer, or in a container, the plain-text editors will have
different versions, or may not be present at all! This can be very annoying
and frustrating!
With this commit, Maneage now installs GNU Nano as part of the basic
tools. GNU Nano is a very simple and small plain text editor (the installed
size is only ~3.5MB, and it is friendly to new users). Therefore, any
Maneaged project can assume atleast Nano will be present (in particular
when no editor is available on the running system!). GNU Emacs and VIM
(both without extra dependencies, in particular without GUI support) are
also optionally available in 'high-level.mk' (by adding them to
'TARGETS.conf').
The basic idea for the more advanced editors (Emacs and VIM) is that
project authors can add their favorite editor while they are working on the
project, but upon publication they can remove them from 'TARGETS.conf'.
A few other minor things came up during this work and are now also fixed:
- The 'file' program and its libraries like 'libmagic' were linking to
system's 'libseccomp'! This dependency then leaked into Nano (which
depends on 'libmagic'). But this is just an extra feature of 'file',
only for the Linux kernel. Also, we have no dependency on it so far. So
'file' is not configured to not build with 'libseccomp'.
- A typo was fixed in the line where the physical core information is
being read on macOS.
- The top-level directories when running './project shell' are now quoted
(in case they have special characters).
|
|
Until now, no machine-related specifications were being documented in the
workflow. This information can become helpful when observing differences in
the outcome of both software and analysis segments of the workflow by
others (some software may behave differently based on host machine).
With this commit, the host machine's 'hardware class' and 'byte-order' are
collected and now available as LaTeX macros for the authors to use in the
paper. Currently it is placed in the acknowledgments, right after
mentioning the Maneage commit.
Furthermore, the project and configuration scripts are now capable of
dealing with input directory names that have SPACE (and other special
characters) by putting them inside double-quotes. However, having spaces
and metacharacters in the address of the build directory could cause
build/install failure for some software source files which are beyond the
control of Maneage. So we now check the user's given build directory
string, and if the string has any '@', '#', '$', '%', '^', '&', '*', '(',
')', '+', ';', and ' ' (SPACE), it will ask the user to provide a different
directory.
|
|
Some very minor conflicts came up and were easily corrected. They were
mostly in parts that are also shared with the demonstration in the core
Maneage branch.
|
|
Until now, if the software source tarballs already existed on the system
they would be copied inside the project. However, the software source
tarballs are sometimes/mostly larger than their actual product and can
consume significant space (~375 MB in the core branch!).
With this commit, when the software are present on the system, their
symbolic link will be placed in 'BDIR/software/tarballs', not a full
copy. Also, because the tarballs in software tarball directory may
themselves be links, we use 'realpath' to find the final place of the
actual file and link to that location. Therefore if 'realpath' can't be
found (prior to installing Coreutils in Maneage), we will copy the tarballs
from the given software tarball directory. After Maneage has installed
Coreutils, the project's own 'realpath' will be used. Of course, if the
software are downloaded, their full downloaded copy will be kept in
'BDIR/software/tarballs', nothing has changed in the downloading scenario.
|
|
It was a long time that the Maneage software versions hadn't been updated.
With this commit, the versions of all basic software were checked and 17 of
that had newer versions were updated. Also, 16 high-level programs and
libraries were updated as well as 7 Python modules. The full list is
available below.
Basic Software (affecting all projects)
---------------------------------------
bash 5.0.11 -> 5.0.18
binutils 2.32 -> 2.35
coreutils 8.31 -> 8.32
curl 7.65.3 -> 7.71.1
file 5.36 -> 5.39
gawk 5.0.1 -> 5.1.0
gcc 9.2.0 -> 10.2.0
gettext 0.20.2 -> 0.21
git 2.26.2 -> 2.28.0
gmp 6.1.2 -> 6.2.0
grep 3.3 -> 3.4
libbsd 0.9.1 -> 0.10.0
ncurses 6.1 -> 6.2
perl 5.30.0 -> 5.32.0
sed 4.7 -> 4.8
texinfo 6.6 -> 6.7
xz 5.2.4 -> 5.2.5
Custom programs/libraries
-------------------------
astrometrynet 0.77 -> 0.80
automake 0.16.1 -> 0.16.2
bison 3.6 -> 3.7
cfitsio 3.47 -> 3.48
cmake 3.17.0 -> 3.18.1
freetype 2.9 -> 2.10.2
gdb 8.3 -> 9.2
ghostscript 9.50 -> 9.52
gnuastro 0.11 -> 0.12
libgit2 0.28.2 -> 1.0.1
libidn 1.35 -> 1.36
openmpi 4.0.1 -> 4.0.4
R 3.6.2 -> 4.0.2
python 3.7.4 -> 3.8.5
wcslib 6.4 -> 7.3
yaml 0.2.2 -> 0.2.5
Python modules
--------------
cython 0.29.6 -> 0.29.21
h5py 2.9.0 -> 2.10.0
matplotlib 3.1.1 -> 3.3.0
mpi4py 3.0.2 -> 3.0.3
numpy 1.17.2 -> 1.19.1
pybind11 2.4.3 -> 2.5.0
scipy 1.3.1 -> 1.5.2
|
|
When the host C compiler is used (either by calling '--host-cc' or on OSs
that we can't build the GNU C Compiler), Maneage will also not build the
Fortran compiler 'gfortran'. Until now, the './project configure' script
would give a big warning about the need for 'gfortran' and the fact that it
is missing, and would for 5 seconds, but it would continue anyway.
For projects that don't need 'gfortran', this can be confusing to the users
and for those that need 'gfortran', it means that a lot of time and cpu
cycles are wasted compiling non-fortran software that are unusable in the
end.
With this commit, the 'need_gfortarn' variable has been added
'reproduce/software/shell/configure.sh', in a new part that is devoted to
project-specific features. If it equals '0', then the 'gfortran' test (and
message!) isn't done at all, but if it is set to '1', then the configure
stage will halt immediately gfortran is not found and not built.
The default operations of the core Maneage branch don't need 'gfortran', so
by default it is set to 0. But 'gfortran' is necessary for all projects
that use Numpy (Python's numeric library) for example. So if your project
needs 'gfortran', please set this new variable to 1. As mentioned in the
comments of 'configure.sh', ideally we should detect this automatically,
but we haven't had the time to implement it yet.
|
|
Prior to this commit, compilation of OpenMPI used the default OpenMPI
choices of deciding which libraries should be used in relating to a job
scheduler [1] (such as Slurm [2]). Given that the user on a multi-user
cluster has to accept the sysadmin's choice of a job scheduler, the
question of whether to (1) link with OpenMPI's own libraries (and increase
the reproducibility of the science project) or rather (2) link with the
sysadmin managed libraries (more likely to be compatible with the host's
job scheduler), is an open question of which the best strategy for
reproducibility needs to be debated and studied.
In this commit, strategy (1) is adopted. The options '--withpmix=internal'
and '--with-hwloc=internal' are added to the configure command. The working
assumption is that the Maneage version of OpenMPI is likely to be modern
enough to be compatible with the native job scheduler such as
Slurm. Compilation without any 'pmix' option gave a fail in at least one
case; it appears that an external pmix library was sought by the configure
script.
As of OpenMPI 4.0.1, the internal libevent library is used by default, so
there appears to be no option to force it to be chosen internally.
This commit also includes the option '--without-verbs'. This option
removes a library related to "infiniband", "verbs", "openib" and "BTL";
this library appears to be deprecated. See [3], [4] for discussion.
Please add feedback and discussion to the Maneage task about openmpi
linking strategies (1) (internal) and (2) (external) at Savannah [5].
[1] https://en.wikipedia.org/wiki/Job_scheduler#Batch_queuing_for_HPC_clusters
[2] https://en.wikipedia.org/wiki/Slurm_Workload_Manager - To avoid a name
clash, 'slurm-wlm' is the metapackage in Debian for the client
commands, the compute node daemon, and the central node daemon. An
unrelated package 'slurm' also exists.
[3] https://www-lb.open-mpi.org/faq/?category=openfabrics#ofa-device-error
[4] https://www-lb.open-mpi.org/faq/?category=building
[5] https://savannah.nongnu.org/task/index.php?15737
|
|
Until now, if a project needed the healpy software package, Maneage would
crash with the following error message (abridged for full name in build
directory). This was caused by a typo in the version of 'healpix' (the
dependency of 'healpy').
make: *** No rule to make target '.../version-info/proglib/healpix-'
With this commit, the typo in line 334 of 'python.mk' is fixed, so that
when '$(ipydir)/healpy-$(healpy-version)' gets called it correctly searches
for a rule to make '$(ibidir)/healpix-$(healpix-version)'.
|
|
In the previous commit (Commit 1bc00c9: Only using clang in macOS systems
that also have GCC) we set the used C compiler for high-level programs to
be 'clang' on macOS systems. But I forgot to do the same kind of change in
the configure script (to prefer 'clang' when we are testing for a C
compiler on the host).
With this commit, the compiler checking phases of the configure script have
been improved, so on macOS systems, we now first search for 'clang', then
search for 'gcc'.
While doing this, I also noticed that the 'rpath' checking command was done
before we actually define 'instdir'!!! So in effect, the 'rpath' directory
was being set to '/lib'! So with this commit, this test has been taken to
after defining 'instdir'.
|
|
Until now, when Maneage was built on a macOS that had both a clang and GCC,
we would make links to both. But this cause many conflicts in some
high-level programs (for example Numpy and etc, all the programs where we
have explicity set 'export CC=clang' before the build recipe). This happens
because the GCC that is built on a macOS isn't complete for some
operations.
To fix this problem, when we are on a macOS, we explicity set 'gcc' to
point to 'clang' and 'g++' to point to 'clang++'. We also don't link to the
host's C-preprocessor ('cpp') on macOS systems because this is only a GNU
feature and using the GNU CPP is also known to have some basic
problems. For example this was reported by Mahdieh Nabavi (which was the
main trigger for this work):
ld: Symbol not found: ___keymgr_global
Referenced from: /Users/Mahdieh/build/software/installed/bin/cpp
Expected in: /usr/lib/libSystem.B.dylib
Also, to avoid linking to another link on the host tools (in the 'makelink'
function of 'basic.mk'), we are now using 'realpath'.
|
|
Until now, when reading the host's PATH environment variable we weren't
accounting for directory names with a space character. This was most
prominently visible in the 'low-level-links' step where we put links to
some core system components into the project's build directory (mainly for
prorietary systems like macOS).
To address the problem, double quotations have been placed around the part
that we extract 'ccache' from the PATH, and the part where we make the
symbolic link. In the process the comments above 'makelink' were made more
clear and 'low-level-links' now depends on 'grep' (which is the
highest-level program it uses).
This bug was reported by Mahdieh Navabi.
|
|
Until this commit, once Libidn was installed, insted of its own name and
version, the name and version of Libjpeg were saved (in the target if
Libidn). This robably come from a copy/paste of the rule.
With this commit, this minor bug has been corrected. I also added my name
as an author of `reproduce/software/make/xorg.mk' Makefile since I added
some code there.
|
|
There weren't any conflicts in this merge.
|
|
After recently adding util-linux to Maneage build-tree, we had forgot to
delete the unpacked and built source directory after it was installed! This
has been corrected with this commit.
|
|
Until now, when the user specified an input and software directory, the raw
string they entered was used. But when this string was a relative location,
this could be problematic in general scenarios.
With this commit, the same function that finds the absolute location of the
build directory is used to find the absolute address of the data and
software directories.
|
