| Age | Commit message (Collapse) | Author | Lines |
|---|
|
This commit primarily affects the configuration step of Maneage'd projects,
and in particular, updated versions of the many of the software (see
P.S.). So it shouldn't affect your high-level analysis other than the
version bumps of the software you use (and the software's possibly
improve/changed behavior).
The following software (and thus their dependencies) couldn't be updated as
described below:
- Cryptography: isn't building because it depends on a new
setuptools-rust package that has problems
(https://savannah.nongnu.org/bugs/index.php?61731), so it has been
commented in 'versions.conf'.
- SecretStorage: because it depends on Cryptography.
- Keyring: because it depends on SecretStorage.
- Astroquery: because it depends on Keyring.
This is a "squashed" commit after rebasing a development branch of 60
commits corresponding to a roughly two-month time interval. The following
people contributed to this branch.
- Boudewijn Roukema added all the R software infrastructure and the R
packages, as well as greatly helping in fixing many bugs during the
update.
- Raul Infante-Sainz helped in testing and debugging the build.
- Pedram Ashofteh Ardakani found and fixed a bug.
- Zahra Sharbaf helped in testing and found several bugs.
Below a description of the most noteworthy points is given.
- Software tarballs: all updated software now have a unified format
tarball (ustar; if not possible, pax) and unified compression (Lzip) in
Maneage's software repository in Zenodo
(https://doi.org/10.5281/zenodo.3883409). For more on this See
https://savannah.nongnu.org/task/?15699 . This won't affect any extra
software you would like to add; you can use any format recognized by
GNU Tar, and all common compression algorithms. This new requirement is
only for software that get merged to the core Maneage branch.
- Metastore (and thus libbsd and libmd) moved to highlevel: Metastore
(and the packages it depends on) is a high-level product that is only
relevant during the project development (like Emacs!): when the user
wants the file meta data (like dates) to be unchanged after checking
out branches. So it should be considered a high-level software, not
basic. Metastore also usually causes many more headaches and error
messages, so personally, I have stopped using it! Instead I simply
merge my branches in a separate clone, then pull the merge commit: in
this way, the files of my project aren't re-written during the checkout
phase and therefore their dates are untouched (which can conflict with
Make's dates on configuration files).
- The un-official cloned version of Flex (2.6.4-91 until this commit) was
causing problems in the building of Netpbm, so with this commit, it has
been moved back to version 2.6.4.
- Netpbm's official page had version 10.73.38 as the latest stable
tarball that was just released in late 2021. But I couldn't find our
previously-used version 10.86.99 anywhere (to see when it was released
and why we used it! Its at last more than one year old!). So the
official stable version is being used now.
- Improved instructions in 'README.md' for building software environment
in a Docker container (while having project source and output data
products on the local system; including the usage of the host's
'/dev/shm' to speed up temporary operations).
- Until now, the convention in Maneage was to put eight SPACE characters
before the comment lines within recipes. This was done because by
default GNU Emacs (also many other editors) show a TAB as eight
characters. However, in other text editors, online browsers, or even
the Git diff, a TAB can correspond to a different number of
characters. In such cases, the Maneage recipes wouldn't look too
interesting (the comments and the recipe commands would show a
different indentation!).
With this commit, all the comment lines in the Makefiles within the
core Maneage branch have a hash ('#') as their first character and a
TAB as the second. This allows the comment lines in recipes to have the
same indentation as code; making the code much more easier to read in a
general scenario including a 'git diff' (editor agnostic!).
P.S. List of updated software with their old and new versions
- Software with no version update are not mentioned.
- The old version of newly added software are shown with '--'.
Name (Basic) Old version New version
------------ ----------- -----------
Bzip2 1.0.6 1.0.8
CURL 7.71.1 7.79.1
Dash 0.5.10.2 0.5.11.5
File 5.39 5.41
Flock 0.2.3 0.4.0
GNU Bash 5.0.18 5.1.8
GNU Binutils 2.35 2.37
GNU Coreutils 8.32 9.0
GNU GCC 10.2.0 11.2.0
GNU M4 1.4.18 1.4.19
GNU Readline 8.0 8.1.1
GNU Tar 1.32 1.34
GNU Texinfo 6.7 6.8
GNU diffutils 3.7 3.8
GNU findutils 4.7.0 4.8.0
GNU gmp 6.2.0 6.2.1
GNU grep 3.4 3.7
GNU gzip 1.10 1.11
GNU libunistring 0.9.10 1.0
GNU mpc 1.1.0 1.2.1
GNU mpfr 4.0.2 4.1.0
GNU nano 5.2 6.0
GNU ncurses 6.2 6.3
GNU wget 1.20.3 1.21.2
Git 2.28.0 2.34.0
Less 563 590
Libxml2 2.9.9 2.9.12
Lzip 1.22-rc2 1.22
OpenSLL 1.1.1a 3.0.0
Patchelf 0.10 0.13
Perl 5.32.0 5.34.0
Podlators -- 4.14
Name (Highlevel) Old version New version
---------------- ----------- -----------
Apachelog4cxx 0.10.0-603 0.12.1
Astrometry.net 0.80 0.85
Boost 1.73.0 1.77.0
CFITSIO 3.48 4.0.0
Cmake 3.18.1 3.21.4
Eigen 3.3.7 3.4.0
Expat 2.2.9 2.4.1
FFTW 3.3.8 3.3.10
Flex 2.6.4-91 2.6.4
Fontconfig 2.13.1 2.13.94
Freetype 2.10.2 2.11.0
GNU Astronomy Utilities 0.12 0.16.1-e0f1
GNU Autoconf 2.69.200-babc 2.71
GNU Automake 1.16.2 1.16.5
GNU Bison 3.7 3.8.2
GNU Emacs 27.1 27.2
GNU GDB 9.2 11.1
GNU GSL 2.6 2.7
GNU Help2man 1.47.11 1.48.5
Ghostscript 9.52 9.55.0
ICU -- 70.1
ImageMagick 7.0.8-67 7.1.0-13
Libbsd 0.10.0 0.11.3
Libffi 3.2.1 3.4.2
Libgit2 1.0.1 1.3.0
Libidn 1.36 1.38
Libjpeg 9b 9d
Libmd -- 1.0.4
Libtiff 4.0.10 4.3.0
Libx11 1.6.9 1.7.2
Libxt 1.2.0 1.2.1
Netpbm 10.86.99 10.73.38
OpenBLAS 0.3.10 0.3.18
OpenMPI 4.0.4 4.1.1
Pixman 0.38.0 0.40.0
Python 3.8.5 3.10.0
R 4.0.2 4.1.2
SWIG 3.0.12 4.0.2
Util-linux 2.35 2.37.2
Util-macros 1.19.2 1.19.3
Valgrind 3.15.0 3.18.1
WCSLIB 7.3 7.7
Xcb-proto 1.14 1.14.1
Xorgproto 2020.1 2021.5
Name (Python) Old version New version
------------- ----------- -----------
Astropy 4.0 5.0
Beautifulsoup4 4.7.1 4.10.0
Beniget -- 0.4.1
Cffi 1.12.2 1.15.0
Cryptography 2.6.1 36.0.1
Cycler 0.10.0 0.11.0+}
Cython 0.29.21 0.29.24
Esutil 0.6.4 0.6.9
Extension-helpers -- 0.1
Galsim 2.2.1 2.3.3
Gast -- 0.5.3
Jinja2 -- 3.0.3
MPI4py 3.0.3 3.1.3
Markupsafe -- 2.0.1
Numpy 1.19.1 1.21.3
Packaging -- 21.3
Pillow -- 8.4.0
Ply -- 3.11
Pyerfa -- 2.0.0.1
Pyparsing 2.3.1 3.0.4
Pythran -- 0.11.0
Scipy 1.5.2 1.7.3
Setuptools 41.6.0 58.3.0
Six 1.12.0 1.16.0
Uncertainties 3.1.2 3.1.6
Wheel -- 0.37.0
Name (R) Old version New version
-------- ----------- -----------
Cli -- 2.5.0
Colorspace -- 2.0-1
Cowplot -- 1.1.1
Crayon -- 1.4.1
Digest -- 0.6.27
Ellipsis -- 0.3.2
Fansi -- 0.5.0
Farver -- 2.1.0
Ggplot2 -- 3.3.4
Glue -- 1.4.2
GridExtra -- 2.3
Gtable -- 0.3.0
Isoband -- 0.2.4
Labeling -- 0.4.2
Lifecycle -- 1.0.0
Magrittr -- 2.0.1
MASS -- 7.3-54
Mgcv -- 1.8-36
Munsell -- 0.5.0
Pillar -- 1.6.1
R-Pkgconfig -- 2.0.3
R6 -- 2.5.0
RColorBrewer -- 1.1-2
Rlang -- 0.4.11
Scales -- 1.1.1
Tibble -- 3.1.2
Utf8 -- 1.2.1
Vctrs -- 0.3.8
ViridisLite -- 0.4.0
Withr -- 2.4.2
|
|
This commit provides a hack/correction to the unwrapped GCC source files
that sym-links the generic file 'libgcc/unwind-generic.h' to the two
directories in which a file includes "unwind.h" or <unwind.h>. The aim is
that the gcc compilation system uses this header file from the internal gcc
source files instead of searching for a system-level file 'unwind.h'.
This commit also unaliases two 'ls' commands in some build recipes of
'basic.mk' in case the host system (normally at user level) has aliased the
command to something like 'ls -F'. In the situation that sometimes occurs
of library files being given executable status, the '-F' decorative option
could lead to an asterisk being included in a string that is not expected
to contain asterisks. If the system shell does not contain the 'alias'
command at all, then a fallback of 'true' should provide safe
behaviour. The notation of the 'sed' command is also clarified.
This solves bug #61240: https://savannah.nongnu.org/bugs/index.php?61240
|
|
Until now, the 'RPATH' variable (specifying where to look for shared
libraries) wasn't being set in the 'libcrypto' library of OpenSSL (it was
only set for the 'libssl' library).
Also, Gettext used the host Emacs for some operations during installation
that could cause the following crash (because we are giving priority to
local libraries, which the host Emacs doesn't recognize):
emacs: /BDIR/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found
(required by /lib64/libk5crypto.so.3)
With this commit both these bugs have been fixed: 1) Patchelf is run on the
'libcrypto' library also and 2) we pass the '--without-emacs' configuration
option to the configure script of Gettext.
These bugs were found by Elham Saremi.
|
|
With a recent update of macOS systems (macOS Big Sur 11.2.3 and Xcode
12.4), there are many warnings when building C programs (for example the
simple program we compile to check the compiler, or some of the software
like `gzip'). It prints hundreds of warning lines for every source file
that are irrelevant for our builds, but really clutters the output.
With this commit, these warnings are disabled by adding
`-Wno-nullability-completeness' to the 'CPPFLAGS' environment
variable. This has also been added to the very first check of the C
compiler in the configure step.
|
|
Until now when making a link to the system's 'dl' and 'pthread' libraries
we were simply linking the installed location on the system (in
'/usr/lib'). However, in some systems, these may themselves be links to
other locations and this could cause linking problems.
With this commit, we now use 'realpath' to extract the absolute address of
the final file that the libraries may link to, and directly link to them.
A minor cosmetic correction was also made in the build rule for CFITSIO:
the long line was broken into two!
|
|
Until now, the build directory contained a 'software/' directory (that
hosted all the built software), a 'tex/' subdirectory for the final
building of the paper, and many other directories containing
intermediate/final data of the specific project. But this mixing of built
software and data is against our modularity and minimal complexity
principles: built software and built data are separate things and keeping
them separate will enable many optimizations.
With this commit, the build directory of the core Maneage branch will only
contain two sub-directories: 'software/' and 'analysis/'. The 'software/'
directory has the same contents as before and is not touched in this
commit. However, the 'analysis/' directory is new and everything created in
the './project make' phase of the project will be created inside of this
directory.
To facilitate easy access to these top-level built directories, two new
variables are defined at the top of 'initialize.mk': 'badir', which is
short for "built-analysis directory" and 'bsdir', which is short for
"built-software directory".
HOW TO IMPLEMENT THIS CHANGE IN YOUR PROJECT. It is easy: simply replace
all occurances of '$(BDIR)' in your project's subMakefiles (except the ones
below) to '$(badir)'. To confirm if everything is fine before building your
project from scratch after merging, you can run the following command to
see where 'BDIR' is used and confirm the only remaning cases.
$ grep -r BDIR reproduce/analysis/*
--> make/verify.mk: innobdir=$$(echo $$infile | sed -e's|$(BDIR)/||g'); \
--> make/initialize.mk:badir=$(BDIR)/analysis
--> make/initialize.mk:bsdir=$(BDIR)/software
--> make/initialize.mk: $$sys_rm -rf $(BDIR)
--> make/top-prepare.mk:all: $(BDIR)/software/preparation-done.mk
'BDIR' should only be present in lines of the files above. If you see
'$(BDIR)' used anywhere else, simply change it to '$(badir)'. Ofcourse, if
your project assumes BDIR in other contexts, feel free to keep it, it will
not conflict. If anything un-expected happens, please post a comment on the
link below (you need to be registered on Savannah to post a comment):
https://savannah.nongnu.org/task/?15855
One consequence of this change is that the 'analysis/' subdirectory can be
optionally mounted on a separate partition. The need for this actually came
up for some new users of Maneage in a Docker image. Docker can fix
portability problems on systems that we haven't yet supported (even
Windows!), or had a chance to fix low-level issues on. However, Docker
doesn't have a GUI interface. So to see the built PDF or intermediate data,
it was necessary to copy the built data to the host system after every
change, which is annoying during working on a project. It would also need
two copies of the source: one in the host, one in the container. All these
frustrations can be fixed with this new feature.
To describe this scenario, README.md now has a new section titled "Only
software environment in the Docker image". It explains step-by-step how you
can make a Docker image to only host the built software environment. While
your project's source, software tarballs and 'BDIR/analysis' directories
are on your host operating system. It has been tested before this commit
and works very nicely.
|
|
Until now, when building GNU Binutils on GNU Linux operating systems, we
would simply put a link to the host's core C library components (the
'*crt*' files). However, the symbolic link wasn't "forced"! So if it
already existed in the build directory, it would crash.
With this commit a '-f' option has been added to the 'ln' command and this
fixed the problem.
This bug was reported by Zahra Sharbaf.
|
|
After correctly setting Less to depend on 'ncurses', I noticed its still
not linking to Maneage's 'ncurses', but pointing to my host system's
'ncurses' (that happens to have the same version! So it would crash on a
system with a different version). This shows that like some other software,
we need to manually correct the RPATH inside Less.
With this command, the necessary call to 'patchelf' has been added and with
it, the installed 'less' command properly linked to Maneage's internal
build of 'ncurses'.
|
|
Until now, the 'less' software package (used to view large files easily on
the command-line and used by Git for things like 'git diff' or 'git log')
only depended on 'patchelf' (which is a very low-level software).
However, as Boud reported in bug #59811 [1], building less would crash with
an error saying "Cannot find terminal libraries" in some systems (including
the proposed Docker image of 'README.md' which I confirmed
afterwards). Looking into the 'configure' script of 'less', I noticed that
'less' is actually just checking for some functions provided by the ncurses
library!
With this commit, 'less' depends on 'ncurses'. I was able to confirm that
with this change, 'less' successfully builds within the Docker image.
[1] https://savannah.nongnu.org/bugs/?59811
|
|
Having entered 2021, it was necessary to update the copyright years at the
top of the source files. We recommend that you do this for all your
project-specific source files also.
|
|
Less is rarely used in non-interactive mode and is primarily intended for
interactively viewing large files. So its need within Maneage (for batch
processing) wasn't often felt until now. However, when running './project
shell' (which completely closes-off the outside environment), or building a
Maneage'd project within a minimal container that doesn't have less, it
becomes hard to use Git (and in particular its 'diff' output which depends
on 'less').
With this commit, Less has been added as a dependency of Git in
'basic.mk'. In total its built product is roughly 800KB and builds within a
second or two. So it isn't a burden on any project. But it can be very
useful when the projects are being developed within the Maneage environment
itself.
|
|
In a recent build on a macOS, we recognized that Texinfo needs the
'libintl.h' headers of Gettext. However, Gettext depends on M4, and until
now we had set M4 to depend on Texinfo. Therefore adding Gettext as a
dependency of Texinfo would cause a circular dependency.
On the macOS, we temporarily disabled M4's Texinfo dependency, and the
build went through. I also checked on my GNU/Linux system: temporarily
renamed all Texinfo built files from my system and done a clean build of M4
and it succeeded. To be further safe, I built Maneage from this commit
(where M4 doesn't depend on Texinfo) in a Docker container, and it went
through with no problems. So the current M4 version indeed doesn't need
Texinfo. I think adding Texinfo as a dependency of M4 was a historic issue
from the early days.
In the process, I also cleaned 'basic.mk' a little:
- A "# Level N" comment was added on top of each group of software that
can be built in parallel (generally).
- GNU Nano was moved to the end of the file (to be "Level 6").
- Some comments were edited in some places.
|
|
While a project is under development, the raw analysis software are not the
only necessary software in a project. We also need tools to all the edit
plain-text files within the Maneaged project. Usually people use their
operating system's plain-text editor. However, when working on the project
on a new computer, or in a container, the plain-text editors will have
different versions, or may not be present at all! This can be very annoying
and frustrating!
With this commit, Maneage now installs GNU Nano as part of the basic
tools. GNU Nano is a very simple and small plain text editor (the installed
size is only ~3.5MB, and it is friendly to new users). Therefore, any
Maneaged project can assume atleast Nano will be present (in particular
when no editor is available on the running system!). GNU Emacs and VIM
(both without extra dependencies, in particular without GUI support) are
also optionally available in 'high-level.mk' (by adding them to
'TARGETS.conf').
The basic idea for the more advanced editors (Emacs and VIM) is that
project authors can add their favorite editor while they are working on the
project, but upon publication they can remove them from 'TARGETS.conf'.
A few other minor things came up during this work and are now also fixed:
- The 'file' program and its libraries like 'libmagic' were linking to
system's 'libseccomp'! This dependency then leaked into Nano (which
depends on 'libmagic'). But this is just an extra feature of 'file',
only for the Linux kernel. Also, we have no dependency on it so far. So
'file' is not configured to not build with 'libseccomp'.
- A typo was fixed in the line where the physical core information is
being read on macOS.
- The top-level directories when running './project shell' are now quoted
(in case they have special characters).
|
|
It was a long time that the Maneage software versions hadn't been updated.
With this commit, the versions of all basic software were checked and 17 of
that had newer versions were updated. Also, 16 high-level programs and
libraries were updated as well as 7 Python modules. The full list is
available below.
Basic Software (affecting all projects)
---------------------------------------
bash 5.0.11 -> 5.0.18
binutils 2.32 -> 2.35
coreutils 8.31 -> 8.32
curl 7.65.3 -> 7.71.1
file 5.36 -> 5.39
gawk 5.0.1 -> 5.1.0
gcc 9.2.0 -> 10.2.0
gettext 0.20.2 -> 0.21
git 2.26.2 -> 2.28.0
gmp 6.1.2 -> 6.2.0
grep 3.3 -> 3.4
libbsd 0.9.1 -> 0.10.0
ncurses 6.1 -> 6.2
perl 5.30.0 -> 5.32.0
sed 4.7 -> 4.8
texinfo 6.6 -> 6.7
xz 5.2.4 -> 5.2.5
Custom programs/libraries
-------------------------
astrometrynet 0.77 -> 0.80
automake 0.16.1 -> 0.16.2
bison 3.6 -> 3.7
cfitsio 3.47 -> 3.48
cmake 3.17.0 -> 3.18.1
freetype 2.9 -> 2.10.2
gdb 8.3 -> 9.2
ghostscript 9.50 -> 9.52
gnuastro 0.11 -> 0.12
libgit2 0.28.2 -> 1.0.1
libidn 1.35 -> 1.36
openmpi 4.0.1 -> 4.0.4
R 3.6.2 -> 4.0.2
python 3.7.4 -> 3.8.5
wcslib 6.4 -> 7.3
yaml 0.2.2 -> 0.2.5
Python modules
--------------
cython 0.29.6 -> 0.29.21
h5py 2.9.0 -> 2.10.0
matplotlib 3.1.1 -> 3.3.0
mpi4py 3.0.2 -> 3.0.3
numpy 1.17.2 -> 1.19.1
pybind11 2.4.3 -> 2.5.0
scipy 1.3.1 -> 1.5.2
|
|
Until now, when Maneage was built on a macOS that had both a clang and GCC,
we would make links to both. But this cause many conflicts in some
high-level programs (for example Numpy and etc, all the programs where we
have explicity set 'export CC=clang' before the build recipe). This happens
because the GCC that is built on a macOS isn't complete for some
operations.
To fix this problem, when we are on a macOS, we explicity set 'gcc' to
point to 'clang' and 'g++' to point to 'clang++'. We also don't link to the
host's C-preprocessor ('cpp') on macOS systems because this is only a GNU
feature and using the GNU CPP is also known to have some basic
problems. For example this was reported by Mahdieh Nabavi (which was the
main trigger for this work):
ld: Symbol not found: ___keymgr_global
Referenced from: /Users/Mahdieh/build/software/installed/bin/cpp
Expected in: /usr/lib/libSystem.B.dylib
Also, to avoid linking to another link on the host tools (in the 'makelink'
function of 'basic.mk'), we are now using 'realpath'.
|
|
Until now, when reading the host's PATH environment variable we weren't
accounting for directory names with a space character. This was most
prominently visible in the 'low-level-links' step where we put links to
some core system components into the project's build directory (mainly for
prorietary systems like macOS).
To address the problem, double quotations have been placed around the part
that we extract 'ccache' from the PATH, and the part where we make the
symbolic link. In the process the comments above 'makelink' were made more
clear and 'low-level-links' now depends on 'grep' (which is the
highest-level program it uses).
This bug was reported by Mahdieh Navabi.
|
|
Until now, in order to build Ghostscript, the project used the host's Xorg
libraries. This was because we hadn't yet added the necessary build rules
for them.
With this commit, the instructions to build the necessary Xorg libraries
for Ghostscript have also been added. Also, the shared Ghostscript library
has been built with this commit and two sets of standard fonts are also
included, setting us on the path to build TeXLive from source later.
This task was done with the help and support of Raul Infante-Sainz.
|
|
POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause
conflicts to your project if you have changed the software building
Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and
'python.mk'). If your project has only added analysis, it shouldn't be
affected.
This is a large commit, involving a long series of corrections in a
differnt branch which is now finally being merged into the core Maneage
branch. All changes were related and came up naturally as the low-level
infrastructure was improved. So separating them in the end for the final
merge would have been very time consuming and we are merging them as one
commit.
In general, the software building Makefiles are now much more easier to
read, modify and use, along with several new features that have been
added. See below for the full list.
- Until now, Maneage needed the host to have a 'make' implementation
because Make was necessary to build Lzip. Lzip is then used to
uncompress the source of our own GNU Make. However, in the
minimalist/slim versions of operating systems (for example used to build
Docker images) Make isn't included by default. Since Lzip was the only
program before our own GNU Make was installed, we consulting Antonio
Diaz Diaz (creator of Lzip) and he kindly added the necessary
functionality to a new version of Lzip, which we are using now. Hence we
don't need to assume a Make implementation on the host any more. With
this commit, Lzip and GNU Make are built without Make, allowing
everything else to be safely built with our own custom version of GNU
Make and not using the host's 'make' at all.
- Until recently (Commit 3d8aa5953c4) GNU Make was built in
'basic.mk'. Therefore 'basic.mk' was written in a way that it can be
used with other 'make' implementations also (i.e., important shell
commands starting with '&&' and ending in '\' without any comments
between them!). Furthermore, to help in style uniformity, the rules in
'high-level.mk' and 'python.mk' also followed a similar structure. But
due to the point above, we can now guarantee that GNU Make is used from
the very first Makefile, so this hard-to-read structure has been removed
in the software build recipes and they are much more readable and
edit-friendly now.
- Until now, the default backup servers where at some fixed URLs, on our
own pages or on Gitlab. But recently we uploaded all the necessary
software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is
more suitable for this task (it promises longevity, has a fixed DOI,
while allowing us to add new content, or new software tarball
versions). With this commit, a small script has been written to extract
the most recent Zenodo upload link from the Zenodo DOI and use it for
downloading the software source codes.
- Until now, we primarily used the webpage of each software for
downloading its tarball. But this caused many problems: 1) Some of them
needed Javascript before the download, 2) Some URLs had a complex
dependency on the version number, 3) some servers would be randomly down
for maintenance and etc. So thanks to the point above, we now use the
Zenodo server as the primary download location. However, if a user wants
to use a custom software that is not (yet!) in Zenodo, the download
script gives priority to a custom URL that the users can give as Make
variables. If that variable is defined, then the script will use that
URL before going onto Zenodo. We now have a special place for such URLs:
'reproduce/software/config/urls.conf'. The old URLs (which are a good
documentation themselves) are preserved here, but are commented by
default.
- The software source code downloading and checksum verification step has
been moved into a Make function called 'import-source' (defined in the
'build-rules.mk' and loaded in all software Makefiles). Having taken all
the low-level steps there, I noticed that there is no more need for
having the tarball as a separate target! So with this commit, a single
rule is the only place that needs to be edited/added (greatly
simplifying the software building Makefiles).
- Following task #15272, A new option has been added to the './project'
script called '--all-highlevel'. When this option is given, the contents
of 'TARGETS.conf' are ignored and all the software in Maneage are built
(selected by parsing the 'versions.conf' file). This new option was
added to confirm the extensive changes made in all the software building
recipes and is great for development/testing purposes.
- Many of the software hadn't been tested for a long time! So after using
the newly added '--all-highlevel', we noticed that some need to be
updated. In general, with this commit, 'libpaper' and 'pcre' were added
as new software, and the versions of the following software was updated:
'boost', 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in'
shell script was added in 'reproduce/software/shell/' which is installed
with 'libpaper'.
- Even though we intentionally add the necessary flags to add RPATH inside
the built executable at compilation time, some software don't do it
(different software on different operating systems!). Until now, for
historical reasons this check was done in different ways for different
software on GNU/Linux sytems. But now it is unified: if 'patchelf' is
present we apply it. Because of this, 'patchelf' has been put as a
top-level prerequisite, right after Tar and is installed before anything
else.
- In 'versions.conf', GNU Libtool is recognized as 'libtool', but in
'basic.mk', it was 'glibtool'! This caused many confusions and is
corrected with this commit (in 'basic.mk', it is also 'libtool').
- A new argument is added to the './project' script to allow easy loading
of the project's shell and environment for fast/temporary testing of
things in the same environment as the project. Before activating the
project's shell, we completely remove all host environment variables to
simulate the project's environment. It can be called with this command:
'./project shell'. A simple prompt has also been added to highlight that
the user is using the Maneage shell!
|
|
In Commit 105467fe6402 (Software tarballs are downloaded even if not
built), we introduced tests to download the tarballs of software even if
they don't need to be built on the respective host. However some small
typos in the checks existed that could cause a crash on macOS. In
particular in the building of PatchELF and libbsd we had forgot to add the
necessary 'x' before the 'yes' in the conditional to check if a we are on
macOS or not.
With this commit these two checks have been corrected. Also, in the
building of 'isl' and 'mpc', we now check for 'host_cc' (signifying that
the user wants to use their host C compiler for the high-level step)
instead of 'on_mac_os'. The reason is that even on non-macOS systems, a
user may not want to build the C compiler from scratch and use the
'--host-cc' option. In such cases, they don't need to compile 'isl' and
'mpc'.
|
|
Some low-level software aren't necessary on some operating systems, for
example GCC can't be built on macOS, hence we don't build it and the
GCC-only dependencies. Also, on GNU/Linux systems users could configure
with '--host-cc' to avoid all the time it takes to build GCC when doing a
fast test.
Until now, in such cases not only was the software not installed, but the
tarballs of the software were also not downloaded. Hence making the output
of '--dist-software' incomplete (as in bug #58561).
With this commit, we now import all the necessary tarballs, when the
software isn't necessary for the particular system, it won't be built or
cited, but its tarball will be present anyway, thus allowing the output of
'--dist-software' to be complete.
|
|
After trying a clean build of Maneage in a Docker image (with a minimal
debian:stable-20200607-slim OS), I noticed that the building of OpenSSL is
failing because it doesn't find the proper Perl functionality. To fix it,
with this commit, Perl is set as a prerequisite of OpenSSL and this fixed
the problem.
|
|
Until now, Maneage would only build Flock before building everything else
using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid
parallel downloads during the building of software (which could cause
network problems). But after recently trying Maneage on FreeBSD (which is
not yet complete, see bug #58465), we noticed that the BSD implemenation of
Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq'
parts) and its shell also had some peculiarities.
It was thus decided to also install our own minimalist shell, Make and
compressor program before calling 'basic.mk'. In this way, 'basic.mk' can
now assume the same GNU Make features that high-level.mk and python.mk
assume. The pre-make building of software is now organized in
'reproduce/software/shell/pre-make-build.sh'.
Another nice feature of this commit is for macOS users: until now the
default macOS Make had problems for parallel building of software, so
'basic.mk' was built in one thread. But now that we can build the core
tools with GNU Make on macOS too, it uses all threads. Furthermore, since
we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have
to finish every line of a long rule with a backslash to keep variables and
such.
Generally, the pre-make software are now organized like this: first we
build Lzip before anything else: it is downloaded as a simple '.tar' file
that is not compressed (only ~400kb). Once Lzip is built, the pre-make
phase continues with building GNU Make, Dash (a minimalist shell) and
Flock. All of their tarballs are in '.tar.lz'. Maneage then enters
'basic.mk' and the first program it builds is GNU Gzip (itself packaged as
'.tar.lz'). Once Gzip is built, we build all the other compression software
(all downloaded as '.tar.gz'). Afterwards, any compression standard for
other software is fine because we have it.
In the process, a bug related to using backup servers was found in
'reproduce/analysis/bash/download-multi-try' for calling outside of
'basic.mk' and removed Bash-specific features. As a result of that bug-fix,
because we now have multiple servers for software tarballs, the backup
servers now have their own configuration file in
'reproduce/software/config/servers-backup.conf'. This makes it much easier
to maintain the backup server list across the multiple places that we need
it.
Some other minor fixes:
- In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'.
- In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD
(which doesn't have GCC).
- We are now using 'uname -s' to specify if we are on a Linux kernel or
not, if not, we are still using the old 'on_mac_os' variable.
- While I was trying to build on FreeBSD, I noticed some further
corrections that could help. For example the 'makelink' Make-function
now takes a third argument which can be a different name compared to the
actual program (used for examle to make a link to '/usr/bin/cc' from
'gcc'.
- Until now we didn't know if the host's Make implementation supports
placing a '@' at the start of the recipe (to avoid printing the actual
commands to standard output). Especially in the tarball download phase,
there are many lines that are printed for each download which was really
annoying. We already used '@' in 'high-level.mk' and 'python.mk' before,
but now that we also know that 'basic.mk' is called with our custom GNU
Make, we can use it at the start for a cleaner stdout.
- Until now, WCSLIB assumed a Fortran compiler, but when the user is on a
system where we can't install GCC (or has activated the '--host-cc'
option), it may not be present and the project shouldn't break because
of this. So with this commit, when a Fortran compiler isn't present,
WCSLIB will be built with the '--disable-fortran' configuration option.
This commit (task #15667) was completed with help/checks by Raul
Infante-Sainz and Boud Roukema.
|
|
One of the main reasons to building Maneage is to properly
acknowledge/attribute the authors of software in research. So we have
adopted a standard of never referring to the GNU-based operating systems
running the Linux kernel simply as "Linux", we avoid terms like "Open
Sourse" and use Free Software instead (in the same spirit).
With this commit, a few instances of the cases above have been corrected,
they had slipped through our fingers when we initially imported them into
the project. In the special case of the "Journal for Open Source Software",
we simply replaced it with its abbreviation (JOSS). This was done because
in effect we were generally using journal name abbreviations in almost all
the citations already. To avoid any inconsistancies, the names of the three
other journals that weren't abbreviated are also abbreviated.
|
|
With this commit, Maneage now includes instructions to build the memory
tracing tool Valgrind and the program 'patch' (to apply corrections/patches
in text files and in particular the sources of programs).
For this version of Valgrind, some patches were necessary for an interface
with OpenMPI 2.x (which is the case now). Also note that this version of
Valgrind's checks can fail with GCC 10.1.x (when using '--host-cc'), and
the failures aren't due to internal problems but due to how the tests are
designed (https://bugs.gentoo.org/707598). So currently if any of
Valgrind's checks fail, Maneage still assumes that Valgrind was built and
installed successfully.
While testing on macOS, we noticed that it needs the macOS-specific 'mig'
program which we can't build in Maneage. DESCRIPTION: The mig command
invokes the Mach Interface Generator to generate Remote Procedure Call
(RPC) code for client-server style Mach IPC from specification files. So a
symbolic link to the system's 'mig' is now added to the project's programs
on macOS systems.
This commit's build of Patch and Valgrind has been tested on two GNU/Linux
distributions (Debian and ArchLinux) as well as macOS.
Work on this commit started by Boud Roukema, but also involved tests and
corrections by Mohammad Akhlaghi and Raul Infante-Sainz.
|
|
In time, some of the copyright license description had been mistakenly
shortened to two paragraphs instead of the original three that is
recommended in the GPL. With this commit, they are corrected to be exactly
in the same three paragraph format suggested by GPL.
The following files also didn't have a copyright notice, so one was added
for them:
reproduce/software/make/README.md
reproduce/software/bibtex/healpix.tex
reproduce/analysis/config/delete-me-num.conf
reproduce/analysis/config/verify-outputs.conf
|
|
Until now Maneage used the host's GNU Gettext if it was present. Gettext is
a relatively low-level software that enables programs to print messages in
different languages based on the host environment. Even though it has not
direct effect on the running of the software for Maneage and the lanugage
environment in Maneage is pre-determined, it is necessary to have it
because if the basic programs see it in the host they will link with it and
will have problems if/when the host's Gettext is updated.
With this commit (which is actually a squashed rebase of 9 commits by Raul
and Mohammad), Gettext and its two extra dependencies (libxml2 and
libunistring) are now installed within Maneage as a basic software and
built before GNU Bash. As a result, all programs built afterwards will
successfully link with our own internal version of Gettext and
libraries. To get this working, some of the basic software dependencies had
to updated and re-ordered and it has been tested in both GNU/Linux and
macoS.
Some other minor issues that are fixed with this commit
- Until this commit, when TeX was not installed, the warning message
saying how to run the configure step in order to re-configure the
project was not showing the option `-e'. However, the use of this option
is more convenient than entering the top-build directory and etc every
time. So with this commit, the warning message has been changed in order
use the option `-e' in the re-configure of the project.
- Until now, on macOS systems, Bash was not linking with our internally
built `libncurses'. With this commit, this has been fixed by setting
`--withcurses=yes' for Bash's configure script.
|
|
Until now, when you changed the version of a software in an already-built
system, its tarball would be downloaded, but it wouldn't actually
build. The only way would be to force the build by deleting the main target
of that file (under `.local/version-info/TYPE/PROGRAM'). This was because
the tarballs were an order-only prerequisite which was implemented some
time ago based on some theoretical argument that if the tarball dates
changes, the software should not be rebuilt (because we check the
checksum).
However, the problems this causes are more than those it solves: Users may
forget to delete the main target of the program and mistakenly think that
they are using the new version. The fact that all the numbers going into
the paper also contain this number further hides this.
With this commit, tarballs are no longer order-only and any time a version
of a software is updated, it will be automatically built and not cause
confusion and manual intervention by the users. As a result of this change,
I also had to correct the way we find the tarball from the list of
prerequisites.
|
|
Until this commit, the configure step would fail with an error when
compiling libgit2 on a test system. The origin of this bug, on the OS that
was tested, appears to be that in OpenSSL Version 1.1.1a, openssl/ec.h
fails to include openssl/openconf.h. The bug is described in more detail at
https://savannah.nongnu.org/bugs/index.php?58263
With this commit, this is fixed by manually inserting a necessary
components. In particular, `sed` is used to insert a preprocessor
instruction into `openssl/openconf.h`, defining `DEPRECATED_1_2_0(f)`, for
an arbitrary section of code `f`, to include that code rather than exclude
it or warn about it.
This commit is valid provided that openssl remains at a version earlier
than 1.2.0. Starting at version 1.2.0, deprecation warnings should be run
normally. We have thus moved the version of OpenSSL in `versions.conf' to
the section for programs that need to be manually checked for version
updates with a note to remind the user when reaching that version.
Other packages that use OpenSSL may benefit from this commit, not just
libgit2.
|
|
Until now, if GCC couldn't be built for any reason, Maneage would crash and
the user had no way forward. Since GCC is complicated, it may happen and is
frustrating to wait until the bug is fixed. Also, while debugging Maneage,
when we know GCC has no problem, because it takes so long, it discourages
testing.
With this commit, we have re-activated the `--host-cc' option. It was
already defined in the options of `./project', but its affect was nullified
by hard-coding it to zero in the configure script on GNU/Linux systems. So
with this commit that has been removed and the user can use their own C
compiler on a GNU/Linux operating system also.
Furthermore, to inform the user about this option and its usefulness, when
GCC fails to build, a clear warning message is printed, instructing the
user to post the problem as a bug and telling them how to continue building
the project with the `--host-cc' option.
|
|
Until Commit 3409a54 (from yesterday), pkg-config was found correctly in
`reproduce/software/make/basic.mk` by searching for `pkg`. However, commit
a21ea20 made an improvement in the regular expression for relating package
names and download filenames, and the string `pkg-config` with the new
regex no longer simplifies to `pkg`. The result of this was that the
basic.mk could not find `pkg-config` in the list of packages, since it was
still listed as `pkg`. This blocked downloading for a system without
pkg-config preloaded.
With this commit (of just a few bytes), the bug is fixed.
|
|
Until now, the sed script for determining URL download rules in the three
software building Makefiles (`basic.mk', `high-level.mk' and `python.mk')
considered package names such as `fftw-3...` and `fftw2-2.1...` to be
identical. As the example above shows, this would make it hard to include
some software that may hav conflicting non-number names.
With this commit, the SED script that is used to separate the version from
the tarball name only matches numbers that are after a dash
(`-'). Therefore considers `fftw-3...` and `fftw-2...` to be identical, but
`fftw-3-...` and `fftw2-2.1...` to be different. As a result of this
change, the `elif' check for some of the other programs like `m4', or
`help2man' was also corrected in all three Makefiles.
While doing this check on all the software, we noticed that `zlib-version'
is being repeated two times in `version.conf' so it was removed. It caused
no complications, because both were the same number, but could lead to bugs
later.
|
|
Until now, throughout Maneage we were using the old name of "Reproducible
Paper Template". But we have finally decided to use Maneage, so to avoid
confusion, the name has been corrected in `README-hacking.md' and also in
the copyright notices.
Note also that in `README-hacking.md', the main Maneage branch is now
called `maneage', and the main Git remote has been changed to
`https://gitlab.com/maneage/project' (this is a new GitLab Group that I
have setup for all Maneage-related projects). In this repository there is
only one `maneage' branch to avoid complications with the `master' branch
of the projects using Maneage later.
|
|
Of the GCC dynamically linked libraries we need to manually add RPATH to
all and for `libstdc++' we also need to tell it to link with
`libiconv'. Until now, the conditional to check for libstdc++ was not
working and thus libiconv wasn't been added to it.
With this commit the conditional has been corrected and is now
working. Also, to help in reading the logs, an echo statement was added
after every call to PatchELF.
|
|
Until now, when a the raw tarball of some software wasn't usable, I would
put it under my own webpage, or `akhlaghi.org/reproduce-software'. That
same address was also used as a backup server. However, now the project has
a proper name: Maneage. So I changed the directory on my own server to
`akhlaghi.org/maneage-software'.
With this commit, this new address has replaced the old one. But to avoid
crashes in projects that haven't yet merged with the main Maneage branch,
the old `reproduce-software' still works (its actually a symbolic link to
the new directory now).
|
|
Until a few commits ago, PatchELF was built statically because it was used
to patch `libstdc++' at the end of the GCC building phase, but PatchELF
also depends on `libstdc++', so it would crash. However, recently when
patching the GCC libraries, we don't directly apply Patchelf to the
library, first we copy it to a temporary place, do the patching, then put
it in its proper place. So the problem above won't happen any more.
With this commit, I am thus removing the static flag from patchelf and
letting it built dynamically all the time. The main problem was that some
systems don't have a static C++ library, so PatchELF couldn't be built
statically. Instead of adding more checks, we just fixed the core
foundation of the problem.
|
|
Until now the software configuration parameters were defined under the
`reproduce/software/config/installation/' directory. This was because the
configuration parameters of analysis software (for example Gnuastro's
configurations) were placed under there too. But this was terribly
confusing, because the run-time options of programs falls under the
"analysis" phase of the project.
With this commit, the Gnuastro configuration files have been moved under
the new `reproduce/analysis/config/gnuastro' directory and the software
configuration files are directly under `reproduce/software/config'. A clean
build was done with this change and it didn't crash, but it may cause
crashes in derived projects, so after merging with Maneage, please
re-configure your project to see if anything has been missed. Please let us
know if there is a problem.
|
|
Until now, the M4 that was built on macOS had internal problems (as
discussed in #1): it would simply print `Abort trap: 6' in the output and
abort. After looking at the build of Homebrew, I noticed that they apply a
patch (correct one line) to fix this problem. To be able to apply that
patch on macOS systems, I had fully open up the build recipe of M4 and
atleast on the testing system, it was built successfully.
Also, after successfully building M4, and thus Autoconf and thus Minizip,
we were able to build XLSX I/O on a macOS and found out that the internal
library's full address wasn't being put in the libraries and
executables. With this commit, we now use macOS's `install_name_tool' to
correct the positions of the two `libxlsxio_*' libraries in all its
executables.
This commit was originally written by Mohammad Akhlaghi
|
|
Until now, like all software on GNU/Linux systems GCC would be built in RAM
(to speed up the build slightly and also not put too much stress on the
HDDs/SSDs). But some systems don't have enough RAM for building GCC and
will complain and crash.
With this commit, we have added a check on the amount of free space in the
`build_tmp' directory (which will be `/dev/shm' on GNU/Linux systems). If
the amount of free space isn't more than 10GB, then GCC won't be built
there and a temporary directory will be built under the `$(BDIR)/software'
directory for it.
This bug was found by Zahra Sharbaf.
This fixes bug #57853.
|
|
Until now we were mistakenly reporting the version of SED instead of
Texinfo.
With this commit, we corrected it!
This was reported by Raul Infante Sainz.
|
|
Until now, we defined `LIBRARY_PATH' to fix the problem of the `ld' linker
of Binutils needing several `*crt*.o' files to run. However, some software
(for example ImageMagick) over-write `LIBRARY_PATH', therefore there is no
other way than to put a link to these necessary files in our local build
directory.
With this commit, we fixed the problem by putting a link to the system's
relevant files in the local library directory. This fixed the problem with
ImageMagick. Later, when we build the GNU C Library in the project, we
should remove this step.
This bug reported by Raul Castellanos Sanchez.
|
|
Until now, the main download script could only check one server for the
given URL. However, ultimately the actual server that a file is downloaded
from is irrelevant for this project: we actually check its
checksum. Especially in the case of software (which are distributed over
many servers), this can usually be very annoying: the servers may not
properly communicate with the running system and even the 10 trials won't
be enough.
With this commit, the download script
`reproduce/analysis/bash/download-multi-try' can take a new optional
argument (a 5th argument). It assumes this argument is a space-separated
list of server(s) to use as backup for the original URL. When downloading
from the original URL fails, it will look into this list and try
downloading the same file from each given server.
|
|
Until now the shell scripts in the software building phase were in the
`reproduce/software/bash' directory. But given our recent change to a
POSIX-only start, the `configure.sh' shell script (which is the main
component of this directory) is no longer written with Bash.
With this commit, to fix that problem, that directory's name has been
changed to `reproduce/software/shell'.
|
|
Until now, the initial project scripts were primarily tested with GNU
Bash. But Bash is not generally available on all systems (it has many
features beyond POSIX). Because of this, effectively we were imposing the
requirement on the user that they must have Bash installed. We recently
started this with setting the shebang of `project' and
`reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul
and Gaspar reported an error on their systems.
To fix the problem, I installed Dash (a minimalist POSIX-compliant shell)
on my computer and temporarily set the shebangs to `/bin/dash', ran the
project configuration step and fixed all issues that came up. With this
commit, it can go all the way to building GCC on my system's Dash. After
this stage (when `high-level.mk' is called), there is no problem, because
we have our own version of GNU Bash and that installed version is used.
Probably some more issues still remain and will hopefully be found in the
future.
While doing this, I also noticed the following two minor issues:
- The `./project configure' option `--input-dir' was not recognized
because it was mistakenly checking `--inputdir'. It has been corrected.
- The test C programs now use the `<<EOF' method instead of `echo'.
- In `basic.mk', the extra space between `syspath' and `:=' was removed
(it was an ancient relic!).
|
|
Until now Perl was built after Coreutils, but I recently noticed that
Coreutils actually uses Perl while creating its manpages. So it is now
built before Coreutils.
Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils
can't build its man page for `md5sum'. The problem was found to be due to
the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our
installed library path in `basic.mk'. Therefore, it would crash because the
server had an older version of OpenSSL than the one that the template's
Coreutils was built with.
In the meantime (while addressing the issues above, because we only had one
thread on the AWS server) I also noticed a few programs that were using a
summarize compilation command (that just prints `CC xxx.c' instead of the
whole command) so I fixed them by adding `V=1'.
This bug was found by Idafen Santana Pérez.
|
|
Until now, the configuration Makefiles (in
`reproduce/software/config/installation' and `reproduce/analysis/config')
had a `.mk' suffix, similar to the workhorse Makefiles. Although they are
indeed Makefiles, but given their nature (to only keep configuration
parameters), it is confusing (especially to early users) for them to also
have a `.mk' (similar to the analysis or software building Makefiles).
To address this issue, with this commit, all the configuration Makefiles
(in those directories) are now given a `.conf' suffix. This is also assumed
for all the files that are loaded.
The configuration (software building) and running of the template have been
checked with this change from scratch, but please report any error that may
not have been noticed.
THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS
FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE
SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES
ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
|
|
GNU Make 4.3 was just announced, so I have updated it here is well. This
was important because until now the installable version was in alpha-mode
(4.2.90), now its a stable version.
|
|
Until now, GCC wouldn't build properly on Debian-based operating systems
because `ld' needed to link with several necessary C library features like
`crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to
add the directory containing them to `LIBRARY_PATH'. In the previous
commit, I actually searched for these files, but while testing on another
system, I noticed that it can be problematic (other architectures may
exist).
With this commit, we are actually finding the build architecture of the
running GCC (which is the same as the `ld') and using that to fix a fixed
directory to `LIBRARY_PATH'.
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
Until now, while building Perl, the `-Dlddlflags="-shared $$LDFLAGS"'
option was being used for all systems. However, this caused `symbol(s) not
found for architecture x86_64' problems on macOS systems.
With this commit, this configure option is only passed for GNU/Linux
systems. With it, Perl can be built nicely on both GNU/Linux and macOS.
|
|
Until now when building Matplotlib on macOS systems, we were using the
default C compiler. However, while Yahya Sefidbakht (previously) and
Mahdieh Nabavi (now) were trying to build the template, on their macOS
using the GNU Compiler Collection (GCC), we found that Matplotlib needs
special macOS headers that GCC doesn't recognize.
With this commit, when Matplotlib is being built on macOS systems, it uses
`clang' and this fixed the problem (so far checked on Mahdieh's machine).
|
