Until this commit, the LaTeX style and settings of the core Maneage branch were very generic and customizing for each separate journal required some time to prepare the LaTeX style. With this commit, a first attempt at customization of Maneage for the LaTeX styles of the A&A journal. Note that EDP Science (the publisher of A&A) has only claimed copyright in their style files, but hasn't actually specified a license. Given that they have no objection to arXiv or Overleaf freely distributing their style files, there is no problem is releasing these files within this branch of Maneage also (only for people who want to use their style to publish in their journal).
Summary: after merging this commit into your project, it should be
re-configured since the location of software installation files like
'LOCAL.conf' or the LaTeX macros of the software environment have
changed. But it should not affect the analysis phase of your project.
Until this commit, it was not possible to run a pre-built Maneage'd project
(in a container) on a newly cloned Maneage'd project source. This was
because the containers should be read-only, but during the various checks
of the configuration (to verify that we are using the same software
environment in the container and the source), we were writing/testing many
things in the build directory, and 'LOCAL.conf' which was actually in the
source directory!
Furthermore, the '.local' and '.build' were built at configure time, making
it hard to run the same container from a newly cloned Maneage'd project. To
make things harder for the scenario above, the 'configure.sh' script would
pause on every message and didn't have a quiet mode (making it practically
impossible to run './project configure' before './project make' on every
container run).
With this commit, all these issues have been addressed and it is now
possible to simply get a built container, clone a Maneage'd project and run
the analysis (using the built environment of the container that is verified
on every run). The respective changes/additions are described below:
- The high-level container scripts ('apptainer.sh' and 'docker.sh', along
with their READMEs) have been moved to the 'reproduce/software/shell'
directory and the old 'reproduce/software/containers' directory has been
deleted. This is because we have classified the software files by their
language/format and the container scripts are scripts in the end.
- The './project' script:
- Now has two extra options: '--quiet' and '--no-pause'. Both are
directly passed to the 'configure.sh' script. They will respectively
disable any informative printed message or any pause after that
message (if it is printed).
- The '--build-dir' option is now also relevant for './project make':
when it is given, it will re-create the two '.build' and '.local'
symbolic links at the top source directory in all scenarios
('configure', 'make' or 'shell'). This will allow both the
configuration, analysis and shell phases to safely assume they exist
and match the user's desire at run-time.
- The build/analysis directory's sub-directories that need to be built
before 'top-make.mk' are now built in a separate function to help in
readability.
- The 'configure.sh' script:
- For developers: a new 'check_elapsed' variable has been defined that
will enable the newly added 'elapsed_time_from_prev_step'
function. This function should be used from now on at the end of
every major step to help find bottlenecks.
- The targets of the software in 'pre-make-build.sh' now also have the
version of the software in their file name. Until now, they didn't have
the version, so there was no way to detect if the software has been
updated or not in the source. For Lzip and Make (that also get built
after GCC), the ones in this script have a '-pre-make' suffix also.
- 'Local.conf.in' now has descriptions for every variable.
- The '-std=gnu17' option is now used instead of '-std=c17' for basic
software that cannot be built without specifying the C standard in GCC
15.1 (described in previous commit: 2881fc0a6205). See [1] for more
details; in summary: '-std=gnu17' is also supported on macOS's Clang and
has some features that 'pkg-config' needs
- Generally: some longer code lines have been broken or indentation
decreased to fit the 75 character line length. This has not reduced
readability however. For example the long 'echo' commands are now
replaced by multiple 'printf's, or the indentation is still clearly
visible.
The seeds of the work on this commit started by a branch containing three
commits by Giacomo Lorenzetti (133 insertions, 100 deletions). Upon merging
with the main 'maneage' branch, they were generalized and re-organized to
become this commit.
The following issues have also been addressed with this commit:
- The LaTeX calls (during the building of 'paper.pdf') do not contain
Maneage'd dynamic libraries. This is because we don't build the LaTeX
binaries from source, an TeXLive manager uses the host environment.
- The 'docker.sh' script:
- Adds the '--project-name' option: its internal variable existed, but
the option for the user to define it at run-time was not.
- Ported to macOS: it does not check being a member of the 'docker'
group, and finds the number of threads using macOS-specific tools.
- The 'apptainer.sh' script:
- Now installs 'wget' in the base container also (necessary when the
user doesn't have the tarballs).
[1] https://savannah.nongnu.org/bugs/?67068#comment2
Summary: this will not affect already built/configured projects, only improving the portability of future builds. Until this commit, no specific C standard version was given during the build of Basic software (which use the host's c compiler, culminating in the building of GCC within Maneage). On the other hand, GCC 15.1 was recentely released and made available on some operating systems. GCC 15.1 has made C23 its standard C version (from C17), as a result, some of the basic software crashed and didn't allow Maneage to built on those operating systems. With this commit, the '-std=c17' flag has been added to software that crashed when the host GCC was 15.1. Also, they have been grouped with a description of this problem in 'versions.conf' so we check the need for this option in future version updates. In parallel, some minor edits/clarifications were made in the two 'README-apptainer.md' and 'README-docker.md' files to be more useful.
Summary: this will not affect any analysis. Until this commit, the old version of 'setuptools-rust' did not install with the updated Python version (see [1]). With this commit, 'setuptools-rust' is upgraded to version 1.10.2 (from 1.1.2) and 'semantic-version' 2.10.0 (needed by 'setuptools-rust') is added. In addition this commit: - removes a duplicate rule for building 'cycler' in 'python.mk'. - comments all the un-commented '*-url' variables of 'urls.conf'. [1] https://savannah.nongnu.org/bugs/?61731
Summary: this will not affect your high-level projects Until this commit, the 'reproduce/software/config/TARGETS.conf' included both Gnuastro and Astropy. But the default analysis does not use Astropy, therefore to avoid taking too much time for a default build, it should only be added when the user needs it. It was present because of the tests in the previous commit to update the Python environment, but I forgot to remove it before pushing. With this commit, TARGETS.conf only contains Gnuastro.
Summary: this is software update that involves Astropy and its
dependencies. In case you are using any of those, it is recommended to
reproduce the analysis of your project to make sure that the calls to the
various packages have not changed. If you are not using Astropy or its
dependencies, this commit will not affect your analysis.
Until this commit, Astropy and the whole Python environment of its
prerequisites were last updated more than 2 years go. So after the update
of the Basic and some high-level (Gnuastro dependencies) software, it was
necessary to update Astropy and its dependencies (for those who use it).
With this commit, the software listed in the last item below have been
updated as well as the following minor technical fixes:
- Support bug 110738 (https://savannah.nongnu.org/support/?110738): when
the build directory had the same base-name as the source directory (for
exapmle '/PATH/proj' and /PATH/proj-build'), Maneage would complain and
not continue.
- The Python software are now built using the wheels system with
'gpep517' (https://pypi.org/project/gpep517). Python packages that are
not required by astropy/matplotlib/scipy have mostly not been updated,
but updating their dependencies and adding the build method as
'GPEP517' should be straightforward.
- Updated software (dependencies of Astropy). Those with 'N/A' in their
previous version string weren't in Maneage prior to this commit.
High-level software (not written in Python):
boost 1.77.0 1.87.0
ninja build system n/a 1.12.1
openblas 0.3.21 0.3.29
Python packages:
python 3.10.6 3.13.2
astropy 5.1 7.0.1
astropy-iers-data n/a 0.2025.2.24.0.34.4
contourpy n/a 1.3.1
cppy n/a 1.3.1
cycler 0.11.0 0.12.1
cython 0.29.24 3.0.12
extension-helpers 0.1 1.2.0
features n/a 0.5.12
flit-core n/a 3.11.0
fonttools n/a 4.56.0
gpep517 n/a 16
kiwisolver 1.0.1 1.4.8
matplotlib 3.3.0 3.10.0
meson n/a 1.7.0
meson-python n/a 0.17.1
numpy 1.21.3 2.2.3
packaging 21.3 24.2
pillow 8.4.0 11.1.0
pybind11 2.5.0 2.13.6
pyerfa-version 2.0.0.1 2.0.1.5
pyproject-metadata n/a 0.9.0
pyparsing 3.0.4 3.2.1
python-dateutil 2.8.0 2.9.0.post0
python-installer n/a 0.7.0
pyyaml 5.1 6.0.2
setuptools 58.3.0 75.8.0
setuptools_scm 3.3.3 8.2.0
scipy 1.7.3 1.15.2
six 1.16.0 1.17.0
wheel 0.37.0 0.45.1
SUMMARY: no change is necessary for your pipeline after upgrading to this
commit.
Until this commit, the following noteworthy portability problems existed
that have been fixed as described in each item.
- In commit 8908587 (titled "IMPORTANT: 73 software upgrades and added
offline mode"; committed on 2025-02-11); the versions of all (except
those that were not updated upstream) basic software as well as the
high-level software necessary for Gnuastro were updated. This included
Python. However, the update of Python could cause crashes in the
Python-related software that are also in Maneage but not tested for
that commit.
-- With this commit, Python has been revered back to 3.10.6 (from
3.13.2) and it has been taken to a dedicated part of 'versions.conf'
(reminding the reader that the version of Python should only be
updated by the Python maintainer: to ensure it doesn't conflict with
Python-based builds).
- Following the update of XZ Utils in 8908587, the custom build recipe
that was necessary is no longer necessary. This was reported by Boud
Roukema (see https://savannah.nongnu.org/bugs/index.php?62700).
-- With this commit, XZ Utils uses our basic high-level build scripts.
- CMake could use out-of-Maneage libraries for those that we already had
in Maneage on some systems. This was reported by Boud Roukema (see
https://savannah.nongnu.org/bugs/?63043)
-- With this commit, CMake is built with '--no-system-libs'; so it
builds and statically links to all its necessary libraries
internally (not confusing between Maneage and the host).
- Building Binutils 2.39 was likely to fail on older GCC versions due to
special features (such as the 'gprofng' feature when building with GCC
6.2.0). This was reported by Boud Roukema in
https://savannah.nongnu.org/bugs/index.php?63242.
-- With this commit, the 'gprofng' feature of Binutils has been
disabled by default to allow reproducibility on older systems. If
you need it, remove the '--enable-gprofng=no' option in the build
recipe of Binutils.
- The Maneage tarball repository is now also kept in 'gitlab.cefca.es'.
-- With this commit, a new server URL has been added for this.
- The list of TeXLive packages included some packages that get installed
in the basic scheme that we do for the initial setup of TeXLive.
-- With this commit, the extra packages have been removed.
Summary: This is a software update to make Maneage more portable and up to
date. It does not involve any Maneage infrastructure changes. You should
just re-build your project to make sure the updated software haven't
removed/changed any of their features that you were using.
Until now, the software in Maneage had not been updated for almost 2.5
years (since August 2022) and were starting to get dated! Also, when no
internet is available (for example in a HPC), the Zenodo check would always
cause a crash and needed to be commented manually.
With this commit, all the basic software that had been updated as well as
the dependencies of Gnuastro (which we commit to maintaining) have been
updated after a check of their webpage (see the full list below). To fix
the offline issue, a new '--offline' option has been added to the project
script. The other following changes were made:
- tarball-prepare: has become more resilient (for dealing with file name
with spaces or Lzip, which should not be compressed).
- Readme: improve documentation when used in Docker.
Below you can see the list of basic software that have been updated. The
only basic software that have not been updated upstream since the previous
Maneage software update are 'bison', 'flock', 'm4', 'pkgconfig', 'unzip'
and 'zip'.
bash 5.2-rc2 5.2.37
binutils 2.39 2.43.1
coreutils 9.1 9.6
curl 7.84.0 8.11.1
dash 0.5.11-057cd65 0.5.12
diffutils 3.8 3.10
file 5.42 5.46
findutils 4.9.0 4.10.0
gawk 5.1.1 5.3.1
gcc 12.1.0 14.2.0
gettext 0.21 0.23.1
git 2.37.1 2.48.1
gmp 6.2.1 6.3.0
grep 3.7 3.11
gzip 1.12 1.13
help2man 1.49.2 1.49.3
isl 0.24 0.27
less 590 668
libiconv 1.17 1.18
libtool 2.4.7 2.5.4
libunistring 1.0 1.3
libxml2 2.9.12 2.13.5
lzip 1.23 1.25
make 4.3 4.4.1
mpc 1.2.1 1.3.1
mpfr 4.1.0 4.2.1
nano 6.4 8.3
ncurses 6.3 6.5
openssl 3.0.5 3.4.0
perl 5.36.0 5.40.1
podlators 4.14 6.0.2
readline 8.2-rc2 8.2.13
sed 4.8 4.9
tar 1.34 1.35
texinfo 6.8 7.2
wget 1.21.2 1.25.0
which 2.21 2.23
xz 5.2.5 5.6.3
zlib 1.2.11 1.3.1
The high-level software that have been updated.
autoconf 2.71 2.72
automake 1.16.5 1.17
cfitsio 4.1.0 4.5.0
cmake 3.24.0 3.31.5
expat 2.4.1 2.6.4
fontconfig 2.14.0 2.16.0
freetype 2.11.0 2.13.3
ghostscript 9.56.1 10.04.0
gnuastro 0.18 0.23
gsl 2.7 2.8
libffi 3.4.2 3.4.7
libgit2 1.3.0 1.9.0
libice 1.0.10 1.1.2
libidn 1.38 1.42
libjpeg 9e 9f
libpaper 1.1.28 1.1.29
libpng 1.6.37 1.6.46
libpthread-stubs 0.4 0.5
libsm 1.2.3 1.2.5
libtiff 4.4.0 4.7.0
libxau 1.0.9 1.0.12
libxcb 1.15 1.17
libxdmcp 1.1.3 1.1.5
libxext 1.3.4 1.3.6
libxt 1.2.1 1.3.1
python 3.10.6 3.13.12
util-macros 1.19.3 1.20.2
wcslib 7.11 8.4
xcb-proto 1.15 1.17
xorgproto 2022.1 2024.1
xtrans 1.4.0 1.5.2
SUMMARY: no change necessary in your project, this commit only involves
changes in how already-existing software are built. Some handy options have
also been added to the top-level project script and the copyright years
have been updated.
Until now, if the host's '/bin/sh' had conflicts with the Maneage
environment, the configuration of Maneage would crash as soon as we entered
the building of high-level software. The full scenario is described in the
comments of the newly added 'reproduce/software/shell/prep-source.sh'. This
is most relevant when building older Maneage'd project in newer
environments.
With this commit, the following changes were made to avoid the problem
above:
- Maneage edits the source code of all installed software to replace
'/bin/sh' with Maneage's own shell before the programs are
built. Through this, we were able to solve the problem described
above.
- The portable '#!/usr/bin/env sh' shebangs are now used at the start of
the scripts that run during configure time so it uses the first
available shell that it finds in its PATH (the system's before Dash is
built), then Dash, and after Dash is built, Bash.
- For TeXLive, since we don't install it from source, it was necessary to
add the libraries necessary for the local '/bin/sh' in LD_LIBRARY_PATH.
Some high-level options have been added to the './project' script to
simplify certain operations:
--keep-going: do not stop upon the first crash, but keep going on to
build targets until all build-able targets have been built. This is
very useful for debugging large pipelines and allows you to isolate the
problematic part of your project.
--highlight-all: equivalent to calling both '--highlight-new' &
'--highlight-notes'.
SUMMARY: no change is necessary in your project, unless you use the Fortran features of WCSLIB in your project. Until now, there were two compilation failures on recent macOS computers with an M1 CPU: Less would crash because it couldn't find the relevant PCRE (perl-compatible regular expression) libraries and WCSLIB would crash because the LLVM compiler's Fortran features could not be built. With this commit, both issues have been fixed by disabling the relevant feature. Extensive comments have been placed in both places in case your project needs these features, so please see the comments in the relevant part of 'reproduce/software/make/basic.mk' for Less and 'reproduce/software/make/high-level.mk' for WCSLIB. In fact the previous solution (where we would not have Fortran features in WCSLIB on macOS systems was problematic and non-reproducibile (the features of WCSLIB depended on the operating system!). Another minor change was that for macOS, we now directly use the version-string of WCSLIB to fix the internal linking issue there. As a result, WCSLIB is no longer a "Version-dependent build" software (in 'reproduce/software/config/versions.conf'). Recall that these are software that when changing the version, it is also necessary to inspect their build recipe. These two issues and their fix were discovered and fixed with the help of James Robinson.