Summary: will not affect existing configuration or analysis.

Until this commit, if the user gave the build directory interactively (had
not run './project configure' with '--build-dir'), the '.build' and
'.local' symbolic links were not created, resulting in a crash shortly
afterwards (when Maneage tried to write 'LOCAL.conf' in the build
directory!).

With this commit, the problem is solved by creating these links also when
the build directory is given interactively and after all the sanity checks
have passed.

Summary: after merging this commit into your project, it should be
re-configured since the location of software installation files like
'LOCAL.conf' or the LaTeX macros of the software environment have
changed. But it should not affect the analysis phase of your project.

Until this commit, it was not possible to run a pre-built Maneage'd project
(in a container) on a newly cloned Maneage'd project source. This was
because the containers should be read-only, but during the various checks
of the configuration (to verify that we are using the same software
environment in the container and the source), we were writing/testing many
things in the build directory, and 'LOCAL.conf' which was actually in the
source directory!

Furthermore, the '.local' and '.build' were built at configure time, making
it hard to run the same container from a newly cloned Maneage'd project. To
make things harder for the scenario above, the 'configure.sh' script would
pause on every message and didn't have a quiet mode (making it practically
impossible to run './project configure' before './project make' on every
container run).

With this commit, all these issues have been addressed and it is now
possible to simply get a built container, clone a Maneage'd project and run
the analysis (using the built environment of the container that is verified
on every run). The respective changes/additions are described below:

 - The high-level container scripts ('apptainer.sh' and 'docker.sh', along
   with their READMEs) have been moved to the 'reproduce/software/shell'
   directory and the old 'reproduce/software/containers' directory has been
   deleted. This is because we have classified the software files by their
   language/format and the container scripts are scripts in the end.

 - The './project' script:

    - Now has two extra options: '--quiet' and '--no-pause'. Both are
      directly passed to the 'configure.sh' script. They will respectively
      disable any informative printed message or any pause after that
      message (if it is printed).

    - The '--build-dir' option is now also relevant for './project make':
      when it is given, it will re-create the two '.build' and '.local'
      symbolic links at the top source directory in all scenarios
      ('configure', 'make' or 'shell'). This will allow both the
      configuration, analysis and shell phases to safely assume they exist
      and match the user's desire at run-time.

    - The build/analysis directory's sub-directories that need to be built
      before 'top-make.mk' are now built in a separate function to help in
      readability.

 - The 'configure.sh' script:

    - For developers: a new 'check_elapsed' variable has been defined that
      will enable the newly added 'elapsed_time_from_prev_step'
      function. This function should be used from now on at the end of
      every major step to help find bottlenecks.

 - The targets of the software in 'pre-make-build.sh' now also have the
   version of the software in their file name. Until now, they didn't have
   the version, so there was no way to detect if the software has been
   updated or not in the source. For Lzip and Make (that also get built
   after GCC), the ones in this script have a '-pre-make' suffix also.

 - 'Local.conf.in' now has descriptions for every variable.

 - The '-std=gnu17' option is now used instead of '-std=c17' for basic
   software that cannot be built without specifying the C standard in GCC
   15.1 (described in previous commit: 2881fc0a6205). See [1] for more
   details; in summary: '-std=gnu17' is also supported on macOS's Clang and
   has some features that 'pkg-config' needs

 - Generally: some longer code lines have been broken or indentation
   decreased to fit the 75 character line length. This has not reduced
   readability however. For example the long 'echo' commands are now
   replaced by multiple 'printf's, or the indentation is still clearly
   visible.

The seeds of the work on this commit started by a branch containing three
commits by Giacomo Lorenzetti (133 insertions, 100 deletions). Upon merging
with the main 'maneage' branch, they were generalized and re-organized to
become this commit.

The following issues have also been addressed with this commit:

 - The LaTeX calls (during the building of 'paper.pdf') do not contain
   Maneage'd dynamic libraries. This is because we don't build the LaTeX
   binaries from source, an TeXLive manager uses the host environment.

 - The 'docker.sh' script:

    - Adds the '--project-name' option: its internal variable existed, but
      the option for the user to define it at run-time was not.

    - Ported to macOS: it does not check being a member of the 'docker'
      group, and finds the number of threads using macOS-specific tools.

 - The 'apptainer.sh' script:

    - Now installs 'wget' in the base container also (necessary when the
      user doesn't have the tarballs).

[1] https://savannah.nongnu.org/bugs/?67068#comment2

Summary: it is necessary to re-configure your project (just running
'./project configure -e', not deleting 'build/software' to re-build
software) after this commit, see "Affected files" item below).

Until now, we only had a relatively long set of manual instructions for
building Maneage within Docker in the top-level README. This was hard to
automate, focing Maneage users to write custom commands based on the
instructions and maintain those scripts outside of Maneage. As a result,
experience could not be shared between projects (or at most in the README
file!).

With this commit, a new 'reproduce/software/containers' directory has been
created within Maneage that contains two scripts (with a unified interface)
greatly simplifying the building of the project's software environment
within a container (one script for Apptainer and one for Docker). Two
READMEs have been added for each container to help in their first time
usage. Also, the old checklist within the main README has been replaced
with a short introduction on containers and points the interested readers
to the custom README of each container technology.

Since we wanted the containers to be read-only after build, we needed to
fully decouple the 'build/software' and 'build/analysis', such that
'./project configure' only writes to the former and './project make' only
writes the latter. The file and directories mentioned in the affected files
are cases that both project phases was writing to the 'build/software' and
'build/analysis' directories.

Affected files: 'preparation-done.mk' and 'lockdir' which were previously
in the 'build/software' directory are now made during the 'make' phase and
the 'configure' phase no longer builds the 'build/analysis' or anything
within it. Also, the software version LaTeX macros (which were previously
written during the 'configure' phase in the 'analysis' directory) are now
written in the software directory and copied into the analysis for usage in
LaTeX while building the paper.

Other minor additions in this commit:

  - The './project' script has a new '--timing' option to write the
    starting and ending times of the project in a file. It also builds the
    high-level analysis directories when './project make' is called (but
    before calling 'top-make.mk'.

  - The 'tar' calls in the custom build commands of the software building
    Makefiles now have the '--no-same-owner --no-same-permissions' options
    like the 'tar' call within the 'uncompress' function of
    'build-rules.mk'.

This commit was originally written by Giacomo Lorenzetti only for Apptainer
on the registered commit date. It was later re-implemented from scratch by
Mohammad Akhlaghi to have a unified interface for both Apptainer and Docker
and merged into Maneage on 2025-04-23.

Summary: this will not affect the analysis of any project.

Until this commit, the part of the configuration script which would check
for the necessary host libraries used by the host shell was not portable
(would not work on some systems). Also, the list of paths had an extra ':'
at the end (allowing the searching of the running directory for libraries,
which is a bug and can have unpredictable effects).

With this commit, both issues have been addressed in the configuration
script.

Summary: this is software update that involves Astropy and its
dependencies. In case you are using any of those, it is recommended to
reproduce the analysis of your project to make sure that the calls to the
various packages have not changed. If you are not using Astropy or its
dependencies, this commit will not affect your analysis.

Until this commit, Astropy and the whole Python environment of its
prerequisites were last updated more than 2 years go. So after the update
of the Basic and some high-level (Gnuastro dependencies) software, it was
necessary to update Astropy and its dependencies (for those who use it).

With this commit, the software listed in the last item below have been
updated as well as the following minor technical fixes:

  - Support bug 110738 (https://savannah.nongnu.org/support/?110738): when
    the build directory had the same base-name as the source directory (for
    exapmle '/PATH/proj' and /PATH/proj-build'), Maneage would complain and
    not continue.

  - The Python software are now built using the wheels system with
    'gpep517' (https://pypi.org/project/gpep517). Python packages that are
    not required by astropy/matplotlib/scipy have mostly not been updated,
    but updating their dependencies and adding the build method as
    'GPEP517' should be straightforward.

  - Updated software (dependencies of Astropy). Those with 'N/A' in their
    previous version string weren't in Maneage prior to this commit.

    High-level software (not written in Python):

       boost                   1.77.0          1.87.0
       ninja build system      n/a             1.12.1
       openblas                0.3.21          0.3.29

    Python packages:

       python                  3.10.6          3.13.2
       astropy                 5.1             7.0.1
       astropy-iers-data       n/a             0.2025.2.24.0.34.4
       contourpy               n/a             1.3.1
       cppy                    n/a             1.3.1
       cycler                  0.11.0          0.12.1
       cython                  0.29.24         3.0.12
       extension-helpers       0.1             1.2.0
       features                n/a             0.5.12
       flit-core               n/a             3.11.0
       fonttools               n/a             4.56.0
       gpep517                 n/a             16
       kiwisolver              1.0.1           1.4.8
       matplotlib              3.3.0           3.10.0
       meson                   n/a             1.7.0
       meson-python            n/a             0.17.1
       numpy                   1.21.3          2.2.3
       packaging               21.3            24.2
       pillow                  8.4.0           11.1.0
       pybind11                2.5.0           2.13.6
       pyerfa-version          2.0.0.1         2.0.1.5
       pyproject-metadata      n/a             0.9.0
       pyparsing               3.0.4           3.2.1
       python-dateutil         2.8.0           2.9.0.post0
       python-installer        n/a             0.7.0
       pyyaml                  5.1             6.0.2
       setuptools              58.3.0          75.8.0
       setuptools_scm          3.3.3           8.2.0
       scipy                   1.7.3           1.15.2
       six                     1.16.0          1.17.0
       wheel                   0.37.0          0.45.1

SUMMARY: this is just a houskeeping commit (no effect on your project).

Until this commit, two of the configure.sh checks for being on a macOS
system had typos (and were thus being wrongly evaluated). In one case, the
variable was called with a '$$' (instead of a single '$') and in the other
a '!=' was used (instead of a '=') resulting in a macOS command ('otool')
being called on GNU/Linux systems.

With this commit, both typos are corrected.

SUMMARY: this commit does not affect your project.

Until this commit, the checklist to update the Maneage branch of a project
was within a separate subsection and the descriptions where comments in a
large code block. This made it hard to find and follow in the large
README-hacking.md file. Also, I noticed that the 'tarballs-prepare.sh'
didn't have a copyright notice and that one of the demo copyright
statements in README-hacking was wrong.

With this commit, the large code block has been broken into an enumerated
list (with small code-boxes under each item) and brought into a dedicated
section, which makes it easier to find. The two other minor points have
also been addressed.

SUMMARY: no change is necessary for your pipeline after upgrading to this
commit.

Until this commit, the following noteworthy portability problems existed
that have been fixed as described in each item.

  - In commit 8908587 (titled "IMPORTANT: 73 software upgrades and added
    offline mode"; committed on 2025-02-11); the versions of all (except
    those that were not updated upstream) basic software as well as the
    high-level software necessary for Gnuastro were updated. This included
    Python. However, the update of Python could cause crashes in the
    Python-related software that are also in Maneage but not tested for
    that commit.
    -- With this commit, Python has been revered back to 3.10.6 (from
       3.13.2) and it has been taken to a dedicated part of 'versions.conf'
       (reminding the reader that the version of Python should only be
       updated by the Python maintainer: to ensure it doesn't conflict with
       Python-based builds).

  - Following the update of XZ Utils in 8908587, the custom build recipe
    that was necessary is no longer necessary. This was reported by Boud
    Roukema (see https://savannah.nongnu.org/bugs/index.php?62700).
    -- With this commit, XZ Utils uses our basic high-level build scripts.

  - CMake could use out-of-Maneage libraries for those that we already had
    in Maneage on some systems. This was reported by Boud Roukema (see
    https://savannah.nongnu.org/bugs/?63043)
    -- With this commit, CMake is built with '--no-system-libs'; so it
       builds and statically links to all its necessary libraries
       internally (not confusing between Maneage and the host).

  - Building Binutils 2.39 was likely to fail on older GCC versions due to
    special features (such as the 'gprofng' feature when building with GCC
    6.2.0). This was reported by Boud Roukema in
    https://savannah.nongnu.org/bugs/index.php?63242.
    -- With this commit, the 'gprofng' feature of Binutils has been
       disabled by default to allow reproducibility on older systems. If
       you need it, remove the '--enable-gprofng=no' option in the build
       recipe of Binutils.

  - The Maneage tarball repository is now also kept in 'gitlab.cefca.es'.
    -- With this commit, a new server URL has been added for this.

  - The list of TeXLive packages included some packages that get installed
    in the basic scheme that we do for the initial setup of TeXLive.
    -- With this commit, the extra packages have been removed.

Summary: This is a software update to make Maneage more portable and up to
date. It does not involve any Maneage infrastructure changes. You should
just re-build your project to make sure the updated software haven't
removed/changed any of their features that you were using.

Until now, the software in Maneage had not been updated for almost 2.5
years (since August 2022) and were starting to get dated! Also, when no
internet is available (for example in a HPC), the Zenodo check would always
cause a crash and needed to be commented manually.

With this commit, all the basic software that had been updated as well as
the dependencies of Gnuastro (which we commit to maintaining) have been
updated after a check of their webpage (see the full list below). To fix
the offline issue, a new '--offline' option has been added to the project
script. The other following changes were made:

 - tarball-prepare: has become more resilient (for dealing with file name
   with spaces or Lzip, which should not be compressed).

 - Readme: improve documentation when used in Docker.

Below you can see the list of basic software that have been updated. The
only basic software that have not been updated upstream since the previous
Maneage software update are 'bison', 'flock', 'm4', 'pkgconfig', 'unzip'
and 'zip'.

    bash                5.2-rc2            5.2.37
    binutils            2.39               2.43.1
    coreutils           9.1                9.6
    curl                7.84.0             8.11.1
    dash                0.5.11-057cd65     0.5.12
    diffutils           3.8                3.10
    file                5.42               5.46
    findutils           4.9.0              4.10.0
    gawk                5.1.1              5.3.1
    gcc                 12.1.0             14.2.0
    gettext             0.21               0.23.1
    git                 2.37.1             2.48.1
    gmp                 6.2.1              6.3.0
    grep                3.7                3.11
    gzip                1.12               1.13
    help2man            1.49.2             1.49.3
    isl                 0.24               0.27
    less                590                668
    libiconv            1.17               1.18
    libtool             2.4.7              2.5.4
    libunistring        1.0                1.3
    libxml2             2.9.12             2.13.5
    lzip                1.23               1.25
    make                4.3                4.4.1
    mpc                 1.2.1              1.3.1
    mpfr                4.1.0              4.2.1
    nano                6.4                8.3
    ncurses             6.3                6.5
    openssl             3.0.5              3.4.0
    perl                5.36.0             5.40.1
    podlators           4.14               6.0.2
    readline            8.2-rc2            8.2.13
    sed                 4.8                4.9
    tar                 1.34               1.35
    texinfo             6.8                7.2
    wget                1.21.2             1.25.0
    which               2.21               2.23
    xz                  5.2.5              5.6.3
    zlib                1.2.11             1.3.1

The high-level software that have been updated.

    autoconf            2.71               2.72
    automake            1.16.5             1.17
    cfitsio             4.1.0              4.5.0
    cmake               3.24.0             3.31.5
    expat               2.4.1              2.6.4
    fontconfig          2.14.0             2.16.0
    freetype            2.11.0             2.13.3
    ghostscript         9.56.1             10.04.0
    gnuastro            0.18               0.23
    gsl                 2.7                2.8
    libffi              3.4.2              3.4.7
    libgit2             1.3.0              1.9.0
    libice              1.0.10             1.1.2
    libidn              1.38               1.42
    libjpeg             9e                 9f
    libpaper            1.1.28             1.1.29
    libpng              1.6.37             1.6.46
    libpthread-stubs    0.4                0.5
    libsm               1.2.3              1.2.5
    libtiff             4.4.0              4.7.0
    libxau              1.0.9              1.0.12
    libxcb              1.15               1.17
    libxdmcp            1.1.3              1.1.5
    libxext             1.3.4              1.3.6
    libxt               1.2.1              1.3.1
    python              3.10.6             3.13.12
    util-macros         1.19.3             1.20.2
    wcslib              7.11               8.4
    xcb-proto           1.15               1.17
    xorgproto           2022.1             2024.1
    xtrans              1.4.0              1.5.2

SUMMARY: no change necessary in your project, this commit only involves
changes in how already-existing software are built. Some handy options have
also been added to the top-level project script and the copyright years
have been updated.

Until now, if the host's '/bin/sh' had conflicts with the Maneage
environment, the configuration of Maneage would crash as soon as we entered
the building of high-level software. The full scenario is described in the
comments of the newly added 'reproduce/software/shell/prep-source.sh'. This
is most relevant when building older Maneage'd project in newer
environments.

With this commit, the following changes were made to avoid the problem
above:

  - Maneage edits the source code of all installed software to replace
    '/bin/sh' with Maneage's own shell before the programs are
    built. Through this, we were able to solve the problem described
    above.

  - The portable '#!/usr/bin/env sh' shebangs are now used at the start of
    the scripts that run during configure time so it uses the first
    available shell that it finds in its PATH (the system's before Dash is
    built), then Dash, and after Dash is built, Bash.

  - For TeXLive, since we don't install it from source, it was necessary to
    add the libraries necessary for the local '/bin/sh' in LD_LIBRARY_PATH.

Some high-level options have been added to the './project' script to
simplify certain operations:

  --keep-going: do not stop upon the first crash, but keep going on to
    build targets until all build-able targets have been built. This is
    very useful for debugging large pipelines and allows you to isolate the
    problematic part of your project.

  --highlight-all: equivalent to calling both '--highlight-new' &
    '--highlight-notes'.