Summary: this commit will not affect your project's software or analysis.

Until now, the ending copyright years of all files in Maneage were
2025. But we have already entered 2026 so it is important to update them.

With this commit, the ending copyright year of all files is changed to
2026.

Summary: will not affect existing configuration or analysis.

Until this commit, if the Maneage readline library (used by the shell) did
not match the local readline library, the 'pdflatex' command would crash
when building the template Maneage branch. This was because PGFPlots uses
the local shell and we do not build TeXLive from source within Maneage
yet. Previously, we had added a modification to 'LD_LIBRARY_PATH' before
the 'pdflatex' command to use 'sys_library_sh_path', but the variable was
never initilized (because it was only available within the configuration
phase)!

With this commit, the problem is solved by adding that variable within
'LOCAL.conf' so it is available in the analysis phase of a Maneage'd
project also. In order to do so, the set of commands that set the
'sys_library_sh_path' variable within 'configure.sh' was brought up before
the creation of 'LOCAL.conf'.

In case you have had this problem, the solution is to delete
'.build/software/config/LOCAL.conf' and re-configure your project by
manually setting the necessary directories. In this way, the 'LOCAL.conf'
is created again with the new variable.

This bug was found with the help of Faezeh Bidjarchian.

Summary: will not affect existing configuration or analysis.

Until now, some of the comments in the 'dist' rule of initialize (to build
the tarball of the project) were out of date (for example referenced the
'LOCAL.conf' file in the top source directory) or not sufficiently clear.

With this commit, those comments have been edited to improve clarity.

Summary: will not affect existing configuration or analysis.

Until now, the 'clean' rule of 'initialize.mk' paid careful attention to
not remove certain files that were created by the project configuration
phase in the analysis directory. However, in the last few commits, we have
fully decoupled the file creation between the analysis and configure phases
of Maneage. Therefore that extra complexity is no longer necessary also.

With this commit, when './project make clean' is given, besides the
possible LaTeX generated files in the top source directory, we simply
delete the full 'build/analysis/' directory.

Summary: after merging this commit into your project, it should be
re-configured since the location of software installation files like
'LOCAL.conf' or the LaTeX macros of the software environment have
changed. But it should not affect the analysis phase of your project.

Until this commit, it was not possible to run a pre-built Maneage'd project
(in a container) on a newly cloned Maneage'd project source. This was
because the containers should be read-only, but during the various checks
of the configuration (to verify that we are using the same software
environment in the container and the source), we were writing/testing many
things in the build directory, and 'LOCAL.conf' which was actually in the
source directory!

Furthermore, the '.local' and '.build' were built at configure time, making
it hard to run the same container from a newly cloned Maneage'd project. To
make things harder for the scenario above, the 'configure.sh' script would
pause on every message and didn't have a quiet mode (making it practically
impossible to run './project configure' before './project make' on every
container run).

With this commit, all these issues have been addressed and it is now
possible to simply get a built container, clone a Maneage'd project and run
the analysis (using the built environment of the container that is verified
on every run). The respective changes/additions are described below:

 - The high-level container scripts ('apptainer.sh' and 'docker.sh', along
   with their READMEs) have been moved to the 'reproduce/software/shell'
   directory and the old 'reproduce/software/containers' directory has been
   deleted. This is because we have classified the software files by their
   language/format and the container scripts are scripts in the end.

 - The './project' script:

    - Now has two extra options: '--quiet' and '--no-pause'. Both are
      directly passed to the 'configure.sh' script. They will respectively
      disable any informative printed message or any pause after that
      message (if it is printed).

    - The '--build-dir' option is now also relevant for './project make':
      when it is given, it will re-create the two '.build' and '.local'
      symbolic links at the top source directory in all scenarios
      ('configure', 'make' or 'shell'). This will allow both the
      configuration, analysis and shell phases to safely assume they exist
      and match the user's desire at run-time.

    - The build/analysis directory's sub-directories that need to be built
      before 'top-make.mk' are now built in a separate function to help in
      readability.

 - The 'configure.sh' script:

    - For developers: a new 'check_elapsed' variable has been defined that
      will enable the newly added 'elapsed_time_from_prev_step'
      function. This function should be used from now on at the end of
      every major step to help find bottlenecks.

 - The targets of the software in 'pre-make-build.sh' now also have the
   version of the software in their file name. Until now, they didn't have
   the version, so there was no way to detect if the software has been
   updated or not in the source. For Lzip and Make (that also get built
   after GCC), the ones in this script have a '-pre-make' suffix also.

 - 'Local.conf.in' now has descriptions for every variable.

 - The '-std=gnu17' option is now used instead of '-std=c17' for basic
   software that cannot be built without specifying the C standard in GCC
   15.1 (described in previous commit: 2881fc0a6205). See [1] for more
   details; in summary: '-std=gnu17' is also supported on macOS's Clang and
   has some features that 'pkg-config' needs

 - Generally: some longer code lines have been broken or indentation
   decreased to fit the 75 character line length. This has not reduced
   readability however. For example the long 'echo' commands are now
   replaced by multiple 'printf's, or the indentation is still clearly
   visible.

The seeds of the work on this commit started by a branch containing three
commits by Giacomo Lorenzetti (133 insertions, 100 deletions). Upon merging
with the main 'maneage' branch, they were generalized and re-organized to
become this commit.

The following issues have also been addressed with this commit:

 - The LaTeX calls (during the building of 'paper.pdf') do not contain
   Maneage'd dynamic libraries. This is because we don't build the LaTeX
   binaries from source, an TeXLive manager uses the host environment.

 - The 'docker.sh' script:

    - Adds the '--project-name' option: its internal variable existed, but
      the option for the user to define it at run-time was not.

    - Ported to macOS: it does not check being a member of the 'docker'
      group, and finds the number of threads using macOS-specific tools.

 - The 'apptainer.sh' script:

    - Now installs 'wget' in the base container also (necessary when the
      user doesn't have the tarballs).

[1] https://savannah.nongnu.org/bugs/?67068#comment2

Summary: it is necessary to re-configure your project (just running
'./project configure -e', not deleting 'build/software' to re-build
software) after this commit, see "Affected files" item below).

Until now, we only had a relatively long set of manual instructions for
building Maneage within Docker in the top-level README. This was hard to
automate, focing Maneage users to write custom commands based on the
instructions and maintain those scripts outside of Maneage. As a result,
experience could not be shared between projects (or at most in the README
file!).

With this commit, a new 'reproduce/software/containers' directory has been
created within Maneage that contains two scripts (with a unified interface)
greatly simplifying the building of the project's software environment
within a container (one script for Apptainer and one for Docker). Two
READMEs have been added for each container to help in their first time
usage. Also, the old checklist within the main README has been replaced
with a short introduction on containers and points the interested readers
to the custom README of each container technology.

Since we wanted the containers to be read-only after build, we needed to
fully decouple the 'build/software' and 'build/analysis', such that
'./project configure' only writes to the former and './project make' only
writes the latter. The file and directories mentioned in the affected files
are cases that both project phases was writing to the 'build/software' and
'build/analysis' directories.

Affected files: 'preparation-done.mk' and 'lockdir' which were previously
in the 'build/software' directory are now made during the 'make' phase and
the 'configure' phase no longer builds the 'build/analysis' or anything
within it. Also, the software version LaTeX macros (which were previously
written during the 'configure' phase in the 'analysis' directory) are now
written in the software directory and copied into the analysis for usage in
LaTeX while building the paper.

Other minor additions in this commit:

  - The './project' script has a new '--timing' option to write the
    starting and ending times of the project in a file. It also builds the
    high-level analysis directories when './project make' is called (but
    before calling 'top-make.mk'.

  - The 'tar' calls in the custom build commands of the software building
    Makefiles now have the '--no-same-owner --no-same-permissions' options
    like the 'tar' call within the 'uncompress' function of
    'build-rules.mk'.

This commit was originally written by Giacomo Lorenzetti only for Apptainer
on the registered commit date. It was later re-implemented from scratch by
Mohammad Akhlaghi to have a unified interface for both Apptainer and Docker
and merged into Maneage on 2025-04-23.

SUMMARY: no change necessary in your project, this commit only involves
changes in how already-existing software are built. Some handy options have
also been added to the top-level project script and the copyright years
have been updated.

Until now, if the host's '/bin/sh' had conflicts with the Maneage
environment, the configuration of Maneage would crash as soon as we entered
the building of high-level software. The full scenario is described in the
comments of the newly added 'reproduce/software/shell/prep-source.sh'. This
is most relevant when building older Maneage'd project in newer
environments.

With this commit, the following changes were made to avoid the problem
above:

  - Maneage edits the source code of all installed software to replace
    '/bin/sh' with Maneage's own shell before the programs are
    built. Through this, we were able to solve the problem described
    above.

  - The portable '#!/usr/bin/env sh' shebangs are now used at the start of
    the scripts that run during configure time so it uses the first
    available shell that it finds in its PATH (the system's before Dash is
    built), then Dash, and after Dash is built, Bash.

  - For TeXLive, since we don't install it from source, it was necessary to
    add the libraries necessary for the local '/bin/sh' in LD_LIBRARY_PATH.

Some high-level options have been added to the './project' script to
simplify certain operations:

  --keep-going: do not stop upon the first crash, but keep going on to
    build targets until all build-able targets have been built. This is
    very useful for debugging large pipelines and allows you to isolate the
    problematic part of your project.

  --highlight-all: equivalent to calling both '--highlight-new' &
    '--highlight-notes'.

SUMMARY: just house-cleaning, no need to do anything major in your
branch. Just update the copyright years in files that you have added.

Until now, the latest copyright years of the whole Maneage source code was
2022! As of this commit, we have already moved to 2023 for 5 months!
Furthermore, there were a few other minor issues that needed correction:

 - The URL to download input datasets wasn't quoted in 'initialize.mk' or
   the download script! As a result, when the input URL had characters that
   are meaningful to the shell (like '&'), the download command would not
   work.

 - The only program that had 'make check' in the 'basic.mk' programs was
   MPFR. At that stage, we still haven't built our own compiler at this
   stage, this is not accurate.

 - The 'pyerfa' and 'extension-helpers' packages in Python need
   'setuptools_scm' on some systems. But until now, it was not in the list
   of their prerequisites.

With this commit, all the issues above have been corrected.

SUMMARY: Nothing special is necessary for your existing projects. This
commit just addds two new features (read the commit description for more):
  1. To provide a user and password to servers that need authentication
     before they allow downloading of proprietary data,
  2. To use the FITS Standard's DATASUM for file verification (for cases
     where the file is not static on the server, and is generated upon
     receiving your download request).

Until now, Maneage didn't have any infrastructure for databases that
require authentication (through a user or password, when calling
'wget'). Furthermore, when the downloaded file is automatically generated
by the server upon request, the server usually adds metadata (like file
date, or query number and etc) in the header. Therefore the simple SHA256
checksum of the file would differ on every download! This made it very hard
to verify if the data (not headers) are unchanged.

With this commit, both these problems have been addressed:

 - Server authentication: the 'reproduce/software/config/LOCAL.conf' now
   contains three new variables for this purpose. With them, you can give
   your username and password, along with the authentication method of the
   server. The comments on top of these three variables give a full
   description of their usage.

 - Verifying only the data in a file (ignoring the headers): The
   'reproduce/analysis/config/INPUTS.conf' now accepts two new optional
   variables for each input file using the FITS standard's DATASUM
   convention: 'INPUT-%-fitsdatasum' and 'INPUT-%-fitshdu'. If the SHA256
   isn't specified for a file, Maneage will use these to verify the
   file. With the latter, you specify the HDU of the data you want to
   verify and with the former you give the DATASUM value for that HDU. As
   the name suggests, this is only valid for FITS files. If we find other
   formats that support a similar behavior, we can add this feature for
   those formats also. This is also thoroughly discussed in the comments of
   'reproduce/analysis/config/INPUTS.conf'.

This commit was done with the help of Pedram Ashofte Ardakani, Sepideh
Eskandarlou and Mohammadreza Khellat.

Until now, when some configuration files were generated in the preparation
phase, only the auto-generated (in the preparation phase) Makefiles (with
the suffix '*.mk') were always included in 'initialize.mk'. As a
consequence, if there were any configuration files (with suffix '*.conf'),
they would not be automatically added, and it was necessary to manually
include them.

Since auto-generated configuration files are also one common output of the
preparation phase of a project, it is better to include them automatically.

With this commit, the '*.conf' configuration files generated in the
preparation phase are added by 'initialize.mk' automatically (if
necessary).

In the process, the comments in the final target of
'reproduce/analysis/make/prepare.mk' were updated to be more clear.