IMPORTANT: Apptainer and Docker containers, minor restructuringHEADmaneage

Summary: it is necessary to re-configure your project (just running
'./project configure -e', not deleting 'build/software' to re-build
software) after this commit, see "Affected files" item below).

Until now, we only had a relatively long set of manual instructions for
building Maneage within Docker in the top-level README. This was hard to
automate, focing Maneage users to write custom commands based on the
instructions and maintain those scripts outside of Maneage. As a result,
experience could not be shared between projects (or at most in the README
file!).

With this commit, a new 'reproduce/software/containers' directory has been
created within Maneage that contains two scripts (with a unified interface)
greatly simplifying the building of the project's software environment
within a container (one script for Apptainer and one for Docker). Two
READMEs have been added for each container to help in their first time
usage. Also, the old checklist within the main README has been replaced
with a short introduction on containers and points the interested readers
to the custom README of each container technology.

Since we wanted the containers to be read-only after build, we needed to
fully decouple the 'build/software' and 'build/analysis', such that
'./project configure' only writes to the former and './project make' only
writes the latter. The file and directories mentioned in the affected files
are cases that both project phases was writing to the 'build/software' and
'build/analysis' directories.

Affected files: 'preparation-done.mk' and 'lockdir' which were previously
in the 'build/software' directory are now made during the 'make' phase and
the 'configure' phase no longer builds the 'build/analysis' or anything
within it. Also, the software version LaTeX macros (which were previously
written during the 'configure' phase in the 'analysis' directory) are now
written in the software directory and copied into the analysis for usage in
LaTeX while building the paper.

Other minor additions in this commit:

  - The './project' script has a new '--timing' option to write the
    starting and ending times of the project in a file. It also builds the
    high-level analysis directories when './project make' is called (but
    before calling 'top-make.mk'.

  - The 'tar' calls in the custom build commands of the software building
    Makefiles now have the '--no-same-owner --no-same-permissions' options
    like the 'tar' call within the 'uncompress' function of
    'build-rules.mk'.

This commit was originally written by Giacomo Lorenzetti only for Apptainer
on the registered commit date. It was later re-implemented from scratch by
Mohammad Akhlaghi to have a unified interface for both Apptainer and Docker
and merged into Maneage on 2025-04-23.

1 files changed, 57 insertions, 7 deletions

diff --git a/project b/project
index ac801b8..c30bfbf 100755
--- a/project
+++ b/project
@@ -33,6 +33,7 @@ set -e
 jobs=0     # 0 is for the default for the 'configure.sh' script.
 group=
 debug=
+timing=0
 host_cc=0
 offline=
 operation=
@@ -89,7 +90,7 @@ RECOMMENDATION: If this is the first time you are configuring this
 template, please don't use the options and let the script explain each
 parameter in full detail by simply running './project configure'.
 
-Project 'make' special features.
+Project 'make' special tagets
   ./project make           Build the project on one thread
   ./project make -jN       Built the project in parallel on N threads.
   ./project make clean     Clean all files generated by 'make' (not software).
@@ -127,6 +128,7 @@ Configure and Make options:
 
 Make (analysis) options:
   -p, --prepare-redo       Re-do preparation (only done automatically once).
+  -t, --timing             Starting and ending times written in 'timing.txt'.
 
 Make (final PDF) options:
       --refresh-bib        Force refresh the bibliography.
@@ -216,6 +218,8 @@ do
   # Make options (analysis):
   -p|--prepare-redo)      prepare_redo=1;                                                  shift;;
   -p=*|--prepare-redo=*)  on_off_option_error --prepare-redo;                              shift;;
+  -t|--timing)            timing=1;                                                        shift;;
+  -t=*|--timing=*)        on_off_option_error --timing;                                    shift;;
 
   # Make options (final PDF):
   --refresh-bib)          [ -f tex/src/references.tex ] && touch tex/src/references.tex;   shift;;
@@ -389,6 +393,7 @@ EOF
 
 # Run operations in controlled environment
 # ----------------------------------------
+perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x"
 controlled_env() {
 
     # Get the full address of the build directory:
@@ -423,7 +428,6 @@ controlled_env() {
 
 # Do requested operation
 # ----------------------
-perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x"
 configscript=./reproduce/software/shell/configure.sh
 case $operation in
 
@@ -444,8 +448,11 @@ case $operation in
         # to make sure they have them, we are activating the executable
         # flags by default here every time './project configure' is run. If
         # any other file in your project needs such flags, add them here.
-        chmod +x reproduce/software/shell/* reproduce/software/config/*.sh \
-              reproduce/analysis/bash/*
+        if ! [ -x reproduce/software/shell/configure.sh ]; then
+            chmod +x reproduce/analysis/bash/* \
+                     reproduce/software/shell/* \
+                     reproduce/software/config/*.sh
+        fi
 
         # If the user requested, clean the TeX directory from the extra
         # (to-be-built) directories that may already be there (and will not
@@ -499,22 +506,62 @@ case $operation in
             configuration_necessary
         fi
 
+        # Make sure that the necessary analysis directories directory exist
+        # in the build directory. These will be necessary in various phases
+        # of hte analysis and having them inside the lower-level Make steps
+        # will require setting them as prerequisites for many basic jobs
+        # (thus making the Makefiles harder to read and add potentials for
+        # bugs: forgetting to add them for example). Also, we don't want
+        # the configure phase to make any edits in the analysis directory,
+        # so they are not built there.
+        badir=.build/analysis
+        texdir=$badir/tex
+        mtexdir=$texdir/macros
+        if ! [ -d $badir   ]; then mkdir $badir;   fi
+        if ! [ -d $texdir  ]; then mkdir $texdir;  fi
+        if ! [ -d $mtexdir ]; then mkdir $mtexdir; fi
+
+        # TeX build directory. If built in a group scenario, the TeX build
+        # directory must be separate for each member (so they can work on their
+        # relevant parts of the paper without conflicting with each other).
+        if [ "x$maneage_group_name" = x ]; then
+            texbdir="$texdir"/build
+        else
+            user=$(whoami)
+            texbdir="$texdir"/build-$user
+        fi
+        tikzdir="$texbdir"/tikz
+        if ! [ -L tex/build ]; then ln -s "$(pwd -P)/$texdir" tex/build;  fi
+        if ! [ -L tex/tikz  ]; then ln -s "$(pwd -P)/$tikzdir" tex/tikz;  fi
+
+        # Register the start of this run (we are appending the new
+        # information so previous information is preserved until the user
+        # intentionally deletes/cleans it).
+        if [ $timing = 1 ]; then echo "start: $(date)" >> timing.txt; fi
+
         # Run data preparation phase (optionally build Makefiles with
         # special values for optimizing the main 'top-make.mk'). But note
         # that data preparation is only done automatically the first time
-        # the project is built (when '.build/software/preparation-done.mk'
+        # the project is built (when '.build/analysis/preparation-done.mk'
         # doesn't yet exist). After that, if the user wants to re-do the
         # preparation they have to use the '--prepare-redo' option.
-        if ! [ -f .build/software/preparation-done.mk ] \
+        if ! [ -f .build/analysis/preparation-done.mk ] \
                 || [ x"$prepare_redo" = x1 ]; then
             controlled_env reproduce/analysis/make/top-prepare.mk
         fi
 
-        # Run the actual project.
+        # Call top-make (highest level analysis Makefile).
         controlled_env reproduce/analysis/make/top-make.mk
+
+        # Register the time of the project's ending.
+        if [ $timing = 1 ]; then echo "end:   $(date)" >> timing.txt; fi
         ;;
 
 
+
+
+
+    # Interactive shell of Maneage.
     shell)
 
         # Make sure the configure script has been completed properly
@@ -550,6 +597,9 @@ case $operation in
         ;;
 
 
+
+
+
     # Operation not specified.
     *)
         cat <<EOF