aboutsummaryrefslogtreecommitdiff
path: root/project
diff options
context:
space:
mode:
authorGiacomo Lorenzetti <glorenzetti@cefca.es>2025-04-03 15:21:16 +0200
committerMohammad Akhlaghi <mohammad@akhlaghi.org>2025-05-12 10:59:26 +0200
commitdf9e291826fbc7e717b40d2d07f1d7607a2f2455 (patch)
treed40c7aeb6b7ef6b09bb7df21080606b58245819a /project
parent2881fc0a6205d593512458c24f3b681d12921005 (diff)
IMPORTANT: software configuration optimized and better modularized
Summary: after merging this commit into your project, it should be re-configured since the location of software installation files like 'LOCAL.conf' or the LaTeX macros of the software environment have changed. But it should not affect the analysis phase of your project. Until this commit, it was not possible to run a pre-built Maneage'd project (in a container) on a newly cloned Maneage'd project source. This was because the containers should be read-only, but during the various checks of the configuration (to verify that we are using the same software environment in the container and the source), we were writing/testing many things in the build directory, and 'LOCAL.conf' which was actually in the source directory! Furthermore, the '.local' and '.build' were built at configure time, making it hard to run the same container from a newly cloned Maneage'd project. To make things harder for the scenario above, the 'configure.sh' script would pause on every message and didn't have a quiet mode (making it practically impossible to run './project configure' before './project make' on every container run). With this commit, all these issues have been addressed and it is now possible to simply get a built container, clone a Maneage'd project and run the analysis (using the built environment of the container that is verified on every run). The respective changes/additions are described below: - The high-level container scripts ('apptainer.sh' and 'docker.sh', along with their READMEs) have been moved to the 'reproduce/software/shell' directory and the old 'reproduce/software/containers' directory has been deleted. This is because we have classified the software files by their language/format and the container scripts are scripts in the end. - The './project' script: - Now has two extra options: '--quiet' and '--no-pause'. Both are directly passed to the 'configure.sh' script. They will respectively disable any informative printed message or any pause after that message (if it is printed). - The '--build-dir' option is now also relevant for './project make': when it is given, it will re-create the two '.build' and '.local' symbolic links at the top source directory in all scenarios ('configure', 'make' or 'shell'). This will allow both the configuration, analysis and shell phases to safely assume they exist and match the user's desire at run-time. - The build/analysis directory's sub-directories that need to be built before 'top-make.mk' are now built in a separate function to help in readability. - The 'configure.sh' script: - For developers: a new 'check_elapsed' variable has been defined that will enable the newly added 'elapsed_time_from_prev_step' function. This function should be used from now on at the end of every major step to help find bottlenecks. - The targets of the software in 'pre-make-build.sh' now also have the version of the software in their file name. Until now, they didn't have the version, so there was no way to detect if the software has been updated or not in the source. For Lzip and Make (that also get built after GCC), the ones in this script have a '-pre-make' suffix also. - 'Local.conf.in' now has descriptions for every variable. - The '-std=gnu17' option is now used instead of '-std=c17' for basic software that cannot be built without specifying the C standard in GCC 15.1 (described in previous commit: 2881fc0a6205). See [1] for more details; in summary: '-std=gnu17' is also supported on macOS's Clang and has some features that 'pkg-config' needs - Generally: some longer code lines have been broken or indentation decreased to fit the 75 character line length. This has not reduced readability however. For example the long 'echo' commands are now replaced by multiple 'printf's, or the indentation is still clearly visible. The seeds of the work on this commit started by a branch containing three commits by Giacomo Lorenzetti (133 insertions, 100 deletions). Upon merging with the main 'maneage' branch, they were generalized and re-organized to become this commit. The following issues have also been addressed with this commit: - The LaTeX calls (during the building of 'paper.pdf') do not contain Maneage'd dynamic libraries. This is because we don't build the LaTeX binaries from source, an TeXLive manager uses the host environment. - The 'docker.sh' script: - Adds the '--project-name' option: its internal variable existed, but the option for the user to define it at run-time was not. - Ported to macOS: it does not check being a member of the 'docker' group, and finds the number of threads using macOS-specific tools. - The 'apptainer.sh' script: - Now installs 'wget' in the base container also (necessary when the user doesn't have the tarballs). [1] https://savannah.nongnu.org/bugs/?67068#comment2
Diffstat (limited to 'project')
-rwxr-xr-xproject327
1 files changed, 199 insertions, 128 deletions
diff --git a/project b/project
index c30bfbf..f2986fb 100755
--- a/project
+++ b/project
@@ -33,6 +33,7 @@ set -e
jobs=0 # 0 is for the default for the 'configure.sh' script.
group=
debug=
+quiet=0
timing=0
host_cc=0
offline=
@@ -43,6 +44,7 @@ keep_going=
check_config=
make_targets=
software_dir=
+pauseformsg=1
clean_texdir=0
prepare_redo=0
highlightnew=0
@@ -107,16 +109,18 @@ Project 'make' special tagets
With the options below you can modify the default behavior.
Configure options:
+ --all-highlevel Build all high-level software (for development).
-b, --build-dir=STR Top directory to build the project in.
- -e, --existing-conf Use (possibly existing) local configuration.
- --host-cc Use host system's C compiler, don't build GCC.
- -i, --input-dir=STR Directory containing input datasets (optional).
- -s, --software-dir=STR Directory containing necessary software tarballs.
--check-config During configuration, show what is being built.
--clean-texdir Remove possibly existing build-time subdirectories
under the project's 'tex/' directory (can happen
when source is from arXiv for example).
- --all-highlevel Build all high-level software (for development).
+ -e, --existing-conf Use (possibly existing) local configuration.
+ -i, --input-dir=STR Directory containing input datasets (optional).
+ --host-cc Use host system's C compiler, don't build GCC.
+ --quiet Do not print basic info messages (with '-e').
+ --no-pause Do not sleep/pause after basic info messages.
+ -s, --software-dir=STR Directory containing necessary software tarballs.
Configure and Make options:
-d, --debug[=FLAGS] In configure: use -j1, no -k, and no Zenodo check.
@@ -180,67 +184,71 @@ do
shell) func_operation_set $1; shift;;
# Configure options:
- -b|--build-dir) build_dir="$2"; check_v "$1" "$build_dir"; shift;shift;;
- -b=*|--build-dir=*) build_dir="${1#*=}"; check_v "$1" "$build_dir"; shift;;
- -b*) build_dir=$(echo "$1" | sed -e's/-b//'); check_v "$1" "$build_dir"; shift;;
- -e|--existing-conf) existing_conf=1; shift;;
+ -e|--existing-conf) existing_conf=1; shift;;
-e*|--existing-conf=*) on_off_option_error --existing-conf -e;;
- --host-cc) host_cc=1; shift;;
+ --host-cc) host_cc=1; shift;;
--host-cc=*) on_off_option_error --host-cc;;
- --offline) offline=1; shift;;
+ --offline) offline=1; shift;;
--offline=*) on_off_option_error --offline;;
- -i|--input-dir) input_dir="$2"; check_v "$1" "$input_dir"; shift;shift;;
- -i=*|--input-dir=*) input_dir="${1#*=}"; check_v "$1" "$input_dir"; shift;;
- -i*) input_dir=$(echo "$1" | sed -e's/-i//'); check_v "$1" "$input_dir"; shift;;
- -s|--software-dir) software_dir="$2"; check_v "$1" "$software_dir"; shift;shift;;
- -s=*|--software-dir=*) software_dir="${1#*=}"; check_v "$1" "$software_dir"; shift;;
- -s*) software_dir=$(echo "$1" | sed -e's/-s//'); check_v "$1" "$software_dir"; shift;;
- --check-config) check_config=1; shift;;
+ -i|--input-dir) input_dir="$2"; check_v "$1" "$input_dir"; shift;shift;;
+ -i=*|--input-dir=*) input_dir="${1#*=}"; check_v "$1" "$input_dir"; shift;;
+ -i*) input_dir=$(echo "$1" | sed -e's/-i//'); check_v "$1" "$input_dir"; shift;;
+ -s|--software-dir) software_dir="$2"; check_v "$1" "$software_dir"; shift;shift;;
+ -s=*|--software-dir=*) software_dir="${1#*=}"; check_v "$1" "$software_dir"; shift;;
+ -s*) software_dir=$(echo "$1" | sed -e's/-s//'); check_v "$1" "$software_dir"; shift;;
+ --check-config) check_config=1; shift;;
--check-config=*) on_off_option_error --check-config;;
- --clean-texdir) clean_texdir=1; shift;;
+ --clean-texdir) clean_texdir=1; shift;;
--clean-texdir=*) on_off_option_error --clean-texdir;;
- --all-highlevel) all_highlevel=1; shift;;
+ --all-highlevel) all_highlevel=1; shift;;
--all-highlevel=*) on_off_option_error --all-highlevel;;
+ --no-pause) pauseformsg=0; shift;;
+ --no-pause=*) on_off_option_error --no-pause;;
+ --quiet) quiet=1; shift;;
+ --quiet=*) on_off_option_error --quiet;;
# Configure and Make options:
- -g|--group) group="$2"; check_v group "$group"; shift;shift;;
- -g=*|--group=*) group="${1#*=}"; check_v group "$group"; shift;;
- -g*) group=$(echo "$1" | sed -e's/-g//'); check_v group "$group"; shift;;
- -j|--jobs) jobs="$2"; check_v jobs "$jobs"; shift;shift;;
- -j=*|--jobs=*) jobs="${1#*=}"; check_v jobs "$jobs"; shift;;
- -j*) jobs=$(echo "$1" | sed -e's/-j//'); check_v jobs "$jobs"; shift;;
- -k|--keep-going) keep_going="--keep-going"; shift;;
- -k=*|--keep-going=*) on_off_option_error --keep-going -k;;
- -k*) on_off_option_error --keep-going -k;;
- -'?'|--help) print_help; exit 0;;
- -'?'*|--help=*) on_off_option_error --help -?;;
+ -b|--build-dir) build_dir="$2"; check_v "$1" "$build_dir";shift;shift;;
+ -b=*|--build-dir=*) build_dir="${1#*=}"; check_v "$1" "$build_dir";shift;;
+ -b*) build_dir=$(echo "$1" | sed -e's/-b//'); check_v "$1" "$build_dir";shift;;
+ -g|--group) group="$2"; check_v group "$group"; shift;shift;;
+ -g=*|--group=*) group="${1#*=}"; check_v group "$group"; shift;;
+ -g*) group=$(echo "$1" | sed -e's/-g//'); check_v group "$group"; shift;;
+ -j|--jobs) jobs="$2"; check_v jobs "$jobs"; shift;shift;;
+ -j=*|--jobs=*) jobs="${1#*=}"; check_v jobs "$jobs"; shift;;
+ -j*) jobs=$(echo "$1" | sed -e's/-j//'); check_v jobs "$jobs"; shift;;
+ -k|--keep-going) keep_going="--keep-going"; shift;;
+ -k=*|--keep-going=*) on_off_option_error --keep-going -k;;
+ -k*) on_off_option_error --keep-going -k;;
+ -'?'|--help) print_help; exit 0;;
+ -'?'*|--help=*) on_off_option_error --help -?;;
# Make options (analysis):
- -p|--prepare-redo) prepare_redo=1; shift;;
- -p=*|--prepare-redo=*) on_off_option_error --prepare-redo; shift;;
- -t|--timing) timing=1; shift;;
- -t=*|--timing=*) on_off_option_error --timing; shift;;
+ -p|--prepare-redo) prepare_redo=1; shift;;
+ -p=*|--prepare-redo=*) on_off_option_error --prepare-redo; shift;;
+ -t|--timing) timing=1; shift;;
+ -t=*|--timing=*) on_off_option_error --timing; shift;;
# Make options (final PDF):
- --refresh-bib) [ -f tex/src/references.tex ] && touch tex/src/references.tex; shift;;
- --highlight-all) highlightnew=1; highlightnotes=1; shift;;
- --highlight-all=*) on_off_option_error --highlight-new;;
- --highlight-new) highlightnew=1; shift;;
- --highlight-new=*) on_off_option_error --highlight-new;;
- --highlight-notes) highlightnotes=1; shift;;
- --highlight-notes=*) on_off_option_error --highlight-notes;;
- -d|--debug) if [ x$operation = x ]; then
- echo "Please set the operation before calling '--debug'"; exit 1
- elif [ x$operation = xconfigure ]; then debug=a; shift;
- elif [ x$operation = xmake ]; then
- if [ x"$2" = x ]; then
- echo "In make-mode, '--debug' needs a value; see GNU Make manual"; exit 1
- else debug="$2"; check_v debug "$debug"; shift;shift; fi
- else
- echo "Operation '$operation' not recognized, please use 'configure' or 'make'"
- fi;;
- -d=*|--debug=*) debug="${1#*=}"; check_v debug "$debug"; shift;;
- -d*) debug=$(echo "$1" | sed -e's/-d//'); check_v debug "$debug"; shift;;
+ --refresh-bib) [ -f tex/src/references.tex ] && touch tex/src/references.tex; shift;;
+ --highlight-all) highlightnew=1; highlightnotes=1; shift;;
+ --highlight-all=*) on_off_option_error --highlight-new;;
+ --highlight-new) highlightnew=1; shift;;
+ --highlight-new=*) on_off_option_error --highlight-new;;
+ --highlight-notes) highlightnotes=1; shift;;
+ --highlight-notes=*) on_off_option_error --highlight-notes;;
+ -d|--debug) if [ x$operation = x ]; then
+ echo "Please set the operation before calling '--debug'"; exit 1
+ elif [ x$operation = xconfigure ]; then debug=a; shift;
+ elif [ x$operation = xmake ]; then
+ if [ x"$2" = x ]; then
+ echo "In make-mode, '--debug' needs a value; see GNU Make manual"; exit 1
+ else debug="$2"; check_v debug "$debug"; shift;shift; fi
+ else
+ echo "Operation '$operation' not recognized, please use 'configure' or 'make'"
+ fi;;
+ -d=*|--debug=*) debug="${1#*=}"; check_v debug "$debug"; shift;;
+ -d*) debug=$(echo "$1" | sed -e's/-d//'); check_v debug "$debug"; shift;;
# Unrecognized option:
-*) echo "$scriptname: unknown option '$1'"; exit 1;;
@@ -294,8 +302,8 @@ EOF
ls $coloropt .build/software/build-tmp || junk=1;
fi
- # Make the temporary directory, delete its contents, then put new
- # links of all built software.
+ # Make the temporary directory, delete its contents, then put
+ # new links of all built software.
if ! [ -d $checkdir ]; then mkdir $checkdir; fi
rm -f $checkdir/*
@@ -316,10 +324,11 @@ EOF
if [ $printresults = 1 ]; then
echo "--- Last 5 packages that were built:"
- # Then sort all the links based on the most recent dates of the
- # files they link to (with '-L').
+ # Then sort all the links based on the most recent dates of
+ # the files they link to (with '-L').
ls -Llt $checkdir \
- | awk '/^-/ && c++<5 {printf "[at %s] %s\n", $(NF-1), $NF}'
+ | awk '/^-/ && c++<5 {printf "[at %s] %s\n", \
+ $(NF-1), $NF}'
fi
else
cat <<EOF
@@ -343,14 +352,15 @@ fi
-# Basic group settings
-# --------------------
+# Group check
+# -----------
if ! [ x$group = x ]; then
# Check if group is usable.
if ! sg "$group" "echo Group \'$group\' exists"; then
- echo "$scriptname: '$group' is not a usable group name on this system.";
- echo "(TIP: you can use the 'groups' command to see your groups)"
+ printf "$scriptname: '$group' is not a usable group name on "
+ printf "this system. TIP: you can use the 'groups' command "
+ printf "to see your groups)\n"
exit 1
fi
@@ -362,9 +372,45 @@ fi
-# Error when configuration isn't run
-configuration_necessary() {
- cat <<EOF
+# Build directory symbolic links
+# ------------------------------
+#
+# The source directory will contain two symbolic links that point to the
+# build directory:
+
+# - .build: the top build directory.
+#
+# - .local: the second to the top software installed directory. They are
+# used during the configuration phase to simplify commands and are also
+# very useful during the development of a maneage'd project (to easily
+# get to the build directory or execut Maneage'd software).
+#
+# This needs to be done on every run because:
+# - './project configure' can be run with a new build directory, and
+# keeping the old '.build' conflicts with the new build directory
+# that the user gave.
+# - './project make' or './project shell' (within a newly cloned source
+# directory from inside a container): the links do not exist but have
+# to be set to the container's build directory.
+# - This is not an expensive operation.
+if ! [ x"$build_dir" = x ]; then
+ rm -f .build .local
+ ln -s $build_dir .build
+ ln -s $build_dir/software/installed .local
+fi
+
+
+
+
+
+# Function to validate configuration
+# ----------------------------------
+#
+# Check if the configuration is missing/incomplete
+configuration_check() {
+ confdone=software/config/hardware-parameters.tex
+ if ! [ -f .build/$confdone ]; then
+ cat <<EOF
The project is either (1) not configured on this system, or (2) the
configuration wasn't successful.
@@ -384,15 +430,69 @@ If there was a problem, please let us know by filling this online form:
http://savannah.nongnu.org/support/?func=additem&group=reproduce
EOF
- exit 1
+ exit 1
+ fi
}
-# Run operations in controlled environment
-# ----------------------------------------
+# Function for TeX Preparations
+# -----------------------------
+#
+# Make sure that the necessary analysis directories directory exist in the
+# build directory. These will be necessary in various phases of hte
+# analysis and having them inside the lower-level Make steps will require
+# setting them as prerequisites for many basic jobs (thus making the
+# Makefiles harder to read and add potentials for bugs: forgetting to add
+# them for example). Also, we don't want the configure phase to make any
+# edits in the analysis directory, so they are not built there.
+tex_preparations () {
+
+ # Extract the location of the build directory.
+ bdir=$(.local/bin/realpath .build)
+
+ # We are using our custom-built 'mkdir' which is guaranteed to have the
+ # '-p' option (that will also build intermediate directories) and won't
+ # complain if the directory already exists.
+ badir=$bdir/analysis
+ texdir=$badir/tex
+ btexdir=$texdir/build
+ tikzdir=$btexdir/tikz
+ mtexdir=$texdir/macros
+ .local/bin/mkdir -p $mtexdir $btexdir $tikzdir
+
+ # If 'tex/build' and 'tex/tikz' are symbolic links then 'rm -f'
+ # will delete them and we can continue. However, when the project
+ # is being built from the tarball (from arXiv for example), these
+ # two are not symbolic links but actual directories with the
+ # necessary built-components to build the PDF in them. In this
+ # case, because 'tex/build' is a directory, 'rm -f' will fail, so
+ # we'll just rename the two directories (as backup) and let the
+ # project build the proper symbolic links afterwards.
+ if rm -f tex/build; then
+ rm -f tex/tikz
+ else
+ mv tex/tikz tex/tikz-from-tarball
+ mv tex/build tex/build-from-tarball
+ fi
+
+ # Build the symbolic links.
+ if ! [ -L tex/tikz ]; then ln -s "$tikzdir" tex/tikz; fi
+ if ! [ -L tex/build ]; then ln -s "$texdir" tex/build; fi
+}
+
+
+
+
+
+# Function to run in controlled environment
+# -----------------------------------------
+#
+# Controlling the environment is necessary for running the analysis. Like
+# the other funcitons here, this is defined to simplify the high-level code
+# within the 'make)' switch statement.
perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x"
controlled_env() {
@@ -464,11 +564,13 @@ case $operation in
# Variables to pass to the configuration script.
export jobs=$jobs
export debug=$debug
+ export quiet=$quiet
export host_cc=$host_cc
export offline=$offline
export build_dir=$build_dir
export input_dir=$input_dir
export scriptname=$scriptname
+ export pauseformsg=$pauseformsg
export maneage_group_name=$group
export software_dir=$software_dir
export existing_conf=$existing_conf
@@ -488,7 +590,8 @@ case $operation in
# creates problems when another group member wants to update
# the software for example. We thus need to manually add the
# group writing flag to all installed software files.
- echo "Enabling group writing permission on all installed software..."
+ printf "Enabling group writing permission on all installed "
+ printf "software...\n"
.local/bin/chmod -R g+w .local/;
fi
;;
@@ -500,43 +603,14 @@ case $operation in
# Batch execution of the project.
make)
- # Make sure the configure script has been completed properly
- # ('configuration-done.txt' exists).
- if ! [ -f .build/software/configuration-done.txt ]; then
- configuration_necessary
- fi
+ # Make sure the configure script is complete and necessary LaTeX
+ # directories are in place.
+ configuration_check
+ tex_preparations
- # Make sure that the necessary analysis directories directory exist
- # in the build directory. These will be necessary in various phases
- # of hte analysis and having them inside the lower-level Make steps
- # will require setting them as prerequisites for many basic jobs
- # (thus making the Makefiles harder to read and add potentials for
- # bugs: forgetting to add them for example). Also, we don't want
- # the configure phase to make any edits in the analysis directory,
- # so they are not built there.
- badir=.build/analysis
- texdir=$badir/tex
- mtexdir=$texdir/macros
- if ! [ -d $badir ]; then mkdir $badir; fi
- if ! [ -d $texdir ]; then mkdir $texdir; fi
- if ! [ -d $mtexdir ]; then mkdir $mtexdir; fi
-
- # TeX build directory. If built in a group scenario, the TeX build
- # directory must be separate for each member (so they can work on their
- # relevant parts of the paper without conflicting with each other).
- if [ "x$maneage_group_name" = x ]; then
- texbdir="$texdir"/build
- else
- user=$(whoami)
- texbdir="$texdir"/build-$user
- fi
- tikzdir="$texbdir"/tikz
- if ! [ -L tex/build ]; then ln -s "$(pwd -P)/$texdir" tex/build; fi
- if ! [ -L tex/tikz ]; then ln -s "$(pwd -P)/$tikzdir" tex/tikz; fi
-
- # Register the start of this run (we are appending the new
- # information so previous information is preserved until the user
- # intentionally deletes/cleans it).
+ # Register the start of this run if requested (we are appending the
+ # new information so previous information is preserved until the
+ # user intentionally deletes/cleans it).
if [ $timing = 1 ]; then echo "start: $(date)" >> timing.txt; fi
# Run data preparation phase (optionally build Makefiles with
@@ -564,36 +638,33 @@ case $operation in
# Interactive shell of Maneage.
shell)
- # Make sure the configure script has been completed properly
- # ('configuration-done.txt' exists).
- if ! [ -f .build/software/configuration-done.txt ]; then
- configuration_necessary
- fi
+ # Make sure the configure script has been completed properly.
+ configuration_check
# Run the project's own shell without inheriting any environment
# from the host. The 'TERM' environment variable is necessary for
# tools like some text editors.
- bdir=`.local/bin/realpath .build`
+ bdir=$(.local/bin/realpath .build)
instdir="$bdir"/software/installed
bindir="$bdir"/software/installed/bin
rcfile=$(pwd)/reproduce/software/shell/bashrc.sh
.local/bin/env -i \
- HOME="$bdir" \
- TERM="$TERM" \
- PATH="$bindir" \
- CCACHE_DISABLE=1 \
- PROJECT_STATUS=shell \
- SHELL="$bindir"/bash \
- COLORTERM="$COLORTERM" \
- PROJECT_RCFILE="$rcfile" \
- LDFLAGS=-L"$instdir"/lib \
- CPPFLAGS=-I"$instdir"/include \
- LD_LIBRARY_PATH="$instdir"/lib \
- OMPI_MCA_plm_rsh_agent=/bin/false \
- PYTHONPATH="$instdir"/lib/python/site-packages \
- PYTHONPATH3="$instdir"/lib/python/site-packages \
- PS1="[\[\033[01;35m\]maneage@\h \W\[\033[32m\]\[\033[00m\]]$ " \
- "$bindir"/bash --noprofile --rcfile "$rcfile"
+ HOME="$bdir" \
+ TERM="$TERM" \
+ PATH="$bindir" \
+ CCACHE_DISABLE=1 \
+ PROJECT_STATUS=shell \
+ SHELL="$bindir"/bash \
+ COLORTERM="$COLORTERM" \
+ PROJECT_RCFILE="$rcfile" \
+ LDFLAGS=-L"$instdir"/lib \
+ CPPFLAGS=-I"$instdir"/include \
+ LD_LIBRARY_PATH="$instdir"/lib \
+ OMPI_MCA_plm_rsh_agent=/bin/false \
+ PYTHONPATH="$instdir"/lib/python/site-packages \
+ PYTHONPATH3="$instdir"/lib/python/site-packages \
+ PS1="[\[\033[01;35m\]maneage@\h \W\[\033[32m\]\[\033[00m\]]$ " \
+ "$bindir"/bash --noprofile --rcfile "$rcfile"
;;