From df9e291826fbc7e717b40d2d07f1d7607a2f2455 Mon Sep 17 00:00:00 2001 From: Giacomo Lorenzetti Date: Thu, 3 Apr 2025 15:21:16 +0200 Subject: IMPORTANT: software configuration optimized and better modularized Summary: after merging this commit into your project, it should be re-configured since the location of software installation files like 'LOCAL.conf' or the LaTeX macros of the software environment have changed. But it should not affect the analysis phase of your project. Until this commit, it was not possible to run a pre-built Maneage'd project (in a container) on a newly cloned Maneage'd project source. This was because the containers should be read-only, but during the various checks of the configuration (to verify that we are using the same software environment in the container and the source), we were writing/testing many things in the build directory, and 'LOCAL.conf' which was actually in the source directory! Furthermore, the '.local' and '.build' were built at configure time, making it hard to run the same container from a newly cloned Maneage'd project. To make things harder for the scenario above, the 'configure.sh' script would pause on every message and didn't have a quiet mode (making it practically impossible to run './project configure' before './project make' on every container run). With this commit, all these issues have been addressed and it is now possible to simply get a built container, clone a Maneage'd project and run the analysis (using the built environment of the container that is verified on every run). The respective changes/additions are described below: - The high-level container scripts ('apptainer.sh' and 'docker.sh', along with their READMEs) have been moved to the 'reproduce/software/shell' directory and the old 'reproduce/software/containers' directory has been deleted. This is because we have classified the software files by their language/format and the container scripts are scripts in the end. - The './project' script: - Now has two extra options: '--quiet' and '--no-pause'. Both are directly passed to the 'configure.sh' script. They will respectively disable any informative printed message or any pause after that message (if it is printed). - The '--build-dir' option is now also relevant for './project make': when it is given, it will re-create the two '.build' and '.local' symbolic links at the top source directory in all scenarios ('configure', 'make' or 'shell'). This will allow both the configuration, analysis and shell phases to safely assume they exist and match the user's desire at run-time. - The build/analysis directory's sub-directories that need to be built before 'top-make.mk' are now built in a separate function to help in readability. - The 'configure.sh' script: - For developers: a new 'check_elapsed' variable has been defined that will enable the newly added 'elapsed_time_from_prev_step' function. This function should be used from now on at the end of every major step to help find bottlenecks. - The targets of the software in 'pre-make-build.sh' now also have the version of the software in their file name. Until now, they didn't have the version, so there was no way to detect if the software has been updated or not in the source. For Lzip and Make (that also get built after GCC), the ones in this script have a '-pre-make' suffix also. - 'Local.conf.in' now has descriptions for every variable. - The '-std=gnu17' option is now used instead of '-std=c17' for basic software that cannot be built without specifying the C standard in GCC 15.1 (described in previous commit: 2881fc0a6205). See [1] for more details; in summary: '-std=gnu17' is also supported on macOS's Clang and has some features that 'pkg-config' needs - Generally: some longer code lines have been broken or indentation decreased to fit the 75 character line length. This has not reduced readability however. For example the long 'echo' commands are now replaced by multiple 'printf's, or the indentation is still clearly visible. The seeds of the work on this commit started by a branch containing three commits by Giacomo Lorenzetti (133 insertions, 100 deletions). Upon merging with the main 'maneage' branch, they were generalized and re-organized to become this commit. The following issues have also been addressed with this commit: - The LaTeX calls (during the building of 'paper.pdf') do not contain Maneage'd dynamic libraries. This is because we don't build the LaTeX binaries from source, an TeXLive manager uses the host environment. - The 'docker.sh' script: - Adds the '--project-name' option: its internal variable existed, but the option for the user to define it at run-time was not. - Ported to macOS: it does not check being a member of the 'docker' group, and finds the number of threads using macOS-specific tools. - The 'apptainer.sh' script: - Now installs 'wget' in the base container also (necessary when the user doesn't have the tarballs). [1] https://savannah.nongnu.org/bugs/?67068#comment2 --- reproduce/software/shell/apptainer-README.md | 71 ++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 reproduce/software/shell/apptainer-README.md (limited to 'reproduce/software/shell/apptainer-README.md') diff --git a/reproduce/software/shell/apptainer-README.md b/reproduce/software/shell/apptainer-README.md new file mode 100644 index 0000000..a7826ec --- /dev/null +++ b/reproduce/software/shell/apptainer-README.md @@ -0,0 +1,71 @@ +# Maneage'd projects in Apptainer + +Copyright (C) 2025-2025 Mohammad Akhlaghi \ +Copyright (C) 2025-2025 Giacomo Lorenzetti \ +See the end of the file for license conditions. + +For an introduction on containers, see the "Building in containers" section +of the `README.md` file within the top-level directory of this +project. Here, we focus on Apptainer with a simple checklist on how to use +the `apptainer-run.sh` script that we have already prepared in this +directory for easy usage in a Maneage'd project. + + + + + +## Building your Maneage'd project in Apptainer + +Through the steps below, you will create an Apptainer image that will only +contain the software environment and keep the project source and built +analysis files (data and PDF) on your host operating system. This enables +you to keep the size of the image to a minimum (only containing the built +software environment) to easily move it from one computer to another. + + 1. Using your favorite text editor, create a `run.sh` in your top Maneage + directory (as described in the comments at the start of the + `apptainer.sh` script in this directory). Just add `--build-only` on + the first run so it doesn't go onto doing the analysis and just sets up + the software environment. Set the respective directory(s) based on your + filesystem (the software directory is optional). The `run.sh` file name + is already in `.gitignore` (because it contains local directories), so + Git will ignore it and it won't be committed by mistake. + + 2. Make the script executable with `chmod +x ./run.sh`, and run it with + `./run.sh`. + + 3. Once the build finishes, the build directory (on your host) will + contain two Singularity Image Format (SIF) files listed below. You can + move them to any other (more permanent) positions in your filesystem or + to other computers as needed. + * `maneage-base.sif`: image containing the base operating system that + was used to build your project. You can safely delete this unless you + need to keep it for future builds without internet (you can give it + to the `--base-name` option of this script). If you want a different + name for this, put the same option in your + * `maneaged.sif`: image with the full software environment of your + project. This file is necessary for future runs of your project + within the container. + + 3. To execute your project remote the `--build-only` and use `./run.sh` to + execute it. If you want to enter your Maneage'd project shell, add the + `--project-shell` option to the call inside `./run.sh`. + + + + + +## Copyright information + +This file is free software: you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) +any later version. + +This file is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along +with this file. If not, see . -- cgit v1.2.1