diff options
81 files changed, 5880 insertions, 3146 deletions
@@ -18,13 +18,14 @@ *~ *\# -*.txt *.aux *.log -*.pdf *.out -*.zip +*.pdf +*.sif *.swp +*.txt +*.zip .nfs* mmap_* *.tar.gz @@ -32,6 +33,7 @@ mmap_* .tex build +run.sh .local .build Makefile @@ -40,7 +42,7 @@ tex/tikz .DS_Store .texlive* LOCAL.conf -docker-run +timing.txt tex/pipeline LOCAL_tmp.mk LOCAL_old.mk diff --git a/README-hacking.md b/README-hacking.md index ad44d3c..99cb6c2 100644 --- a/README-hacking.md +++ b/README-hacking.md @@ -1,566 +1,51 @@ -Maneage: managing data lineage -============================== +# Maneage development -Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ -Copyright (C) 2020-2025 Raul Infante-Sainz <infantesainz@gmail.com>\ +Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ See the end of the file for license conditions. -Maneage is a **fully working template** for doing reproducible research (or -writing a reproducible paper) as defined in the link below. If the link -below is not accessible at the time of reading, please see the appendix at -the end of this file for a portion of its introduction. Some -[slides](http://akhlaghi.org/pdf/reproducible-paper.pdf) are also available -to help demonstrate the concept implemented here. - - http://akhlaghi.org/reproducible-science.html - -Maneage is created with the aim of supporting reproducible research by -making it easy to start a project in this framework. As shown below, it is -very easy to customize Maneage for any particular (research) project and -expand it as it starts and evolves. It can be run with no modification (as -described in `README.md`) as a demonstration and customized for use in any -project as fully described below. - -A project designed using Maneage will download and build all the necessary -libraries and programs for working in a closed environment (highly -independent of the host operating system) with fixed versions of the -necessary dependencies. The tarballs for building the local environment are -also collected in a [separate -repository](http://git.maneage.org/tarballs-software.git/tree/). The final -output of the project is [a -paper](http://git.maneage.org/output-raw.git/plain/paper.pdf). Notice the -last paragraph of the Acknowledgments where all the necessary software are -mentioned with their versions. - -Below, we start with a discussion of why Make was chosen as the high-level -language/framework for project management and how to learn and master Make -easily (and freely). The general architecture and design of the project is -then discussed to help you navigate the files and their contents. This is -followed by a checklist for the easy/fast customization of Maneage to your -exciting research. We continue with some tips and guidelines on how to -manage or extend your project as it grows based on our experiences with it -so far. There is also a publication checklist, describing the recommended -steps to publish your data/code. The main body concludes with a description -of possible future improvements that are planned for Maneage (but not yet -implemented). As discussed above, we end with a short introduction on the -necessity of reproducible science in the appendix. +Maneage (Managing data lineage) is a customizable workflow controller +program, providing low-level control over a project's data lineage and its +history for a designing and operating fully reproducible project. Maneage +is formally defined in [Akhlaghi et +al. 2021](https://scixplorer.org/abs/2021CSE....23c..82A) (Computing in +Science & Engineering, vol. 23, issue 3, pp. 82-91; +DOI:[10.1109/MCSE.2021.3072860](https://doi.org/10.1109/MCSE.2021.3072860)). The +Maneage webpage at https://maneage.org contains a 30 minute video +presentation of Maneage that you can also watch for a fast introduction. + +This `README-hacking.md` file is targeted at those who want to develop +their projects within Maneage (by "hacking", or modifying, it), not just +execute it. The `README.md` file that is also in this directory contains +the instructions for those who only need to run/execute it. Please don't forget to share your thoughts, suggestions and criticisms. Maintaining and designing Maneage is itself a separate project, -so please join us if you are interested. Once it is mature enough, we will -describe it in a paper (written by all contributors) for a formal -introduction to the community. +so please join us if you are interested. The "Development Workflow" section +below describes our general development workflow. One of our planned tasks +is to write a fully featured documentation (similar to the [Gnuastro +documentation](https://www.gnu.org/software/gnuastro/manual)). +[[_TOC_]] -Why Make? ---------- - -When batch processing is necessary (no manual intervention, as in a -reproducible project), shell scripts are usually the first solution that -come to mind. However, the inherent complexity and non-linearity of -progress in a scientific project (where experimentation is key) make it -hard to manage the script(s) as the project evolves. For example, a script -will start from the top/start every time it is run. So if you have already -completed 90% of a research project and want to run the remaining 10% that -you have newly added, you have to run the whole script from the start -again. Only then will you see the effects of the last new steps (to find -possible errors, or better solutions and etc). - -It is possible to manually ignore/comment parts of a script to only do a -special part. However, such checks/comments will only add to the complexity -of the script and will discourage you to play-with/change an already -completed part of the project when an idea suddenly comes up. It is also -prone to very serious bugs in the end (when trying to reproduce from -scratch). Such bugs are very hard to notice during the work and frustrating -to find in the end. - -The Make paradigm, on the other hand, starts from the end: the final -*target*. It builds a dependency tree internally, and finds where it should -start each time the project is run. Therefore, in the scenario above, a -researcher that has just added the final 10% of steps of her research to -her Makefile, will only have to run those extra steps. With Make, it is -also trivial to change the processing of any intermediate (already written) -*rule* (or step) in the middle of an already written analysis: the next -time Make is run, only rules that are affected by the changes/additions -will be re-run, not the whole analysis/project. - -This greatly speeds up the processing (enabling creative changes), while -keeping all the dependencies clearly documented (as part of the Make -language), and most importantly, enabling full reproducibility from scratch -with no changes in the project code that was working during the -research. This will allow robust results and let the scientists get to what -they do best: experiment and be critical to the methods/analysis without -having to waste energy and time on technical problems that come up as a -result of that experimentation in scripts. - -Since the dependencies are clearly demarcated in Make, it can identify -independent steps and run them in parallel. This further speeds up the -processing. Make was designed for this purpose. It is how huge projects -like all Unix-like operating systems (including GNU/Linux or Mac OS -operating systems) and their core components are built. Therefore, Make is -a highly mature paradigm/system with robust and highly efficient -implementations in various operating systems perfectly suited for a complex -non-linear research project. - -Make is a small language with the aim of defining *rules* containing -*targets*, *prerequisites* and *recipes*. It comes with some nice features -like functions or automatic-variables to greatly facilitate the management -of text (filenames for example) or any of those constructs. For a more -detailed (yet still general) introduction see the article on Wikipedia: - - https://en.wikipedia.org/wiki/Make_(software) - -Make is a +40 year old software that is still evolving, therefore many -implementations of Make exist. The only difference in them is some extra -features over the [standard -definition](https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html) -(which is shared in all of them). Maneage is primarily written in GNU Make -(which it installs itself, you don't have to have it on your system). GNU -Make is the most common, most actively developed, and most advanced -implementation. Just note that Maneage downloads, builds, internally -installs, and uses its own dependencies (including GNU Make), so you don't -have to have it installed before you try it out. - - - - - -How can I learn Make? ---------------------- - -The GNU Make book/manual (links below) is arguably the best place to learn -Make. It is an excellent and non-technical book to help get started (it is -only non-technical in its first few chapters to get you started easily). It -is freely available and always up to date with the current GNU Make -release. It also clearly explains which features are specific to GNU Make -and which are general in all implementations. So the first few chapters -regarding the generalities are useful for all implementations. - -The first link below points to the GNU Make manual in various formats and -in the second, you can download it in PDF (which may be easier for a first -time reading). - - https://www.gnu.org/software/make/manual/ - - https://www.gnu.org/software/make/manual/make.pdf - -If you use GNU Make, you also have the whole GNU Make manual on the -command-line with the following command (you can come out of the "Info" -environment by pressing `q`). - -```shell - $ info make -``` - -If you aren't familiar with the Info documentation format, we strongly -recommend running `$ info info` and reading along. In less than an hour, -you will become highly proficient in it (it is very simple and has a great -manual for itself). Info greatly simplifies your access (without taking -your hands off the keyboard!) to many manuals that are installed on your -system, allowing you to be much more efficient as you work. If you use the -GNU Emacs text editor (or any of its variants), you also have access to all -Info manuals while you are writing your projects (again, without taking -your hands off the keyboard!). - - - - - -Published works using Maneage ------------------------------ - -The list below shows some of the works that have already been published -with (earlier versions of) Maneage, and some that have been recently -submitted for peer review. The previous version of Maneage was called -"Reproducible paper template", with a separate git tree. Maneage is -evolving rapidly, so some details will differ between the different -versions. The more recent papers will tend to be the most useful as good -working examples. - - - Borkowska & Roukema - ([2022](https://ui.adsabs.harvard.edu/abs/2021arXiv211214174B), MNRAS - Submitted, arXiv:2112.14174): The live version of the controlled source - is [at Codeberg](https://codeberg.org/boud/gevcurvtest); the main input - dataset, a software snapshot, the software tarballs, the project outputs - and editing history are available at - [zenodo.5806027](https://doi.org/10.5281/zenodo.5806027); and the - archived git history is available at [swh:1:rev:54398b720ddbac269ede30bf1e27fe27f07567f7](https://archive.softwareheritage.org/browse/revision/54398b720ddbac269ede30bf1e27fe27f07567f7). - - - Peper & Roukema - ([2021](https://ui.adsabs.harvard.edu/abs/2021MNRAS.505.1223P), MNRAS, - 505, 1223, DOI:10.1093/mnras/stab1342, arXiv:2010.03742): The live - version of the controlled source is [at - Codeberg](https://codeberg.org/boud/elaphrocentre); the main input - dataset, a software snapshot, the software tarballs, the project outputs - and editing history are available at - [zenodo.4699702](https://zenodo.org/record/4699702); and the archived - git history is available at - [swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e](https://archive.softwareheritage.org/swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e). - - - Roukema ([2021](https://ui.adsabs.harvard.edu/abs/2021PeerJ...911856R), - PeerJ, 9:e11856, arXiv:2007.11779): The live version of the controlled - source is [at Codeberg](https://codeberg.org/boud/subpoisson); the main - input dataset, a software snapshot, the software tarballs, the project - outputs and editing history are available at - [zenodo.4765705](https://zenodo.org/record/4765705); and the archived - git history is available at - [swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37](https://archive.softwareheritage.org/swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37). - - - Akhlaghi et - al. ([2021](https://ui.adsabs.harvard.edu/abs/2021CSE....23c..82A), - CiSE, 23(3), 82 DOI:10.1109/MCSE.2021.3072860 arXiv:2006.03018): The - project's version controlled source is [on - Gitlab](https://gitlab.com/makhlaghi/maneage-paper), necessary software, - outputs and backup of history are available at - [zenodo.3872248](https://doi.org/10.5281/zenodo.3872248); and the - archived git history is available at - [swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717](https://archive.softwareheritage.org/swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717). - - - Infante-Sainz et - al. ([2020](https://ui.adsabs.harvard.edu/abs/2020MNRAS.491.5317I), - MNRAS, 491, 5317): The version controlled project source is available - [on GitLab](https://gitlab.com/infantesainz/sdss-extended-psfs-paper) - and is also archived on Zenodo with all the necessary software tarballs: - [zenodo.3524937](https://zenodo.org/record/3524937). - - - Akhlaghi ([2019](https://arxiv.org/abs/1909.11230), IAU Symposium - 355). The version controlled project source is available - [on GitLab](https://gitlab.com/makhlaghi/iau-symposium-355) and is also - archived on Zenodo with all the necessary software tarballs: - [zenodo.3408481](https://doi.org/10.5281/zenodo.3408481). - - - Section 7.3 of Bacon et - al. ([2017](http://adsabs.harvard.edu/abs/2017A%26A...608A...1B), A&A - 608, A1): The version controlled project source is available [on - GitLab](https://gitlab.com/makhlaghi/muse-udf-origin-only-hst-magnitudes) - and a snapshot of the project along with all the necessary input - datasets and outputs is available in - [zenodo.1164774](https://doi.org/10.5281/zenodo.1164774). - - - Section 4 of Bacon et - al. ([2017](http://adsabs.harvard.edu/abs/2017A%26A...608A...1B), A&A, - 608, A1): The version controlled project is available [on - GitLab](https://gitlab.com/makhlaghi/muse-udf-photometry-astrometry) and - a snapshot of the project along with all the necessary input datasets is - available in [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746). - - - Akhlaghi & Ichikawa - ([2015](http://adsabs.harvard.edu/abs/2015ApJS..220....1A), ApJS, 220, - 1): The version controlled project is available [on - GitLab](https://gitlab.com/makhlaghi/NoiseChisel-paper). This is the - very first (and much less mature!) incarnation of Maneage: the history - of Maneage started more than two years after this paper was - published. It is a very rudimentary/initial implementation, thus it is - only included here for historical reasons. However, the project source - is complete, accurate and uploaded to arXiv along with the paper. - - - - - -Citation --------- - -If you use Maneage in your project please cite Akhlaghi et -al. ([2020](https://arxiv.org/abs/2006.03018), arXiv:2006.03018). It has -been submitted and is under peer review. - -Also, when your paper is published, don't forget to add a notice in your -own paper (in coordination with the publishing editor) that the paper is -fully reproducible and possibly add a sentence or paragraph in the end of -the paper shortly describing the concept. This will help spread the word -and encourage other scientists to also manage and publish their projects in -a reproducible manner. - - - - - - - - - - -Project architecture -==================== - -In order to customize Maneage to your research, it is important to first -understand its architecture so you can navigate your way in the directories -and understand how to implement your research project within its framework: -where to add new files and which existing files to modify for what -purpose. But if this the first time you are using Maneage, before reading -this theoretical discussion, please run Maneage once from scratch without -any changes (described in `README.md`). You will see how it works (note that -the configure step builds all necessary software, so it can take long, but -you can continue reading while its working). - -The project has two top-level directories: `reproduce` and -`tex`. `reproduce` hosts all the software building and analysis -steps. `tex` contains all the final paper's components to be compiled into -a PDF using LaTeX. - -The `reproduce` directory has two sub-directories: `software` and -`analysis`. As the name says, the former contains all the instructions to -download, build and install (independent of the host operating system) the -necessary software (these are called by the `./project configure` -command). The latter contains instructions on how to use those software to -do your project's analysis. - -After it finishes, `./project configure` will create the following symbolic -links in the project's top source directory: `.build` which points to the -top build directory and `.local` for easy access to the custom built -software installation directory. With these you can easily access the build -directory and project-specific software from your top source directory. For -example if you run `.local/bin/ls` you will be using the `ls` of Maneage, -which is probably different from your system's `ls` (run them both with -`--version` to check). - -Once the project is configured for your system, `./project make` will do -the basic preparations and run the project's analysis with the custom -version of software. The `project` script is just a wrapper, and with the -`make` argument, it will first call `top-prepare.mk` and `top-make.mk` -(both are in the `reproduce/analysis/make` directory). - -In terms of organization, `top-prepare.mk` and `top-make.mk` have an -identical design, only minor differences. So, let's continue Maneage's -architecture with `top-make.mk`. Once you understand that, you'll clearly -understand `top-prepare.mk` also. These very high-level files are -relatively short and heavily commented so hopefully the descriptions in -each comment will be enough to understand the general details. As you read -this section, please also look at the contents of the mentioned files and -directories to fully understand what is going on. - -Before starting to look into the top `top-make.mk`, it is important to -recall that Make defines dependencies by files. Therefore, the -input/prerequisite and output of every step/rule must be a file. Also -recall that Make will use the modification date of the prerequisite(s) and -target files to see if the target must be re-built or not. Therefore during -the processing, _many_ intermediate files will be created (see the tips -section below on a good strategy to deal with large/huge files). - -To keep the source and (intermediate) built files separate, the user _must_ -define a top-level build directory variable (or `$(BDIR)`) to host all the -intermediate files (you defined it during `./project configure`). This -directory doesn't need to be version controlled or even synchronized, or -backed-up in other servers: its contents are all products, and can be -easily re-created any time. As you define targets for your new rules, it is -thus important to place them all under sub-directories of `$(BDIR)`. As -mentioned above, you always have fast access to this "build"-directory with -the `.build` symbolic link. Also, beware to *never* make any manual change -in the files of the build-directory, just delete them (so they are -re-built). - -In this architecture, we have two types of Makefiles that are loaded into -the top `Makefile`: _configuration-Makefiles_ (only independent -variables/configurations) and _workhorse-Makefiles_ (Makefiles that -actually contain analysis/processing rules). - -The configuration-Makefiles are those that satisfy these two wildcards: -`reproduce/software/config/*.conf` (for building the necessary software -when you run `./project configure`) and `reproduce/analysis/config/*.conf` -(for the high-level analysis, when you run `./project make`). These -Makefiles don't actually have any rules, they just have values for various -free parameters throughout the configuration or analysis. Open a few of -them to see for yourself. These Makefiles must only contain raw Make -variables (project configurations). By "raw" we mean that the Make -variables in these files must not depend on variables in any other -configuration-Makefile. This is because we don't want to assume any order -in reading them. It is also very important to *not* define any rule, or -other Make construct, in these configuration-Makefiles. - -Following this rule-of-thumb enables you to set these configure-Makefiles -as a prerequisite to any target that depends on their variable -values. Therefore, if you change any of their values, all targets that -depend on those values will be re-built. This is very convenient as your -project scales up and gets more complex. - -The workhorse-Makefiles are those satisfying this wildcard -`reproduce/software/make/*.mk` and `reproduce/analysis/make/*.mk`. They -contain the details of the processing steps (Makefiles containing -rules). Therefore, in this phase *order is important*, because the -prerequisites of most rules will be the targets of other rules that will be -defined prior to them (not a fixed name like `paper.pdf`). The lower-level -rules must be imported into Make before the higher-level ones. - -All processing steps are assumed to ultimately (usually after many rules) -end up in some number, image, figure, or table that will be included in the -paper. The writing of these results into the final report/paper is managed -through separate LaTeX files that only contain macros (a name given to a -number/string to be used in the LaTeX source, which will be replaced when -compiling it to the final PDF). So the last target in a workhorse-Makefile -is a `.tex` file (with the same base-name as the Makefile, but in -`$(BDIR)/tex/macros`). As a result, if the targets in a workhorse-Makefile -aren't directly a prerequisite of other workhorse-Makefile targets, they -can be a prerequisite of that intermediate LaTeX macro file and thus be -called when necessary. Otherwise, they will be ignored by Make. - -Maneage also has a mode to share the build directory between several -users of a Unix group (when working on large computer clusters). In this -scenario, each user can have their own cloned project source, but share the -large built files between each other. To do this, it is necessary for all -built files to give full permission to group members while not allowing any -other users access to the contents. Therefore the `./project configure` and -`./project make` steps must be called with special conditions which are -managed in the `--group` option. - -Let's see how this design is implemented. Please open and inspect -`top-make.mk` it as we go along here. The first step (un-commented line) is -to import the local configuration (your answers to the questions of -`./project configure`). They are defined in the configuration-Makefile -`reproduce/software/config/LOCAL.conf` which was also built by `./project -configure` (based on the `LOCAL.conf.in` template of the same directory). - -The next non-commented set of the top `Makefile` defines the ultimate -target of the whole project (`paper.pdf`). But to avoid mistakes, a sanity -check is necessary to see if Make is being run with the same group settings -as the configure script (for example when the project is configured for -group access using the `./for-group` script, but Make isn't). Therefore we -use a Make conditional to define the `all` target based on the group -permissions. - -Having defined the top/ultimate target, our next step is to include all the -other necessary Makefiles. However, order matters in the importing of -workhorse-Makefiles and each must also have a TeX macro file with the same -base name (without a suffix). Therefore, the next step in the top-level -Makefile is to define the `makesrc` variable to keep the base names -(without a `.mk` suffix) of the workhorse-Makefiles that must be imported, -in the proper order. - -Finally, we import all the necessary remaining Makefiles: 1) All the -analysis configuration-Makefiles with a wildcard. 2) The software -configuration-Makefile that contains their version (just in case its -necessary). 3) All workhorse-Makefiles in the proper order using a Make -`foreach` loop. - -In short, to keep things modular, readable and manageable, follow these -recommendations: 1) Set clear-to-understand names for the -configuration-Makefiles, and workhorse-Makefiles, 2) Only import other -Makefiles from top Makefile. These will let you know/remember generally -which step you are taking before or after another. Projects will scale up -very fast. Thus if you don't start and continue with a clean and robust -convention like this, in the end it will become very dirty and hard to -manage/understand (even for yourself). As a general rule of thumb, break -your rules into as many logically-similar but independent steps as -possible. - -The `reproduce/analysis/make/paper.mk` Makefile must be the final Makefile -that is included. This workhorse Makefile ends with the rule to build -`paper.pdf` (final target of the whole project). If you look in it, you -will notice that this Makefile starts with a rule to create -`$(mtexdir)/project.tex` (`mtexdir` is just a shorthand name for -`$(BDIR)/tex/macros` mentioned before). As you see, the only dependency of -`$(mtexdir)/project.tex` is `$(mtexdir)/verify.tex` (which is the last -analysis step: it verifies all the generated results). Therefore, -`$(mtexdir)/project.tex` is _the connection_ between the -processing/analysis steps of the project, and the steps to build the final -PDF. - -During the research, it often happens that you want to test a step that is -not a prerequisite of any higher-level operation. In such cases, you can -(temporarily) define that processing as a rule in the most relevant -workhorse-Makefile and set its target as a prerequisite of its TeX -macro. If your test gives a promising result and you want to include it in -your research, set it as prerequisites to other rules and remove it from -the list of prerequisites for TeX macro file. In fact, this is how a -project is designed to grow in this framework. - - - - - -File modification dates (meta data) ------------------------------------ - -While Git does an excellent job at keeping a history of the contents of -files, it makes no effort in keeping the file meta data, and in particular -the dates of files. Therefore when you checkout to a different branch, -files that are re-written by Git will have a newer date than the other -project files. However, file dates are important in the current design of -Maneage: Make checks the dates of the prerequisite files and target files -to see if the target should be re-built. - -To fix this problem, for Maneage we use a forked version of -[Metastore](https://github.com/mohammad-akhlaghi/metastore). Metastore use -a binary database file (which is called `.file-metadata`) to keep the -modification dates of all the files under version control. This file is -also under version control, but is hidden (because it shouldn't be modified -by hand). During the project's configuration, Maneage installs to Git hooks -to run Metastore 1) before making a commit to update its database with the -file dates in a branch, and 2) after doing a checkout, to reset the -file-dates after the checkout is complete and re-set the file dates back to -what they were. - -In practice, Metastore should work almost fully invisibly within your -project. The only place you might notice its presence is that you'll see -`.file-metadata` in the list of modified/staged files (commonly after -merging your branches). Since its a binary file, Git also won't show you -the changed contents. In a merge, you can simply accept any changes with -`git add -u`. But if Git is telling you that it has changed without a merge -(for example if you started a commit, but canceled it in the middle), you -can just do `git checkout .file-metadata` and set it back to its original -state. - - - - - -Summary -------- - -Based on the explanation above, some major design points you should have in -mind are listed below. - - - Define new `reproduce/analysis/make/XXXXXX.mk` workhorse-Makefile(s) - with good and human-friendly name(s) replacing `XXXXXX`. - - - Add `XXXXXX`, as a new line, to the values in `makesrc` of the top-level - `Makefile`. - - - Do not use any constant numbers (or important names like filter names) - in the workhorse-Makefiles or paper's LaTeX source. Define such - constants as logically-grouped, separate configuration-Makefiles in - `reproduce/analysis/config/XXXXX.conf`. Then set this - configuration-Makefiles file as a prerequisite to any rule that uses - the variable defined in it. - - - Through any number of intermediate prerequisites, all processing steps - should end in (be a prerequisite of) `$(mtexdir)/verify.tex` (defined in - `reproduce/analysis/make/verify.mk`). `$(mtexdir)/verify.tex` is the sole - dependency of `$(mtexdir)/project.tex`, which is the bridge between the - processing steps and PDF-building steps of the project. - - - +# Maneage checklists -Customization checklist -======================= +This section contians some checklists to help in various stages of your +project: from the first commit, to the final publication. If you notice +anything missing or any in-correct part (probably a change that has not +been explained here), please let us know to correct it. -Take the following steps to fully customize Maneage for your research -project. After finishing the list, be sure to run `./project configure` and -`project make` to see if everything works correctly. If you notice anything -missing or any in-correct part (probably a change that has not been -explained here), please let us know to correct it. -As described above, the concept of reproducibility (during a project) -heavily relies on [version -control](https://en.wikipedia.org/wiki/Version_control). Currently Maneage -uses Git as its main version control system. If you are not already -familiar with Git, please read the first three chapters of the [ProGit -book](https://git-scm.com/book/en/v2) which provides a wonderful practical -understanding of the basics. You can read later chapters as you get more -advanced in later stages of your work. -First custom commit -------------------- +## First custom commit 1. **Get this repository and its history** (if you don't already have it): Arguably the easiest way to start is to clone Maneage and prepare for @@ -580,8 +65,51 @@ First custom commit $ pwd # Just to confirm where you are. ``` - 2. **Prepare to build project**: The `./project configure` command of the - next step will build the different software packages within the + 2. The final job of Maneage is to create your paper's PDF. By default it + uses a custom LaTeX style that resembles that of the Astrophysical + Journal (because the precursor of Maneage was for [Akhlaghi & Ichikawa + 2015](https://ui.adsabs.harvard.edu/abs/2015ApJS..220....1A)). The + journal you plan to submit your paper to will have its own separate + style. So it is best that you start your project by writing in the + desired style. We have already customized Maneage for the official + styles of some journals. To find them, run `git branch -r | grep + journal`. If your planned journal is one of them, you can take the + following steps to start your project based on that journal's style. If + it is not in these, you can ignore this step for now and customize the + style later (you can model based on these branches). In the commands + below, we'll assume you want to prepare for the Astronomy and + Astrophysics journal (A&A). + + ```shell + $ git checkout -b journal origin-maneage/journal-a-and-a + $ git log -1 --oneline | awk '{print $1}' # To keep the commit hash + $ git rebase -i main # See description below + ``` + + In the first text editor that opens after the last command, change all + (except the first) `pick`s into `squash`, then save the change and + close the editor. If you notice any conflicts during the rebase, always + prefer the `HEAD` (or latest `maneage` branch) version for non-TeX + related files; then run `git rebase --continue`. Once the rebase is + finished, the editor will be pre-filled with all the commit messages in + that branch. You do not need those, so you can delete everything and + write a commit message like the following: `A&A journal (commit XXXXX + of Maneage's journal-a-and-a branch)`. Just replace the `XXXXX` with + the output of the second command above. The commit hash is important to + be stored here since it allows you to later check if any updates have + been made in that branch in the future. After completing the git rebase + operation (last command above), run the following commands below to put + the new commit in your `main` branch (and continue working based on + that). + + ```shell + $ git checkout main + $ git merge journal + $ git branch -D journal + ``` + + 3. **Prepare for configuration**: The `./project configure` command of + the next step will build the different software packages within the "build" directory (that you will specify). Nothing else on your system will be touched. However, since it takes long, it is useful to see what it is being built at every instant (its almost impossible to tell @@ -600,7 +128,7 @@ First custom commit $ ./project --check-config ``` - 3. **Test Maneage**: Before making any changes, it is important to test it + 4. **Test Maneage**: Before making any changes, it is important to test it and see if everything works properly with the commands below. If there is any problem in the `./project configure` or `./project make` steps, please contact us to fix the problem before continuing. Since the @@ -618,7 +146,7 @@ First custom commit # Open 'paper.pdf' and see if everything is ok. ``` - 4. **Setup the remote**: You can use any [hosting + 5. **Setup the remote**: You can use any [hosting facility](https://en.wikipedia.org/wiki/Comparison_of_source_code_hosting_facilities) that supports Git to keep an online copy of your project's version controlled history. We recommend [GitLab](https://gitlab.com) because @@ -646,7 +174,7 @@ First custom commit git push origin maneage # Push 'maneage' branch to 'origin' (no tracking). ``` - 5. **Title**, **short description** and **author**: You can start adding + 6. **Title**, **short description** and **author**: You can start adding your name (with your possible coauthors) and tentative abstract in `paper.tex`. You should see the relevant place in the preamble (prior to `\begin{document}`. Just note that some core project metadata like @@ -659,7 +187,7 @@ First custom commit specific journal's style), please feel free to use it your own methods after finishing this checklist and doing your first commit. - 6. **Delete dummy parts**: Maneage contains some parts that are only for + 7. **Delete dummy parts**: Maneage contains some parts that are only for the initial/test run, mainly as a demonstration of important steps, which you can use as a reference to use in your own project. But they not for any real analysis, so you should remove these parts as @@ -712,7 +240,7 @@ First custom commit $ ./project make ``` - 7. **Ignore changes in some Maneage files**: One of the main advantages of + 8. **Ignore changes in some Maneage files**: One of the main advantages of Maneage is that you can later update your infra-structure by merging your `main` branch with the `maneage` branch. This is good for many low-level features that you will likely never modify yourself. But it @@ -744,7 +272,7 @@ First custom commit $ git add .gitattributes ``` - 8. **Copyright and License notice**: It is necessary that _all_ the + 9. **Copyright and License notice**: It is necessary that _all_ the "copyright-able" files in your project (those larger than 10 lines) have a copyright and license notice. Please take a moment to look at several existing files to see a few examples. The copyright notice is @@ -762,11 +290,11 @@ First custom commit and generally, all the files you modified in the previous step. ``` - Copyright (C) 2018-2025 Existing Name <existing@email.address> - Copyright (C) 2025-2025 YOUR NAME <YOUR@EMAIL.ADDRESS> + Copyright (C) 2018-2026 Existing Name <existing@email.address> + Copyright (C) 2026-2026 YOUR NAME <YOUR@EMAIL.ADDRESS> ``` - 9. **Configure Git for fist time**: If this is the first time you are + 10. **Configure Git for fist time**: If this is the first time you are running Git on this system, then you have to configure it with some basic information in order to have essential information in the commit messages (ignore this step if you have already done it). Git will @@ -780,7 +308,7 @@ First custom commit $ git config --global core.editor nano ``` - 10. **Your first commit**: You have already made some small and basic + 11. **Your first commit**: You have already made some small and basic changes in the steps above and you are in your project's `main` branch. So, you can officially make your first commit in your project's history and push it. But before that, you need to make sure @@ -799,7 +327,7 @@ First custom commit $ git push # Push your commit to your remote. ``` - 11. **Read the publication checklist**: The publication checklist below is + 12. **Read the publication checklist**: The publication checklist below is very similar to this one, but for the final phase of your project. For now, you don't have to do any of its steps, but reading it will give you good insight into the later stages of your project. If you already @@ -809,7 +337,7 @@ First custom commit Making it much easier to complete that checklist when you are ready for submission. - 12. **Start your exciting research**: You are now ready to add flesh and + 13. **Start your exciting research**: You are now ready to add flesh and blood to this raw skeleton by further modifying and adding your exciting research steps. You can use the "published works" section in the introduction (above) as some fully working models to learn @@ -817,8 +345,12 @@ First custom commit questions. -Other basic customizations --------------------------- + +## Other basic customizations + +The checklist for the first commit above does not include other basic +customizations that will be necessary once you start using maneaging your +project. Here, we list some of the other steps that are usually necessary. - **High-level software**: Maneage installs all the software that your project needs. You can specify which software your project needs in @@ -900,7 +432,7 @@ Other basic customizations whole project while the project is under development (focus is on the analysis that is done after preparation). Because of this, preparation will be done automatically for the first time that the project is run - (when `.build/software/preparation-done.mk` doesn't exist). After the + (when `.build/software/preparation-data.mk` doesn't exist). After the preparation process completes once, future runs of `./project make` will not do the preparation process anymore (will not call `top-prepare.mk`). They will only call `top-make.mk` for the @@ -921,15 +453,7 @@ Other basic customizations - - - - - - - -Upgrading the Maneage branch of your project -============================================ +## Upgrading the Maneage branch In time, Maneage is going to become more and more mature and robust (thanks to your feedback and the feedback of other users). Bugs will be fixed and @@ -967,7 +491,7 @@ $ git pull # Get recent work in Maneage particular pay close attention to the ones starting with 'IMPORTANT': these may cause a crash in your project (changing something fundamental in Maneage). Replace the `XXXXXXX..YYYYYYY` with hashs mentioned close to -start of the `git pull` command outputs (prevoius step). +start of the `git pull` command outputs (previous step). ```shell $ git log XXXXXXX..YYYYYYY --reverse @@ -993,7 +517,7 @@ $ git status # Just for a check above) with the classification `both modified:`. Open one of these files with your favorite text editor and correct the conflict (placed in between `<<<<<<<`, `=======` and `>>>>>>>`). Once all conflicts in a file are -remoted, the file will be automatically removed from the "Unmerged paths" +removed, the file will be automatically removed from the "Un-merged paths" of `git status`. So run `git status` after correcting the conflicts of each file just to make sure things are clean. TIP: If you want the changes in one file to be only from a special branch (`maneage` or `main`, completely @@ -1040,13 +564,30 @@ $ git push origin maneage +## Citing Maneage + +If you use Maneage in your project please cite Akhlaghi et al. 2025, +published in Computing in Science & Engineering, vol. 23, issue 3, +pp. 82-91; +arXiv: [2006.03018](https://arxiv.org/abs/2006.03018); +Bibcode: +[2021CSE....23c..82A](https://scixplorer.org/abs/2021CSE....23c..82A); +DOI:[10.1109/MCSE.2021.3072860](https://doi.org/10.1109/MCSE.2021.3072860). + +The default `paper.tex` file contains example citation and how to +acknowledge all the software that are used. + +Also, when your paper is published, don't forget to add a notice that you +paper is fully reproducible and add a sentence or paragraph in the end of +the paper shortly describing the concept. This will help spread the word +and encourage other scientists to also manage and publish their projects in +a correctly scientific (reproducible) manner. -Publication checklist -===================== +## Publication checklist Once your project is complete and you are ready to submit/publish the project, we recommend the following steps to ensure the maximum FAIRness of @@ -1121,7 +662,7 @@ future. readers easily access these resources for supplementary material directly from your PDF paper (sources on SoftwareHeritage and data/software on Zenodo). These links are more trusted/reliable in terms - of longevity than Git repositories or private webpages. + of longevity than Git repositories or private web pages. - **Identify and properly format output data**: If you have a plot, figure or table in your paper, you need to verify that data later and publish @@ -1132,7 +673,7 @@ future. * *Keep published data in a special place*: it helps if you keep the to-be-published data files in a special sub-directory under your build directory. In this way, irrespective of which subMakefile builds a - published dataset, they won't be lost/scatterred in the middle of all + published dataset, they won't be lost/scattered in the middle of all the project's intermediate-built files. * *In plain-text*: If the data are in tabular form (for example the X @@ -1182,20 +723,26 @@ future. the plots should be uploaded directly to Zenodo so they can be viewed/downloaded with a simple link in the caption. For example see the last sentence of the caption of Figure 1 in - [arXiv:2006.03018](https://arxiv.org/pdf/2006.03018.pdf), it points to - [the data](https://zenodo.org/record/3872248/files/tools-per-year.txt) - that was used to create that figure's left-side plot. As you see, this - will allow your paper's readers (again, most probably your future-self!) - to directly access the numbers of each visualization (plot/figure) with - a simple click in a trusted server. This also shows the major advantage - of having your data as simple plain-text where possible, as described - above. To help you keep all your to-be-visualized datasets in a single - place, Maneage has the two `tex-publish-dir` and `data-publish-dir` - directories that are defined in `reproduce/analysis/make/initialize.mk`, - see the comments above their definition for more. + [Akhlaghi+2021](https://scixplorer.org/abs/2021CSE....23c..82A), it + points to [the + data](https://zenodo.org/record/3872248/files/tools-per-year.txt) that + was used to create that figure's left-side plot. A more complete example + is [Eskandarlou+2026](https://scixplorer.org/abs/2026A&A...709A.210E); + where the underlying data of all figures are linked in Appendix E. As + you see, this will allow your paper's readers (again, most probably your + future-self!) to directly access the numbers of each visualization + (plot/figure) with a simple click in a trusted server. This also shows + the major advantage of having your data as simple plain-text where + possible, as described above. To help you keep all your to-be-visualized + datasets in a single place, Maneage has the two `figdir` (for + figures/plots) and `drdir` (for data release) directories that are + defined in `reproduce/analysis/make/initialize.mk`, see the comments + above their definition for more. Upon publication, you can just publish + the contents of these two directories. Therefore, be sure to only keep + the actual data in these directories, not intermediate files. - **Verification step**: It is very important to automatically verify the - outptus of your project. Recall from the customization checklist (above) + outputs of your project. Recall from the customization checklist (above) that you can activate verification by setting the `verify-outputs` variable to `yes` in `reproduce/analysis/config/verify-outputs.conf`. So please activate it and look into the `reproduce/analysis/make/verify.mk` @@ -1246,16 +793,16 @@ future. useful for servers like arXiv, or some journals. This tarball is also one of the deliverables you want to publish on Zenodo. Once the tarball is created, copy it to a temporary directory outside of Maneage, unpack - it and run `make` (completely ignoring Maneage's `./project` script). If - you plan to submit your paper to arXiv, the best test is to actually - start a test submission on arXiv to upload the tarball there to see if - it can build your PDF. Once it works, you can delete that temporary - submission for now. Afterwards, try configuring and building it with the - tarball by running its `./project` (from scratch and without the Git - history!). If there is a problem in any of these tests, you can modify - what goes into this tarball in `reproduce/analysis/make/initialize.mk`: - go through the steps and add the necessary components until the checks - pass. + it and run `make` (completely ignoring Maneage's `./project` + script). If you plan to submit your paper to arXiv, the best test is to + actually start a test submission on arXiv to upload the tarball there to + see if it can build your PDF. Once it works, you can delete that + temporary submission for now. Afterwards, try configuring and building + it with the tarball by running its `./project` (from scratch and + without the Git history!). If there is a problem in any of these tests, + you can modify what goes into this tarball in + `reproduce/analysis/make/initialize.mk`: go through the steps and add + the necessary components until the checks pass. - **Upload all deliverables to Zenodo**: With the datasets ready, you can now upload the following deliverables to Zenodo. Except for the data @@ -1270,7 +817,7 @@ future. * **paper-XXXXXXX.pdf**: you shouldn't just download data to the data server, also upload your paper's PDF so its there with the other raw - formats. It will greatly help yourself and others. Most datacenters + formats. It will greatly help yourself and others. Most data centers (like Zenodo) actually also have a PDF viewer that will load automatically before the list of data files. For example see [zenodo.3408481](https://doi.org/10.5281/zenodo.3408481). @@ -1309,7 +856,7 @@ future. repository, Zenodo-DOI (this is not your paper's DOI, its the data/resources DOI), and/or SoftwareHeritage link in the comments. - - *Update `metadata.conf`*: Once you have your final arXiv ID (formated + - *Update `metadata.conf`*: Once you have your final arXiv ID (formatted as: `1234.56789`) put it in `reproduce/analysis/config/metadata.conf`. - **Submission to a journal**: different journals accept submissions in @@ -1319,6 +866,50 @@ future. are public). If not, you can mention that everything is ready for such a submission after acceptance. + * Some journals (like the "Astronomy and Astrophysics" journal) + require that only a single LaTeX file be submitted without any + commens (along with all the figures). In other words, they do not + want any sub-directories: only a single LaTeX file, one `.bbl` file + and one PDF per figure. Here is how you can do this: + ```shell + # If you don't already have 'latexpand' in your OS, install it: + $ tlmgr install latexpand + + # Unpack the tarball and make a directory for a single-LaTeX source. + $ tar -xf maneaged-XXXXXXX.tar.gz + $ cd maneaged-XXXXXXX.tar.gz + $ mkdir one-latex + + ## The '--empty-comments' and subsequent call to AWK (that removes + ## lines starting with '%') is because we need the lines ending in a + ## '%' for some LaTeX macros). + $ latexpand paper.tex --output=one-latex/paper-0.tex \ + --empty-comments + $ awk '!/^%/' one-latex/paper-0.tex > one-latex/paper.tex + $ rm one-latex/paper-0.tex + + ## Copy the other necessary files (if you are using specific journal + ## style files, also copy them). + $ cp paper.bbl one-latex/ + $ cp tex/tikz/*.pdf one-latex/ + + ## Correct the location of the images. + $ sed -i -e's|tex/tikz/|./|' one-latex/paper.tex + ``` + + * You can confirm the contents of the `one-latex` directory made + above by running `pdflatex paper` inside that directory. Your + operating system's LaTeX may not have all the necessary packages, + in that case run `tlmgr install XXXXX` (where `XXXXX` is the + value of `texlive-packages` in + `reproduce/software/config/texlive-packages.conf`. Afterwards, + clean up all the temporary files with this command: + ```shell + $ rm *.log *.aux *.auxlock *.bcf *.out *.xml + ``` + + * Upload the contents of this `one-latex` directory to the journal. + - **Future versions**: Both Zenodo and arXiv allow uploading new versions after your first publication. So it is recommended to put more recent versions of your published projects later (for example after applying @@ -1361,8 +952,513 @@ future. -Tips for designing your project -=============================== + + + + +# Maneage development tools + +Maneage does not define a new programming language. By design, it uses +existing and time-tested tools/software. In this section, we review the +purpose of each tool and include recommendations on how to learn them. The +"Published works using Maneage" section below lists published papers that +have been written within Maneage. Once you have a good knowledge of the +necessary tools listed below, you can try reading the source of previously +published papers as demos to help in the optimal design your own project. + + + +### GNU Info + +The distinguishing feature of a good developer is not memorizing more of a +tool's features, but to know where to look (for the details) when they need +something. Documentation is therefore very important in any development +environment. As part of Maneage's software environment configuration, we +also install any documentation that each software installs (mostly in the +Info documentation format). This includes many of the basic software in +Maneage like Make, AWK, SED, Coreutils (that provides commands such as +`ls`, `cp` and etc), as well as high-level software like GNU Scientific +Library (GSL) and GNU Astronomy Utilities (Gnuastro). + +Info greatly simplifies your access (without taking your hands off the +keyboard!) to many manuals that are installed in your Maneage'd software +environment, allowing you to be much more efficient as you work. Another +nice thing with the Info documentation is that it corresponds the same +version that you are using in your analysis and you do not need any +internet connection. + +If you are not familiar with this documentation format, the good news is +that it has a very nice manual for itself that you can easily enter with +the second command below. The first command takes you into the interactive +Maneage'd software environment. In case you have the software environment +within a container, include the `--sif` option (just like the `./project +make` command). + +```shell +$ ./project shell +$ info info +``` + +We strongly recommend reading this manual: it will open a world of +knowledge for you that is always available, even when you are offline. In +less than an hour, you will become familiar and proficient in the Info +documentation. You can then use Info when reading the documentation of +other software you may want to learn when using Maneage (like Make, AWK, +SED or Gnuastro), and you'll see how powerful and convenient it is. + +If you use the GNU Emacs text editor (or any of its variants), you also +have access to all Info manuals while you are writing your code (again, +without taking your hands off the keyboard!): simply run `Ctrl-h i` inside +Emacs. Emacs (along with Vim) is available within the Project software +environment (which you can access through `./project shell`). + + + +### GNU Make + +When batch processing is necessary (no manual intervention, as in a +reproducible project), shell (or Python) scripts are usually the first +solution for most people. However, the inherent complexity and +non-linearity of progress in a scientific project (where experimentation is +key) make it hard to manage them as the project evolves. For example, a +script will start from the top/start every time it is run. So if you have +already completed 90% of the project and want to run the remaining 10% that +you have newly added, you have to run the whole script from the start +again. + +It is possible to manually ignore/comment parts of a script to only do a +special part. However, such checks/comments will only add to the complexity +of the script and will discourage you from experimenting on an already +completed part of the project when an idea suddenly comes up. It is also +prone to very serious bugs in the end (when trying to reproduce from +scratch); for example you forget to un-comment or remove a temporary +change. Such bugs are very hard to notice during the work and frustrating +to find in the end. + +Make is the solution to these problems. It starts from the end: the final +*target*. Dependencies and recipes (shell scripts) are defined for every +target and using them, Make builds a dependency tree internally. Using +that, Make finds where it should start each time the project is +run. Therefore, in the scenario above, a researcher that has just added the +final 10% of the project, will only have to run those extra steps: without +any modification to previously executed code. With Make, it is also trivial +to change the processing of any intermediate (already written) *rule* (or +step) in the middle of an already written analysis: the next time Make is +run, only rules that are affected by the changes/additions will be re-run, +not the whole analysis/project. + +This greatly speeds up the processing (enabling creative changes), while +keeping all the dependencies clearly documented (as part of the Make +language), and most importantly, enabling full reproducibility from scratch +with no changes in the project code that was working during the +research. This will allow robust results and let the scientists get to what +they do best: experiment and be critical to the methods/analysis without +having to waste energy and time on technical problems that come up as a +result of using scripts alone. + +Furthermore, since the dependencies are clearly demarcated in Make, it can +identify independent steps and run them in parallel. This further speeds up +the processing. Make was designed for this purpose. It is how huge projects +like all Unix-like operating systems (including GNU/Linux or Mac OS +operating systems) and their core components are built. Therefore, Make is +a highly mature paradigm/system with robust and highly efficient +implementations in various operating systems perfectly suited for a complex +non-linear research project. + +Make is a small language with the aim of defining *rules* that are composed +of *targets*, *prerequisites* and *recipes*. It comes with some nice +features like functions or automatic-variables to greatly facilitate the +management of text (filenames for example) or any of those constructs. For +a more detailed (yet still general) introduction see the [Make article on +Wikipedia](https://en.wikipedia.org/wiki/Make_(software)). + +As of 2026, Make is a 50 year old software that is still getting better, +therefore many implementations of Make exist. The only difference in them +is some extra features over the [standard +definition](https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html) +(which is shared in all of them). Maneage is primarily written in GNU Make +that is the most common, most actively developed, and most advanced +implementation. Just note that Maneage downloads, builds, internally +installs, and uses its own dependencies (including GNU Make), so you don't +have to have it installed before you try it out. + +The recipes (operations) that are applied on the dependencies of each +target to build the target are simple shell scripts in Maneage. In case you +are more comfortable with Python, Perl or R scripts, you can save them in +your project's source and simply call them in the recipe. + +The [GNU Make book](https://www.gnu.org/software/make/manual) is arguably +the best place to learn Make. As mentioned in the "Info documentation" +section above, you also have the manual within your Maneage'd software +environment. The main caveat of the GNU Make manual is that all its +examples and demonstrations are based on compiling programs in C; not data +analysis. But fortunately beyond the first chapter, the rest of the +examples in the manual are sufficiently generic. + +To replace the first chapter (and give examples of how Make can be used in +data analysis), along with some early adopters of Maneage and as part of +the IAC's Short Meetings on Astro-Computing Knowledge (SMACK) series, we +prepared [a hands-on video +demonstration](https://peertube.stream/w/iJitjS3r232Z8UPMxKo6jq) that you +can watch. In it we show how you can write your Make recipes in any +high-level scripting language. + +Once you have watched the video above, read the GNU Make manual at least +until the end of Chapter 8 (called "Functions for transforming text"). You +can use the Info format manual within Maneage (as described in the "Info +documentation" section above) with `info make`. + + + +### GNU Bash + +The main shell that is installed within Maneage is GNU Bash. Like GNU Make, +it also has an Info manual, but unfortunately it is not written with +beginners in mind: it digs deep into very technical details from the +start. In case you are using Gnuastro in your analysis, its [general +tutorial](https://www.gnu.org/software/gnuastro/manual/html_node/General-program-usage-tutorial.html) +also includes many useful tips on the shell. We have also prepared the +following hands-on SMACK videos to introduce the shell + +- SMACK 2: [More on the command-line](https://peertube.stream/w/d16286ee-fc46-444f-acb8-645f9de6c7b7). +- SMACK 4: [Shell scripting](https://peertube.stream/w/18d2e386-2ca2-4a1f-a949-c7ca5dcee8ff). + + + +### Git + +Git is the version control system used in Maneage to allow easy tracking of +changes during the development of the project, collaborate with colleagues +and also to import improvements from the core branch into individual +projects. Without Git, such operations become so tedious and prone to human +error that the development of any software-based project with more than one +developer would be practically impossible. + +If you are not yet familiar with Git, the best resource is the free and +open source [Pro Git](https://git-scm.com/book/en) book. Don't be alarmed +by the number of chapters in the Pro Git book. The majority of the book is +for very advanced usages of Git, including setting up servers (that is not +relevant in Maneage). To comfortably use Maneage, mastering the following +chapters is enough + +- Chapter 1: Getting Started +- Chapter 2: Git Basics +- Chapter 3: Git Branching +- Chapter 5: Distributed Git + +After mastering these chapters, you will get a very good feeling of any +further chapters in case you need them. Of course, [Git's own +manual](https://git-scm.com/docs/user-manual.html) is also available for +further details when necessary. + + + + + + + + + + +# Maneage directory architecture + +In order to customize Maneage to your research, it is important to first +understand its souce and build directory architectures. This will help +navigate your way in the directories and understand how to implement your +research project: where to add new files and which existing files to modify +for what purpose. But if this the first time you are using Maneage, before +reading this theoretical discussion, please run Maneage once from scratch +without any changes (described in `README.md`). This will show you how it +works and will allow you to make test changes/edits in the code to test +what you read here. + +A core concept in Maneage is that that there is a very strong distinction +between the "source" (hand-written, plain-text files: most valuable of them +all) and "build" (for downloaded, or automatically generated +datasets/files) directories. + +## Source directory (hand-written files) + +The directory that is hosting this file in your operating system is the +"source" directory. This is where all hand-written files of your project +(the most valuable) are saved and it is fully under version control. The +source directory has two top-level directories: + +- `reproduce/`: contains all the "code" (assuming LaTeX is not "code") and + has two high-level sub-directories for the two high-level phases of + Maneage: + + - `software/`: all the instructions to download, build and install + (independent of the host operating system) the necessary + software. These are executed when you ran the `./project configure`. + + - `analysis/`: all the analysis steps of your project: that are executed + with the `./project make` command. + +- `tex/` contains all the final paper's components to be compiled into a + PDF using LaTeX. After building Maneage once, two symbolic links to the + build directory are placed here that are described in the corresponding + sub-section below. But they are not under version control, the place to + keep any extra hand-written LaTeX source is under the `src/` + sub-directory here. + +At the end of its configuration, Maneage also creates some symbolic links +for easy access to the build directory from your top source directory. See +the corresponding sub-section below for more on those: they are very useful +while you are developing your project. + +Once the project is configured for your system, `./project make` will do +the basic preparations and run the project's analysis with the custom +version of software. The `project` script is just a wrapper, and with the +`make` argument, it will first call `top-prepare.mk` and `top-make.mk` +(both are in the `reproduce/analysis/make` directory). + +In terms of organization, `top-prepare.mk` and `top-make.mk` have an +identical design, only minor differences. So, let's continue Maneage's +architecture with `top-make.mk`. Once you understand that, you'll clearly +understand `top-prepare.mk` also. These very high-level files are +relatively short and heavily commented so hopefully the descriptions in +each comment will be enough to understand the general details. As you read +this section, please also look at the contents of the mentioned files and +directories to fully understand what is going on. + +Before starting to look into the top `top-make.mk`, it is important to +recall that Make defines dependencies by files. Therefore, the +input/prerequisite and output of every step/rule must be a file. Also +recall that Make will use the modification date of the prerequisite(s) and +target files to see if the target must be re-built or not. Therefore during +the processing, _many_ intermediate files will be created (see the tips +section below on a good strategy to deal with large/huge files). + +To keep the source and (intermediate) built files separate, the user _must_ +define a top-level build directory variable (or `$(BDIR)`) to host all the +intermediate files (you defined it during `./project configure`). This +directory doesn't need to be version controlled or even synchronized, or +backed-up in other servers: its contents are all products, and can be +easily re-created any time. As you define targets for your new rules, it is +thus important to place them all under sub-directories of `$(BDIR)`. As +mentioned above, you always have fast access to this "build"-directory with +the `.build` symbolic link. Also, beware to *never* make any manual change +in the files of the build-directory, just delete them (so they are +re-built). + +In this architecture, we have two types of Makefiles that are loaded into +the top `Makefile`: _configuration-Makefiles_ (only independent +variables/configurations) and _workhorse-Makefiles_ (Makefiles that +actually contain analysis/processing rules). + +The configuration-Makefiles are those that satisfy these two wildcards: +`reproduce/software/config/*.conf` (for building the necessary software +when you run `./project configure`) and `reproduce/analysis/config/*.conf` +(for the high-level analysis, when you run `./project make`). These +Makefiles don't actually have any rules, they just have values for various +free parameters throughout the configuration or analysis. Open a few of +them to see for yourself. These Makefiles must only contain raw Make +variables (project configurations). By "raw" we mean that the Make +variables in these files must not depend on variables in any other +configuration-Makefile. This is because we don't want to assume any order +in reading them. It is also very important to *not* define any rule, or +other Make construct, in these configuration-Makefiles. + +Following this rule-of-thumb enables you to set these configure-Makefiles +as a prerequisite to any target that depends on their variable +values. Therefore, if you change any of their values, all targets that +depend on those values will be re-built. This is very convenient as your +project scales up and gets more complex. + +The workhorse-Makefiles are those satisfying this wildcard +`reproduce/software/make/*.mk` and `reproduce/analysis/make/*.mk`. They +contain the details of the processing steps (Makefiles containing +rules). Therefore, in this phase *order is important*, because the +prerequisites of most rules will be the targets of other rules that will be +defined prior to them (not a fixed name like `paper.pdf`). The lower-level +rules must be imported into Make before the higher-level ones. + +All processing steps are assumed to ultimately (usually after many rules) +end up in some number, image, figure, or table that will be included in the +paper. The writing of these results into the final report/paper is managed +through separate LaTeX files that only contain macros (a name given to a +number/string to be used in the LaTeX source, which will be replaced when +compiling it to the final PDF). So the last target in a workhorse-Makefile +is a `.tex` file (with the same base-name as the Makefile, but in +`$(BDIR)/tex/macros`). As a result, if the targets in a workhorse-Makefile +aren't directly a prerequisite of other workhorse-Makefile targets, they +can be a prerequisite of that intermediate LaTeX macro file and thus be +called when necessary. Otherwise, they will be ignored by Make. + +Maneage also has a mode to share the build directory between several +users of a Unix group (when working on large computer clusters). In this +scenario, each user can have their own cloned project source, but share the +large built files between each other. To do this, it is necessary for all +built files to give full permission to group members while not allowing any +other users access to the contents. Therefore the `./project configure` and +`./project make` steps must be called with special conditions which are +managed in the `--group` option. + +Let's see how this design is implemented. Please open and inspect +`top-make.mk` it as we go along here. The first step (un-commented line) is +to import the local configuration (your answers to the questions of +`./project configure`). They are defined in the configuration-Makefile +`reproduce/software/config/LOCAL.conf` which was also built by `./project +configure` (based on the `LOCAL.conf.in` template of the same directory). + +The next non-commented set of the top `Makefile` defines the ultimate +target of the whole project (`paper.pdf`). But to avoid mistakes, a sanity +check is necessary to see if Make is being run with the same group settings +as the configure script (for example when the project is configured for +group access using the `./for-group` script, but Make isn't). Therefore we +use a Make conditional to define the `all` target based on the group +permissions. + +Having defined the top/ultimate target, our next step is to include all the +other necessary Makefiles. However, order matters in the importing of +workhorse-Makefiles and each must also have a TeX macro file with the same +base name (without a suffix). Therefore, the next step in the top-level +Makefile is to define the `makesrc` variable to keep the base names +(without a `.mk` suffix) of the workhorse-Makefiles that must be imported, +in the proper order. + +Finally, we import all the necessary remaining Makefiles: 1) All the +analysis configuration-Makefiles with a wildcard. 2) The software +configuration-Makefile that contains their version (just in case its +necessary). 3) All workhorse-Makefiles in the proper order using a Make +`foreach` loop. + +In short, to keep things modular, readable and manageable, follow these +recommendations: 1) Set clear-to-understand names for the +configuration-Makefiles, and workhorse-Makefiles, 2) Only import other +Makefiles from top Makefile. These will let you know/remember generally +which step you are taking before or after another. Projects will scale up +very fast. Thus if you don't start and continue with a clean and robust +convention like this, in the end it will become very dirty and hard to +manage/understand (even for yourself). As a general rule of thumb, break +your rules into as many logically-similar but independent steps as +possible. + +The `reproduce/analysis/make/paper.mk` Makefile must be the final Makefile +that is included. This workhorse Makefile ends with the rule to build +`paper.pdf` (final target of the whole project). If you look in it, you +will notice that this Makefile starts with a rule to create +`$(mtexdir)/project.tex` (`mtexdir` is just a shorthand name for +`$(BDIR)/tex/macros` mentioned before). As you see, the only dependency of +`$(mtexdir)/project.tex` is `$(mtexdir)/verify.tex` (which is the last +analysis step: it verifies all the generated results). Therefore, +`$(mtexdir)/project.tex` is _the connection_ between the +processing/analysis steps of the project, and the steps to build the final +PDF. + +During the research, it often happens that you want to test a step that is +not a prerequisite of any higher-level operation. In such cases, you can +(temporarily) define that processing as a rule in the most relevant +workhorse-Makefile and set its target as a prerequisite of its TeX +macro. If your test gives a promising result and you want to include it in +your research, set it as prerequisites to other rules and remove it from +the list of prerequisites for TeX macro file. In fact, this is how a +project is designed to grow in this framework. + +## Build directory (automatically generated files) + +The build directory has the same sub-directories as the `reproduce/` +sub-directory of the source. + +- `software/`: all the automatically built products during the Maneage + configuration phase. It includes the following sub-directories: + - `config/`: all the local configurations for your system. The + `LOCAL.conf` file of this directory can contain your server user ID + and password (used when your input data are in servers that need + authentication). + - `installed/`: the top-level software environment of Maneage. Just like + your Unix-based operating system, it has sub-directories like `bin/` + (for executable software), `lib/` (for libraries) and etc. + - `locks/`: This directory is used to keep file locks when necessary (it + may be removed later). File locks are useful to stop the + parallelization. for example when we want all the steps to be done in + parallel, but not for steps like downloading (where the + network/servers can be a bottleneck and actually decrease the speed of + a parallelized operation compared to its serial operation). + - `tarballs/`: The source tarballs of all the software that were used to + build the software environment. If they were in the directory given to + `--software-dir`, they will be symbolic links to the files in that + directory. Otherwise (if they were downloaded), they will be normal + files. +- `analysis/`: all the analysis outputs. The default (in the `maneage` + branch) will create two sub-directories here when you run `./project + make`: + - `inputs/`: all the input files to the project. Similar to the + `tarballs/` directory above, if the files were present on the host + file system (in a directory given to `--input-dir`), a symbolic link + will be placed. Otherwise (if they were downloaded), the raw file will + be present. You can use this distinction (being a symbolic link or + not) to see which files were not on your system and copy them into a + dedicated directory to avoid needing the internet on every new run. + + - `tex/`: This directory contains the built files that are necessary for + LaTeX aspect of you final PDF. It is recommended to not keep large + data files here, only that absolute minimum that you need to create + the final PDF (the contents of this directory will need to be uploaded + to publication severs like arXiv). The following sub-directories are + assumed to always be present here: + - `build\`: the directory that LaTeX actually executes in. The + `paper.tex` (of the top source directory) will be copied here and + the directories containing the other LaTeX files are included into + as directories under the `tex/` directory of the top-level source. + - `macros\`: all the macros from the various stages of the software + and analysis will be present as different files here. As described + in the source directory descriptions, these are the backbone of + Maneage's operation (one for every sub-Makefile). + - `to-publish\`: will contain the raw tables behind every figure. For + example, see Appendix E of [Eskandarlou et + al. 2026](https://scixplorer.org/abs/2026A%26A...709A.210E). + + +## Easy access to build directory from source + +After it finishes, `./project configure` will create the following symbolic +links (also directories) in the project's top source directory for easy +access to the build directory. + +- `.build` points to the top build directory. This is very useful for fast + access to the build directory without having to worry about which system + is hosting your maneage'd project: in one system, your build directory + can be in one location and on another system in another directory. But + thanks to this symbolic link, you do not have to worry which system you + are developing on. + +- `.local` for easy access to the custom built software installation + directory. For example the `ls` command of your project can be accessed + outside of Maneage with `.local/bin/ls`. This is useful when you want to + run your Maneage'd software outside of maneage (although using `./project + shell` is a much better strategy for that). + +There are also two symbolic links under the `tex/` directory of the source +that are listed below. When running `./project make dist`, the contents of +these symbolic links will be copied into the tarball. In this way, +publishers (like arXiv) do not have to run Maneage, they just run LaTeX and +have all the products that are necessary). + +- `tex/build` for direct access to the `analysis/tex/build` directory. As + described above, this directory will contain all the files that go into + the TeX source of the paper. + +- `tex/tikz` for the outputs of [PGF & TiKZ](https://ctan.org/pkg/pgf) (or + [PGFPLots](https://ctan.org/pkg/pgfplots), that is based on it). These + packages can generate plots within LaTeX. Therefore, they use the same + font, line width and colors of the paper, making your plots fit very + nicely (visually) in the final paper. Another nice aspect is that with + these software, you do not need to generate the plots/figures separate + from the paper (PGFPlots can directly read the tables you put in the + `to-publish/` subdirectory mentioned under the build directory. + + + + + + + + + + +# Maneagement tips The following is a list of design points, tips, or recommendations that have been learned after some experience with this type of project @@ -1370,328 +1466,699 @@ management. Please don't hesitate to share any experience you gain after using it with us. In this way, we can add it here (with full giving credit) for the benefit of others. - - **Modularity**: Modularity is the key to easy and clean growth of a - project. So it is always best to break up a job into as many - sub-components as reasonable. Here are some tips to stay modular. - - - *Short recipes*: if you see the recipe of a rule becoming more than a - handful of lines which involve significant processing, it is probably - a good sign that you should break up the rule into its main - components. Try to only have one major processing step per rule. - - - *Context-based (many) Makefiles*: For maximum modularity, this design - allows easy inclusion of many Makefiles: in - `reproduce/analysis/make/*.mk` for analysis steps, and - `reproduce/software/make/*.mk` for building software. So keep the - rules for closely related parts of the processing in separate - Makefiles. - - - *Descriptive names*: Be very clear and descriptive with the naming of - the files and the variables because a few months after the - processing, it will be very hard to remember what each one was - for. Also this helps others (your collaborators or other people - reading the project source after it is published) to more easily - understand your work and find their way around. - - - *Naming convention*: As the project grows, following a single standard - or convention in naming the files is very useful. Try best to use - multiple word filenames for anything that is non-trivial (separating - the words with a `-`). For example if you have a Makefile for - creating a catalog and another two for processing it under models A - and B, you can name them like this: `catalog-create.mk`, - `catalog-model-a.mk` and `catalog-model-b.mk`. In this way, when - listing the contents of `reproduce/analysis/make` to see all the - Makefiles, those related to the catalog will all be close to each - other and thus easily found. This also helps in auto-completions by - the shell or text editors like Emacs. - - - *Source directories*: If you need to add files in other languages for - example in shell, Python, AWK or C, keep the files in the same - language in a separate directory under `reproduce/analysis`, with the - appropriate name. - - - *Configuration files*: If your research uses special programs as part - of the processing, put all their configuration files in a devoted - directory (with the program's name) within - `reproduce/software/config`. It is much cleaner and readable (thus - less buggy) to avoid mixing the configuration files, even if there is - no technical necessity. - - - - **Contents**: It is good practice to follow the following - recommendations on the contents of your files, whether they are source - code for a program, Makefiles, scripts or configuration files - (copyrights aren't necessary for the latter). - - - *Copyright*: Always start a file containing programming constructs - with a copyright statement like the ones that Maneage starts with - (for example in the top level `Makefile`). - - - *Comments*: Comments are vital for readability (by yourself in two - months, or others). Describe everything you can about why you are - doing something, how you are doing it, and what you expect the result - to be. Write the comments as if it was what you would say to describe - the variable, recipe or rule to a friend sitting beside you. When - writing the project it is very tempting to just steam ahead with - commands and codes, but be patient and write comments before the - rules or recipes. This will also allow you to think more about what - you should be doing. Also, in several months when you come back to - the code, you will appreciate the effort of writing them. Just don't - forget to also read and update the comment first if you later want to - make changes to the code (variable, recipe or rule). As a general - rule of thumb: first the comments, then the code. - - - *File title*: In general, it is good practice to start all files with - a single line description of what that particular file does. If - further information about the totality of the file is necessary, add - it after a blank line. This will help a fast inspection where you - don't care about the details, but just want to remember/see what that - file is (generally) for. This information must of course be commented - (its for a human), but this is kept separate from the general - recommendation on comments, because this is a comment for the whole - file, not each step within it. - - - - **Make programming**: Here are some experiences that we have come to - learn over the years in using Make and are useful/handy in research - contexts. - - - *Environment of each recipe*: If you need to define a special - environment (or aliases, or scripts to run) for all the recipes in - your Makefiles, you can use a Bash startup file - `reproduce/software/shell/bashrc.sh`. This file is loaded before every - Make recipe is run, just like the `.bashrc` in your home directory is - loaded every time you start a new interactive, non-login terminal. See - the comments in that file for more. - - - *Automatic variables*: These are wonderful and very useful Make - constructs that greatly shrink the text, while helping in - read-ability, robustness (less bugs in typos for example) and - generalization. For example even when a rule only has one target or - one prerequisite, always use `$@` instead of the target's name, `$<` - instead of the first prerequisite, `$^` instead of the full list of - prerequisites and etc. You can see the full list of automatic - variables - [here](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html). If - you use GNU Make, you can also see this page on your command-line: - - ```shell - $ info make "automatic variables" - ``` - - - *Debug*: Since Make doesn't follow the common top-down paradigm, it - can be a little hard to get accustomed to why you get an error or - un-expected behavior. In such cases, run Make with the `-d` - option. With this option, Make prints a full list of exactly which - prerequisites are being checked for which targets. Looking - (patiently) through this output and searching for the faulty - file/step will clearly show you any mistake you might have made in - defining the targets or prerequisites. - - - *Large files*: If you are dealing with very large files (thus having - multiple copies of them for intermediate steps is not possible), one - solution is the following strategy (Also see the next item on "Fast - access to temporary files"). Set a small plain text file as the - actual target and delete the large file when it is no longer needed - by the project (in the last rule that needs it). Below is a simple - demonstration of doing this. In it, we use Gnuastro's Arithmetic - program to add all pixels of the input image with 2 and create - `large1.fits`. We then subtract 2 from `large1.fits` to create - `large2.fits` and delete `large1.fits` in the same rule (when its no - longer needed). We can later do the same with `large2.fits` when it - is no longer needed and so on. - ``` - large1.fits.txt: input.fits - astarithmetic $< 2 + --output=$(subst .txt,,$@) - echo "done" > $@ - large2.fits.txt: large1.fits.txt - astarithmetic $(subst .txt,,$<) 2 - --output=$(subst .txt,,$@) - rm $(subst .txt,,$<) - echo "done" > $@ - ``` - A more advanced Make programmer will use Make's [call - function](https://www.gnu.org/software/make/manual/html_node/Call-Function.html) - to define a wrapper in `reproduce/analysis/make/initialize.mk`. This - wrapper will replace `$(subst .txt,,XXXXX)`. Therefore, it will be - possible to greatly simplify this repetitive statement and make the - code even more readable throughout the whole project. - - - *Fast access to temporary files*: Most Unix-like operating systems - will give you a special shared-memory device (directory): on systems - using the GNU C Library (all GNU/Linux system), it is `/dev/shm`. The - contents of this directory are actually in your RAM, not in your - persistence storage like the HDD or SSD. Reading and writing from/to - the RAM is much faster than persistent storage, so if you have enough - RAM available, it can be very beneficial for large temporary files to - be put there. You can use the `mktemp` program to give the temporary - files a randomly-set name, and use text files as targets to keep that - name (as described in the item above under "Large files") for later - deletion. For example, see the minimal working example Makefile below - (which you can actually put in a `Makefile` and run if you have an - `input.fits` in the same directory, and Gnuastro is installed). - ``` - .ONESHELL: - .SHELLFLAGS = -ec - all: mean-std.txt - shm-maneage := /dev/shm/$(shell whoami)-maneage-XXXXXXXXXX - large1.txt: input.fits - out=$$(mktemp $(shm-maneage)) - astarithmetic $< 2 + --output=$$out.fits - echo "$$out" > $@ - large2.txt: large1.txt - input=$$(cat $<) - out=$$(mktemp $(shm-maneage)) - astarithmetic $$input.fits 2 - --output=$$out.fits - rm $$input.fits $$input - echo "$$out" > $@ - mean-std.txt: large2.txt - input=$$(cat $<) - aststatistics $$input.fits --mean --std > $@ - rm $$input.fits $$input - ``` - The important point here is that the temporary name template - (`shm-maneage`) has no suffix. So you can add the suffix - corresponding to your desired format afterwards (for example - `$$out.fits`, or `$$out.txt`). But more importantly, when `mktemp` - sets the random name, it also checks if no file exists with that name - and creates a file with that exact name at that moment. So at the end - of each recipe above, you'll have two files in your `/dev/shm`, one - empty file with no suffix one with a suffix. The role of the file - without a suffix is just to ensure that the randomly set name will - not be used by other calls to `mktemp` (when running in parallel) and - it should be deleted with the file containing a suffix. This is the - reason behind the `rm $$input.fits $$input` command above: to make - sure that first the file with a suffix is deleted, then the core - random file (note that when working in parallel on powerful systems, - in the time between deleting two files of a single `rm` command, many - things can happen!). When using Maneage, you can put the definition - of `shm-maneage` in `reproduce/analysis/make/initialize.mk` to be - usable in all the different Makefiles of your analysis, and you won't - need the three lines above it. **Finally, BE RESPONSIBLE:** after you - are finished, be sure to clean up any possibly remaining files (due - to crashes in the processing while you are working), otherwise your - RAM may fill up very fast. You can do it easily with a command like - this on your command-line: `rm -f /dev/shm/$(whoami)-*`. - - - - **Software tarballs and raw inputs**: It is critically important to - document the raw inputs to your project (software tarballs and raw - input data): - - - *Keep the source tarball of dependencies*: After configuration - finishes, the `.build/software/tarballs` directory will contain all - the software tarballs that were necessary for your project. You can - mirror the contents of this directory to keep a backup of all the - software tarballs used in your project (possibly as another version - controlled repository) that is also published with your project. Note - that software web-pages are not written in stone and can suddenly go - offline or not be accessible in some conditions. This backup is thus - very important. If you intend to release your project in a place like - Zenodo, you can upload/keep all the necessary tarballs (and data) - there with your - project. [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746) is - one example of how the data, Gnuastro (main software used) and all - major Gnuastro's dependencies have been uploaded with the project's - source. Just note that this is only possible for [free - software](https://www.gnu.org/philosophy/free-sw.html). - - - *Keep your input data*: The input data is also critical to the - project's reproducibility, so like the above for software, make sure - you have a backup of them, or their persistent identifiers (PIDs). - - - **Version control**: Version control is a critical component of - Maneage. Here are some tips to help in effectively using it. - - - *Regular commits*: It is important (and extremely useful) to have the - history of your project under version control. So try to make commits - regularly (after any meaningful change/step/result). - - - *Commit message*: The commit message is a very important and useful - aspect of version control. To make the commit message useful for - others (or yourself, one year later), it is good to follow a - consistent style. Maneage already has a consistent formatting - (described below), which you can also follow in your project if you - like. You can see many examples by running `git log` in the `maneage` - branch. If you intend to push commits to Maneage, for the consistency - of Maneage, it is necessary to follow these guidelines. 1) No line - should be more than 75 characters (to enable easy reading of the - message when you run `git log` on the standard 80-character - terminal). 2) The first line is the title of the commit and should - summarize it (so `git log --oneline` can be useful). The title should - also not end with a point (`.`, because its a short single sentence, - so a point is not necessary and only wastes space). 3) After the - title, leave an empty line and start the body of your message - (possibly containing many paragraphs). 4) Describe the context of - your commit (the problem it is trying to solve) as much as possible, - then go onto how you solved it. One suggestion is to start the main - body of your commit with "Until now ...", and continue describing the - problem in the first paragraph(s). Afterwards, start the next - paragraph with "With this commit ...". - - - *Project outputs*: During your research, it is possible to checkout a - specific commit and reproduce its results. However, the processing - can be time consuming. Therefore, it is useful to also keep track of - the final outputs of your project (at minimum, the paper's PDF) in - important points of history. However, keeping a snapshot of these - (most probably large volume) outputs in the main history of the - project can unreasonably bloat it. It is thus recommended to make a - separate Git repo to keep those files and keep your project's source - as small as possible. For example if your project is called - `my-exciting-project`, the name of the outputs repository can be - `my-exciting-project-output`. This enables easy sharing of the output - files with your co-authors (with necessary permissions) and not - having to bloat your email archive with extra attachments also (you - can just share the link to the online repo in your - communications). After the research is published, you can also - release the outputs repository, or you can just delete it if it is - too large or un-necessary (it was just for convenience, and fully - reproducible after all). For example Maneage's output is available - for demonstration in [a - separate](http://git.maneage.org/output-raw.git/) repository. - - - *Full Git history in one file*: When you are publishing your project - (for example to Zenodo for long term preservation), it is more - convenient to have the whole project's Git history into one file to - save with your datasets. After all, you can't be sure that your - current Git server (for example GitLab, Github, or Bitbucket) will be - active forever. While they are good for the immediate future, you - can't rely on them for archival purposes. Fortunately keeping your - whole history in one file is easy with Git using the following - commands. To learn more about it, run `git help bundle`. - - - "bundle" your project's history into one file (just don't forget to - change `my-project-git.bundle` to a descriptive name of your - project): - - ```shell - $ git bundle create my-project-git.bundle --all - ``` - - - You can easily upload `my-project-git.bundle` anywhere. Later, if - you need to un-bundle it, you can use the following command. - - ```shell - $ git clone my-project-git.bundle - ``` - - - - - - - - - - -Future improvements -=================== + + +## Modularity + +Modularity is the key to easy and clean growth of a project. So it is +always best to break up a job into as many sub-components as +reasonable. Here are some tips to stay modular. + +- *Short recipes*: if you see the recipe of a rule becoming more than a + handful of lines which involve significant processing, it is probably a + good sign that you should break up the rule into its main + components. Try to only have one major processing step per rule. + +- *Context-based (many) Makefiles*: For maximum modularity, this design + allows easy inclusion of many Makefiles: in + `reproduce/analysis/make/*.mk` for analysis steps, and + `reproduce/software/make/*.mk` for building software. So keep the rules + for closely related parts of the processing in separate Makefiles. + +- *Descriptive names*: Be very clear and descriptive with the naming of the + files and the variables because a few months after the processing, it + will be very hard to remember what each one was for. Also this helps + others (your collaborators or other people reading the project source + after it is published) to more easily understand your work and find + their way around. + +- *Naming convention*: As the project grows, following a single standard or + convention in naming the files is very useful. Try best to use multiple + word filenames for anything that is non-trivial (separating the words + with a `-`). For example if you have a Makefile for creating a catalog + and another two for processing it under models A and B, you can name + them like this: `catalog-create.mk`, `catalog-model-a.mk` and + `catalog-model-b.mk`. In this way, when listing the contents of + `reproduce/analysis/make` to see all the Makefiles, those related to the + catalog will all be close to each other and thus easily found. This also + helps in auto-completions by the shell or text editors like Emacs. + +- *Source directories*: If you need to add files in other languages for + example in shell, Python, AWK or C, keep the files in the same language + in a separate directory under `reproduce/analysis`, with the appropriate + name. + +- *Configuration files*: If your research uses special programs as part of + the processing, put all their configuration files in a devoted directory + (with the program's name) within `reproduce/software/config`. It is much + cleaner and readable (thus less buggy) to avoid mixing the configuration + files, even if there is no technical necessity. + + + +## Contents of source files + +It is good practice to follow the following recommendations on the contents +of your files, whether they are source code for a program, Makefiles, +scripts or configuration files (copyrights aren't necessary for the +latter). + +- *Copyright*: Always start a file containing programming constructs with a + copyright statement like the ones that Maneage starts with (for example + in the top level `Makefile`). + +- *Comments*: Comments are vital for readability (by yourself in two + months, or others). Describe everything you can about why you are doing + something, how you are doing it, and what you expect the result to + be. Write the comments as if it was what you would say to describe the + variable, recipe or rule to a friend sitting beside you. When writing + the project it is very tempting to just steam ahead with commands and + codes, but be patient and write comments before the rules or + recipes. This will also allow you to think more about what you should be + doing. Also, in several months when you come back to the code, you will + appreciate the effort of writing them. Just don't forget to also read + and update the comment first if you later want to make changes to the + code (variable, recipe or rule). As a general rule of thumb: first the + comments, then the code. + +- *File title*: In general, it is good practice to start all files with a + single line description of what that particular file does. If further + information about the totality of the file is necessary, add it after a + blank line. This will help a fast inspection where you don't care about + the details, but just want to remember/see what that file is (generally) + for. This information must of course be commented (its for a human), but + this is kept separate from the general recommendation on comments, + because this is a comment for the whole file, not each step within it. + + + +## Make programming + +Here are some experiences that we have come to learn over the years in +using Make and are useful/handy in research contexts. + +### Environment of each recipe + +If you need to define a special environment (or aliases, or scripts to run) +for all the recipes in your Makefiles, you can use a Bash startup file +`reproduce/software/shell/bashrc.sh`. This file is loaded before every Make +recipe is run, just like the `.bashrc` in your home directory is loaded +every time you start a new interactive, non-login terminal. See the +comments in that file for more. + +### Automatic variables + +These are wonderful and very useful Make constructs that greatly shrink the +text, while helping in read-ability, robustness (less bugs in typos for +example) and generalization. For example even when a rule only has one +target or one prerequisite, always use `$@` instead of the target's name, +`$<` instead of the first prerequisite, `$^` instead of the full list of +prerequisites and etc. You can see the full list of automatic variables +[here](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html). If +you use GNU Make, you can also see this page on your command-line: + +```shell +$ info make "automatic variables" +``` + +### Debugging Make dependencies + +Since Make doesn't follow the common top-down paradigm, it can be a little +hard to get accustomed to why you get an error or un-expected behavior. In +such cases, run Make with the `-d` option. With this option, Make prints a +full list of exactly which prerequisites are being checked for which +targets. Looking (patiently) through this output and searching for the +faulty file/step will clearly show you any mistake you might have made in +defining the targets or prerequisites. + +### Large files + +If you are dealing with very large files (thus having multiple copies of +them for intermediate steps is not possible), one solution is the following +strategy (Also see the next item on "Fast access to temporary files"). Set +a small plain text file as the actual target and delete the large file when +it is no longer needed by the project (in the last rule that needs +it). Below is a simple demonstration of doing this. In it, we use +Gnuastro's Arithmetic program to add all pixels of the input image with 2 +and create `large1.fits`. We then subtract 2 from `large1.fits` to create +`large2.fits` and delete `large1.fits` in the same rule (when its no longer +needed). We can later do the same with `large2.fits` when it is no longer +needed and so on. + +```make +large1.fits.txt: input.fits + astarithmetic $< 2 + --output=$(subst .txt,,$@) + echo "done" > $@ +large2.fits.txt: large1.fits.txt + astarithmetic $(subst .txt,,$<) 2 - --output=$(subst .txt,,$@) + rm $(subst .txt,,$<) + echo "done" > $@ +``` + +A more advanced Make +programmer will use Make's [call +function](https://www.gnu.org/software/make/manual/html_node/Call-Function.html) +to define a wrapper in `reproduce/analysis/make/initialize.mk`. This +wrapper will replace `$(subst .txt,,XXXXX)`. Therefore, it will be possible +to greatly simplify this repetitive statement and make the code even more +readable throughout the whole project. + +### Fast access to temporary files + +Most Unix-like operating systems will give you a special shared-memory +device (directory): on systems using the GNU C Library (all GNU/Linux +system), it is `/dev/shm`. The contents of this directory are actually in +your RAM, not in your persistence storage like the HDD or SSD. Reading and +writing from/to the RAM is much faster than persistent storage, so if you +have enough RAM available, it can be very beneficial for large temporary +files to be put there. You can use the `mktemp` program to give the +temporary files a randomly-set name, and use text files as targets to keep +that name (as described in the item above under "Large files") for later +deletion. For example, see the minimal working example Makefile below +(which you can actually put in a `Makefile` and run if you have an +`input.fits` in the same directory, and Gnuastro is installed). + +```make +.ONESHELL: +.SHELLFLAGS = -ec +all: mean-std.txt +shm-maneage := /dev/shm/$(shell whoami)-maneage-XXXXXXXXXX +large1.txt: input.fits + out=$$(mktemp $(shm-maneage)) + astarithmetic $< 2 + --output=$$out.fits + echo "$$out" > $@ +large2.txt: large1.txt + input=$$(cat $<) + out=$$(mktemp $(shm-maneage)) + astarithmetic $$input.fits 2 - --output=$$out.fits + rm $$input.fits $$input + echo "$$out" > $@ +mean-std.txt: large2.txt + input=$$(cat $<) + aststatistics $$input.fits --mean --std > $@ + rm $$input.fits $$input +``` + +The important point here is that the temporary name template +(`shm-maneage`) has no suffix. So you can add the suffix corresponding to +your desired format afterwards (for example `$$out.fits`, or +`$$out.txt`). But more importantly, when `mktemp` sets the random name, it +also checks if no file exists with that name and creates a file with that +exact name at that moment. So at the end of each recipe above, you'll have +two files in your `/dev/shm`, one empty file with no suffix one with a +suffix. The role of the file without a suffix is just to ensure that the +randomly set name will not be used by other calls to `mktemp` (when running +in parallel) and it should be deleted with the file containing a +suffix. This is the reason behind the `rm $$input.fits $$input` command +above: to make sure that first the file with a suffix is deleted, then the +core random file (note that when working in parallel on powerful systems, +in the time between deleting two files of a single `rm` command, many +things can happen!). When using Maneage, you can put the definition of +`shm-maneage` in `reproduce/analysis/make/initialize.mk` to be usable in +all the different Makefiles of your analysis, and you won't need the three +lines above it. **Finally, BE RESPONSIBLE:** after you are finished, be +sure to clean up any possibly remaining files (due to crashes in the +processing while you are working), otherwise your RAM may fill up very +fast. You can do it easily with a command like this on your command-line: +`rm -f /dev/shm/$(whoami)-*`. + +### Pattern rules with multiple components + +In a large pipeline, as you go from the lower-level to higher-level +targets, the higher-level targets can depend on multiple lower-level +files. However, Make's [pattern +rules](https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html) +only allow the extraction of a single component. The minimal working +example below shows how to account for such situations. + +The targets in this minimal working example are in the format of +`file-*.fits`. The low-level files only have one character instead of the +`*` (for example `file-1.fits`). The high-level targets have three +characters there (with `-` as a delimiter; for example +`file-1-a.fits`). This Makefile will ensure that `file-1-a.fits` is always +built as soon as `file-1.fits` and `file-a.fits` have been created. As +described in the comments ontop of the `$(targets)` rule, this uses GNU +Make's [secondary +expansion](https://www.gnu.org/software/make/manual/html_node/Secondary-Expansion.html). + +```make +# Low-level target file components. +numbers = 1 2 3 4 +letters = a b c d + +# Ultimate (first) target that depends on the last target in this Makefile +# to simplify the writing of the rest (in a human-friendly manner: +# redundant for Make). This is not necessray in Maneage; only to help the +# readability of this file. +all: final +.PHONY: all final + +# This allows using functions in prequisite lists as described below. +.SECONDEXPANSION: + +# Rule to build the alphabetical low-level files. +alphabets := $(foreach i,$(letters),file-$(i).fits) +$(alphabets): file-%.fits: + @echo "Making $@"; touch $@ + +# Rule to build the numeric low-level files. +numerics := $(foreach i,$(numbers),file-$(i).fits) +$(numerics): file-%.fits: + @echo "Making $@"; touch $@ + +# Rule to build the high-level target: each high-level target depends on an +# alphabetic and numeric low-level target. All functions in the +# prerequisite need '$$(...)' instead of '$(...)' to avoid expansion in +# the first parsing of the file and let it get expanded in the second +# parsing (thus necessity of '.SECONDEXPANSION' above). +targets := $(foreach i,$(numbers), \ + $(foreach j,$(letters), \ + file-$(i)-$(j).fits) ) +$(targets): file-%.fits: file-$$(word 1,$$(subst -, ,%)).fits \ + file-$$(word 2,$$(subst -, ,%)).fits + @echo "Prequisities of $@ are: $^" + +# Dependency of 'all' +final: $(targets) +``` + + + +## Inputs (software tarballs and data) + +It is critically important to document the raw inputs to your project +(software tarballs and raw input data): + +- *Keep the source tarball of dependencies*: After configuration finishes, + the `.build/software/tarballs` directory will contain all the software + tarballs that were necessary for your project. You can mirror the + contents of this directory to keep a backup of all the software tarballs + used in your project (possibly as another version controlled repository) + that is also published with your project. Note that software web-pages + are not written in stone and can suddenly go offline or not be + accessible in some conditions. This backup is thus very important. If + you intend to release your project in a place like Zenodo, you can + upload/keep all the necessary tarballs (and data) there with your + project. [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746) is one + example of how the data, Gnuastro (main software used) and all major + Gnuastro's dependencies have been uploaded with the project's + source. Just note that this is only possible for [free + software](https://www.gnu.org/philosophy/free-sw.html). + +- *Keep your input data*: The input data is also critical to the project's + reproducibility, so like the above for software, make sure you have a + backup of them, or their persistent identifiers (PIDs). + + + +## Project outputs + +During your research, it is possible to checkout a specific commit and +reproduce its results. However, the processing can be time +consuming. Therefore, it is useful to also keep track of the final outputs +of your project (at minimum, the paper's PDF) in important points of +history. However, keeping a snapshot of these (most probably large volume) +outputs in the main history of the project can unreasonably bloat it. It is +thus recommended to make a separate Git repo to keep those files and keep +your project's source as small as possible. For example if your project is +called `my-exciting-project`, the name of the outputs repository can be +`my-exciting-project-output`. This enables easy sharing of the output files +with your co-authors (with necessary permissions) and not having to bloat +your email archive with extra attachments also (you can just share the link +to the online repo in your communications). After the research is +published, you can also release the outputs repository, or you can just +delete it if it is too large or un-necessary (it was just for convenience, +and fully reproducible after all). For example Maneage's output is +available for demonstration in [a +separate](http://git.maneage.org/output-raw.git/) repository. + + + +## Git and version control + +Version control is a critical component of Maneage. Here are some tips to +help in effectively using it. + +### Committing + +Every commit is a significant point in the history of your project: when +you consider one component of the job to be complete and ready to move to +the next component. This significance is very subjective: it can be a +significant change in one part of one file, or many files, or even a +temporal criteria: a milestone is reaching and you need to change +topic. Here are some tips on good habits in committing. + +- *Regular commits*: It is important (and extremely useful) to have the + history of your project under version control. So try to make commits + regularly (after any meaningful change/step/result). + +- *Commit message*: The commit message is a very important and useful + aspect of version control. To make the commit message useful for others + (or yourself, one year later), it is good to follow a consistent + style. Maneage already has a consistent formatting (described below), + which you can also follow in your project if you like. You can see many + examples by running `git log` in the `maneage` branch. If you intend to + push commits to Maneage, for the consistency of Maneage, it is necessary + to follow these guidelines. 1) No line should be more than 75 characters + (to enable easy reading of the message when you run `git log` on the + standard 80-character terminal). 2) The first line is the title of the + commit and should summarize it (so `git log --oneline` can be + useful). The title should also not end with a point (`.`, because its a + short single sentence, so a point is not necessary and only wastes + space). 3) After the title, leave an empty line and start the body of + your message (possibly containing many paragraphs). 4) Describe the + context of your commit (the problem it is trying to solve) as much as + possible, then go onto how you solved it. One suggestion is to start the + main body of your commit with "Until now ...", and continue describing + the problem in the first paragraph(s). Afterwards, start the next + paragraph with "With this commit ...". + +### Full Git history in one file + +When you are publishing your project (for example to Zenodo for long term +preservation), it is more convenient to have the whole project's Git +history into one file to save with your datasets. After all, you can't be +sure that your current Git server (for example GitLab, Github, or +Bitbucket) will be active forever. While they are good for the immediate +future, you can't rely on them for archival purposes. Fortunately keeping +your whole history in one file is easy with Git using the following +commands. To learn more about it, run `git help bundle`. + +Run the following command to "bundle" your project's history into one file +(just don't forget to change `my-project-git.bundle` to a descriptive name +of your project): + +```shell +$ git bundle create my-project-git.bundle --all +``` + +You can easily upload `my-project-git.bundle` anywhere. Later, if you need +to un-bundle it, you can use the following command. + +```shell +$ git clone my-project-git.bundle +``` + +### File modification dates (meta data) + +While Git does an excellent job at keeping a history of the contents of +files, it makes no effort in keeping the file meta data, and in particular +the dates of files. Therefore when you checkout to a different branch, +files that are re-written by Git will have a newer date than the other +project files. However, file dates are important in the current design of +Maneage: Make checks the dates of the prerequisite files and target files +to see if the target should be re-built. + +This is relevant, for example, in the `analysis/` side of calculations when +a rule in a `reproduce/analysis/make/SOMETHING.mk` is dependent on +`reproduce/analysis/conf/SOMETHING.conf` (where `SOMETHING` is one of the +science pipeline steps that you have prepared). Switching back and forth +between git branches will update the timestamp on `SOMETHING.conf`, +incorrectly implying, according to the Make rules, that a rule in +SOMETHING.mk has to be run again. + +To address this problem, you can use the `touch` command to backdate the +timestamp of files that you want to be considered to be old. For example +with the command below: + +```shell +$ touch --date=2000-01-01 reproduce/analysis/conf/SOMETHING.conf +$ ls -l reproduce/analysis/conf/SOMETHING.conf # to check +``` + +You may want to use a date that is obviously (to you) fake, so that you +don't accidentally believe the date later on. + + + + + + + + + + +# Maneaged papers already published + +The list below shows some of the works that have already been published +with (earlier versions of) Maneage, and some that have been recently +submitted for peer review. Maneage is evolving rapidly, so some details +will differ between the different versions. The more recent papers will +tend to be the most useful as good working examples if you are just +starting. + +In the list below, by clicking on the paper's publication year, you will be +taken to the paper's indexed page on the [SciX Digital +Library](https://scixplorer.org) (successor to the [NASA +ADS](https://ui.adsabs.harvard.edu)) which contains further bibliographic +information like title, DOI and etc, in a unified format (independent of +which journal the paper is published in). + + - Eskandarlou et + al. ([2026](https://scixplorer.org/abs/2026A%26A...709A.210E), A&A + Volume 709, id.A210, 17 pp; + arXiv:[2603.22166](https://arxiv.org/abs/2603.22166)). The + project's version controlled source is on + [Gitlab](https://gitlab.com/sepideh.esk/alba), necessary software, + outputs and backup of history are available at + [zenodo.17674805](http://doi.org/10.5281/zenodo.17674805); and the + archived git history is available at + [swh:1:dir:7c203a565ec6ef40d8419dbdf70e8639b78f4b99](https://archive.softwareheritage.org/swh:1:dir:7c203a565ec6ef40d8419dbdf70e8639b78f4b99;origin=https://gitlab.com/Sepideh.Esk/alba;visit=swh:1:snp:adc584ff3540ea3151de6ebc8462805e8818b1f4;anchor=swh:1:rev:f9d0f9cb25c80415107c962cc53d369a650e9b82). + - Eskandarlou et + al. ([2026](https://scixplorer.org/abs/2026A%26A...705A..67E), A&A, + Volume 705, id.A67, 14 pp; + arXiv:[2510.12940](https://arxiv.org/abs/2510.12940)). The project's + version controlled source is on + [Gitlab](https://gitlab.com/sepideh.esk/psf-j-plus), necessary software, + outputs and backup of history are available at + [zenodo.17348653](http://doi.org/10.5281/zenodo.17348653); and the + archived git history is available at + [swh:1:dir:fc95ed9ad173de6fa64690e3d71ab041c630f32c](https://archive.softwareheritage.org/swh:1:dir:fc95ed9ad173de6fa64690e3d71ab041c630f32c;origin=https://gitlab.com/Sepideh.Esk/psf-j-plus;visit=swh:1:snp:31bb4550902f7902a0a65d0674106271de776d3e;anchor=swh:1:rev:4860c70d6285a60300e429889e8637e98568a915). + - Saremi et al. ([2025](https://scixplorer.org/abs/2025A%26A...701A.116S), + A&A, Volume 701, id.A116, 18 pp.; + arXiv:[2508.02780](https://arxiv.org/abs/2508.02780)). The project's + version controlled source is on + [Gitlab](https://gitlab.com/nasim-projects/pipeline), necessary + software, outputs and backup of history are available at + [zenodo.16152699](https://doi.org/10.5281/zenodo.16152699); and the + archived git history is available at + [swh:1:dir:b3657cfb6053fd976695bd63c15cb99e5095648a](https://archive.softwareheritage.org/swh:1:dir:b3657cfb6053fd976695bd63c15cb99e5095648a;origin=https://gitlab.com/nasim-projects/pipeline;visit=swh:1:snp:ab7c6f0b9999f42d77154103c1bc082fa23b325c;anchor=swh:1:rev:afeb282c01983cba2a11eb4b2f25d5a40d35c164). + - Eskandarlou & Akhlaghi + ([2024](https://scixplorer.org/abs/2024RNAAS...8..168E), RNAAS, Volume + 8, Issue 6, id.168; + arXiv:[2406.14619](https://arxiv.org/abs/2406.14619)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `polar-plot` + branch). Necessary software, outputs and backup of history are available + at [zenodo.11403643](https://doi.org/10.5281/zenodo.11403643); and the + archived git history is available at + [swh:1:dir:4e09bf85f9f87336fa55920bf67e7bcf6d58bbd5](https://archive.softwareheritage.org/swh:1:dir:4e09bf85f9f87336fa55920bf67e7bcf6d58bbd5;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:557ee1a90de465659659ecc46df0c5ce29d0bb61;anchor=swh:1:rev:375e12e52080006be6a28e10980e79ef54d13d1d). + - Infante-Sainz et + al. ([2024](https://scixplorer.org/abs/2024RNAAS...8...22I), RNAAS, + Volume 8, Issue 1, id.22; + arXiv:[2401.05303](https://arxiv.org/abs/2401.05303)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `radial-profile` + branch). Necessary software, outputs and backup of history are available + at [zenodo.10124582](https://doi.org/10.5281/zenodo.10124582); and the + archived git history is available at + [swh:1:dir:d5029e066916cb64f0d95d20eb88294acc78b2b1](https://archive.softwareheritage.org/swh:1:dir:d5029e066916cb64f0d95d20eb88294acc78b2b1;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:b065324c2ef3b48bc26e8f30e48102a1abd2052f;anchor=swh:1:rev:61764447b16da44538e5ddbf7fb69937ba138e81). + - Infante-Sainz & Akhlaghi + ([2024](https://scixplorer.org/abs/2024RNAAS...8...10I), RNAAS, Volume + 8, Issue 1, id.10; + arXiv:[2401.03814](https://arxiv.org/abs/2401.03814)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `color-faint-gray` + branch). Necessary software, outputs and backup of history are available + at [zenodo.10058165](https://doi.org/10.5281/zenodo.10058165); and the + archived git history is available at + [swh:1:dir:1064a48d4bb58d6684c3df33c6633a04d4141d2d](https://archive.softwareheritage.org/swh:1:dir:1064a48d4bb58d6684c3df33c6633a04d4141d2d;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:a083ff647c571f895d1ccc9f7432fa1b9a1d03a8;anchor=swh:1:rev:ff77b619daa50b05ddd83206d979d1f8a53d040b). + - Peper, Roukema & Bolejko + ([2023](https://scixplorer.org/abs/2023MNRAS.525...91P), MNRAS, 525, 91, + DOI:10.1093/mnras/stad2246, + arXiv:[2304.00591](https://arxiv.org/abs/2304.00591)): The live version + of the controlled source is [at + Codeberg](https://codeberg.org/mpeper/lensing); the main input + dataset, a software snapshot, the software tarballs, the project outputs + and editing history are available at + [zenodo.8103985](https://zenodo.org/record/8103985); and the archived + git history is available at + [swh:1:dir:f57fd0e254bcb976c1ab34b6c9c9708e891f7914](https://archive.softwareheritage.org/swh:1:dir:f57fd0e254bcb976c1ab34b6c9c9708e891f7914;origin=https://codeberg.org/mpeper/lensing;visit=swh:1:snp:984413c49bb02ee0eec88c3ac01614808d57beb7;anchor=swh:1:rev:185c36f1b6263588526be3881078626869041882). + - Eskandarlou et + al. ([2023](https://scixplorer.org/abs/2023RNAAS...7..269E), RNAAS, + Volume 7, Issue 12, id.269; + arXiv:[2312.04263](https://arxiv.org/abs/2312.04263)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `zeropoint` + branch). Necessary software, outputs and backup of history are available + at [zenodo.10256845](https://doi.org/10.5281/zenodo.10256845); and the + archived git history is available at + [swh:1:dir:8b2d1f63be96de3de03aa3e2bb68fa7fa52df56f](https://archive.softwareheritage.org/swh:1:dir:8b2d1f63be96de3de03aa3e2bb68fa7fa52df56f;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:e37e226bab517eef24d854467682b2fcf5d7dc32;anchor=swh:1:rev:ea682783d83707c0e1d114a5de74a100be9f545d). + - Akhlaghi ([2023](https://scixplorer.org/abs/2023RNAAS...7..211A), RNAAS), Volume 7, Issue + 10, id.211; arXiv:[2310.15006](https://arxiv.org/abs/2310.15006)). The + project's version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the + `pointing-simulate` branch). + - Borkowska & Roukema + ([2022](https://scixplorer.org/abs/2022CQGra..39u5007B), Classical and + Quantum Gravity, Volume 39, Issue 21, id.215007, 29 pp; + arXiv:[2112.14174](https://arxiv.org/abs/2112.14174)): The live version + of the controlled source is [at + Codeberg](https://codeberg.org/boud/gevcurvtest); the main input + dataset, a software snapshot, the software tarballs, the project outputs + and editing history are available at + [zenodo.5806027](https://doi.org/10.5281/zenodo.5806027); and the + archived git history is available at + [swh:1:rev:54398b720ddbac269ede30bf1e27fe27f07567f7](https://archive.softwareheritage.org/browse/revision/54398b720ddbac269ede30bf1e27fe27f07567f7). + - Peper & Roukema ([2021](https://scixplorer.org/abs/2021MNRAS.505.1223P), + MNRAS, 505, 1223; arXiv:[2010.03742](https://arxiv.org/abs/2010.03742)): + The live version of the controlled source is [at + Codeberg](https://codeberg.org/boud/elaphrocentre); the main input + dataset, a software snapshot, the software tarballs, the project outputs + and editing history are available at + [zenodo.4699702](https://zenodo.org/record/4699702); and the archived + git history is available at + [swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e](https://archive.softwareheritage.org/swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e). + - Roukema ([2021](https://scixplorer.org/abs/2021PeerJ...911856R), PeerJ, + Vol. 9, article id. e11856; + arXiv:[2007.11779](https://arxiv.org/abs/2007.11779)): The live version + of the controlled source is [at + Codeberg](https://codeberg.org/boud/subpoisson); the main input dataset, + a software snapshot, the software tarballs, the project outputs and + editing history are available at + [zenodo.4765705](https://zenodo.org/record/4765705); and the archived + git history is available at + [swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37](https://archive.softwareheritage.org/swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37). + - Akhlaghi et al. ([2021](https://scixplorer.org/abs/2021CSE....23c..82A), + CiSE, 23(3), 82; arXiv:[2006.03018](https://arxiv.org/abs/2006.03018)): + The project's version controlled source is [on + Gitlab](https://gitlab.com/makhlaghi/maneage-paper), necessary software, + outputs and backup of history are available at + [zenodo.3872248](https://doi.org/10.5281/zenodo.3872248); and the + archived git history is available at + [swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717](https://archive.softwareheritage.org/swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717). + - Infante-Sainz et + al. ([2020](https://scixplorer.org/abs/2020MNRAS.491.5317I), MNRAS, 491, + 5317; arXiv:[1911.01430](https://arxiv.org/abs/1911.01430)): The version + controlled project source is available [on + GitLab](https://gitlab.com/infantesainz/sdss-extended-psfs-paper) and is + also archived on Zenodo with all the necessary software tarballs: + [zenodo.3524937](https://zenodo.org/record/3524937). + - Akhlaghi ([2019](https://scixplorer.org/abs/2019arXiv190911230A), IAU + Symposium 355; + arXiv:[1909.11230](https://arxiv.org/abs/1909.11230)). The version + controlled project source is available [on + GitLab](https://gitlab.com/makhlaghi/iau-symposium-355) and is also + archived on Zenodo with all the necessary software tarballs: + [zenodo.3408481](https://doi.org/10.5281/zenodo.3408481). + +The items below were the precursors to Maneage. In other words, the work to +have reproducible results in the items below lead to what is now +Maneage. They are therefore much less complete than what Maneage is today, +but are important from a historical perspective. + - Section 7.3 of Bacon et + al. ([2017](https://scixplorer.org/abs/2017A%26A...608A...1B), A&A 608, + A1; arXiv:[1710.03002](https://arxiv.org/abs/1710.03002)): The version + controlled project source is available [on + GitLab](https://gitlab.com/makhlaghi/muse-udf-origin-only-hst-magnitudes) + and a snapshot of the project along with all the necessary input + datasets and outputs is available in + [zenodo.1164774](https://doi.org/10.5281/zenodo.1164774). + - Section 4 of Bacon et + al. ([2017](https://scixplorer.org/abs/2017A%26A...608A...1B), A&A, 608, + A1; arXiv:[1710.03002](https://arxiv.org/abs/1710.03002)): The version + controlled project is available [on + GitLab](https://gitlab.com/makhlaghi/muse-udf-photometry-astrometry) and + a snapshot of the project along with all the necessary input datasets is + available in [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746). + - Akhlaghi & Ichikawa + ([2015](https://scixplorer.org/abs/2015ApJS..220....1A), ApJS, 220, 1; + arXiv:[1505.01664](https://arxiv.org/abs/1505.01664)): The version + controlled project is available [on + GitLab](https://gitlab.com/makhlaghi/NoiseChisel-paper). This is the + very first (and much less mature!) incarnation of Maneage: the history + of Maneage started more than two years after this paper was + published. It is a very rudimentary/initial implementation, thus it is + only included here for historical reasons. However, the project source + is complete, accurate and uploaded to arXiv along with the paper. + + + + + + + + + + +# Maneage Development + +This section describes Maneage's own development workflow, not the workflow +of your Maneage-based projects (where you are free to select any workflow +model that you prefer). So it is only intended for those who would like to +contribute to the core Maneage project, not those who only develop their +own project. + +## Development workflow + +Maneage users can only use it by developing it, therefore Maneage is a very +unique software: the boundary between a user and developer is blurry by +design/philosophy. We hope that all users can contribute to the `maneage` +branch for project-agnostic bugs/features that they fix/add. In this +section, the development workflow is described: + - At any point that you find a bug or want to share a fix, anyone is + welcome to open a "Bug" or "Task" on GNU Savannah (which is the main + project management interface of Maneage). In case the submitter already + has a commit to merge, they can fork Maneage (similar to [this forking + tutorial](https://www.gnu.org/software/gnuastro/manual/html_node/Forking-tutorial.html)) + and share their commit in the Savannah post. + - The review of bugs and merge requests will happen at the earliest + possible time for the maintainer. At this point, the maintainer will + create a new development branch over the `maneage` branch in a + development repository (not the main Maneage repository) and all + separate merge requests will be rebased over that branch. Finally, all + the commits in that branch will be `squash`ed into a single commit that + will be merged over the `maneage` branch. Of course, every contributor + will be acknowledged in the squashed commit for their contribution. + - The `squash`ing into a single commit is intentionally done because the + `maneage` branch is shared between all projects that use Maneage and + we do not want to introduce complexity in their history by preserving + all the various development commits that were done before the + merge. + - After the merge, the development branch may be deleted from the + development repository: any historically relevant point that you find + when making a commit should be written as comments within the source + (to be preserved during the squash) and not in the commit message. + - Checklist for the maintainer: + - Make sure all contributors are properly acknowledged. + - In case any software was added in the 'maneage' branch or the version + of existing software was updated, upload/push the new tarballs to all + the backup repositories (in + `reproduce/software/config/servers-backup.conf`). + - The final commit amend (done to edit the final text of the commit + message) should be like this: `git commit --amend --date=now`. This + helps have a date on the commit that matches the merge (independent of + which development commit was squashed first). + +## Future improvements This is an evolving project and as time goes on, it will evolve and become more robust. Some of the most prominent issues we plan to implement in the future are listed below, please join us if you are interested. -Package management ------------------- +### Full documentation in Info + +Currently the only documentation for Maneage's development is this file! +But this format is not too convenient for a complete documentation: README +files are only meant to point readers to the proper place of the manual, +not be the whole documentation themselves! This choice was due to time +constraints. Work has already started on this in the +`[texibook](https://gitlab.com/maneage/project-dev/-/tree/texibook)` branch +that will build a professional manual in info, HTML and PDF formats using +Texinfo during the configuration phase. But this still needs to be +finished. + +### Package management It is important to have control of the environment of the project. Maneage currently builds the higher-level programs (for example GNU Bash, GNU Make, @@ -1738,46 +2205,15 @@ commonly have large overlaps in lower-level programs). -Appendix: Necessity of exact reproduction in scientific research -================================================================ - -In case [the link above](http://akhlaghi.org/reproducible-science.html) is -not accessible at the time of reading, here is a copy of the introduction -of that link, describing the necessity for a reproducible project like this -(copied on February 7th, 2018): - -The most important element of a "scientific" statement/result is the fact -that others should be able to falsify it. The Tsunami of data that has -engulfed astronomers in the last two decades, combined with faster -processors and faster internet connections has made it much more easier to -obtain a result. However, these factors have also increased the complexity -of a scientific analysis, such that it is no longer possible to describe -all the steps of an analysis in the published paper. Citing this -difficulty, many authors suffice to describing the generalities of their -analysis in their papers. - -However, It is impossible to falsify (or even study) a result if you can't -exactly reproduce it. The complexity of modern science makes it vitally -important to exactly reproduce the final result. Because even a small -deviation can be due to many different parts of an analysis. Nature is -already a black box which we are trying so hard to comprehend. Not letting -other scientists see the exact steps taken to reach a result, or not -allowing them to modify it (do experiments on it) is a self-imposed black -box, which only exacerbates our ignorance. - -Other scientists should be able to reproduce, check and experiment on the -results of anything that is to carry the "scientific" label. Any result -that is not reproducible (due to incomplete information by the author) is -not scientific: the readers have to have faith in the subjective experience -of the authors in the very important choice of configuration values and -order of operations: this is contrary to the scientific spirit. +# Copyright information +Contributors:\ +Copyright (C) 2020-2026 Raul Infante-Sainz <infantesainz@gmail.com>\ +Copyright (C) 2021-2026 Boud Roukema <boud@astro.uni.torun.pl> -Copyright information ---------------------- This file is part of Maneage (https://maneage.org). This file is free software: you can redistribute it and/or modify it @@ -1,7 +1,7 @@ Reproducible source for XXXXXXXXXXXXXXXXX ------------------------------------------------------------------------- -Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ +Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ See the end of the file for license conditions. This is the reproducible project source for the paper titled "**XXX XXXXX @@ -292,571 +292,86 @@ light and should be very fast. -### Building in Docker containers - -Docker containers are a common way to build projects in an independent -filesystem, and an almost independent operating system. Containers thus -allow using GNU/Linux operating systems within proprietary operating -systems like macOS or Windows. But without the overhead and huge file size -of virtual machines. Furthermore containers allow easy movement of built -projects from one system to another without rebuilding. Just note that -Docker images are large binary files (+1 Gigabytes) and may not be usable -in the future (for example with new Docker versions not reading old -images). Containers are thus good for temporary/testing phases of a -project, but shouldn't be what you archive for the long term! - -Hence if you want to save and move your maneaged project within a Docker -image, be sure to commit all your project's source files and push them to -your external Git repository (you can do these within the Docker image as -explained below). This way, you can always recreate the container with -future technologies too. Generally, if you are developing within a -container, its good practice to recreate it from scratch every once in a -while, to make sure you haven't forgot to include parts of your work in -your project's version-controlled source. In the sections below we also -describe how you can use the container **only for the software -environment** and keep your data and project source on your host. - -#### Dockerfile for a Maneaged project, and building a Docker image - -Below is a series of recommendations on the various components of a -`Dockerfile` optimized to store the *built state of a maneaged project* as -a Docker image. Each component is also accompanied with -explanations. Simply copy the code blocks under each item into a plain-text -file called `Dockerfile`, in the same order of the items. Don't forget to -implement the suggested corrections (in particular step 4). - -**NOTE: Internet for TeXLive installation:** If you have the project -software tarballs and input data (optional features described below) you -can disable internet. In this situation, the configuration and analysis -will be exactly reproduced, the final LaTeX macros will be created, and all -results will be verified successfully. However, no final `paper.pdf` will -be created to visualize/combine everything in one easy-to-read file. Until -[task 15267](https://savannah.nongnu.org/task/?15267) is complete, we need -internet to install TeXLive packages (using TeXLive's own package manager -`tlmgr`) in the `./project configure` phase. This won't stop the -configuration, and it will finish successfully (since all the analysis can -still be reproduced). We are working on completing this task as soon as -possible, but until then, if you want to disable internet *and* you want to -build the final PDF, please disable internet after the configuration -phase. Note that only the necessary TeXLive packages are installed (~350 -MB), not the full TeXLive collection! - - 0. **Summary:** If you are already familiar with Docker, then the full - Dockerfile to get the project environment setup is shown here (without - any comments or explanations, because explanations are done in the next - items). Note that the last two `COPY` lines (to copy the directory - containing software tarballs used by the project and the possible input - databases) are optional because they will be downloaded if not - available. You can also avoid copying over all, and simply mount your - host directories within the image, we have a separate section on doing - this below ("Only software environment in the Docker image"). Once you - build the Docker image, your project's environment is setup and you can - go into it to run `./project make` manually. - - ```shell - FROM debian:stable-slim - RUN apt update && apt install -y gcc g++ wget - RUN useradd -ms /bin/sh maneager - RUN printf '123\n123' | passwd root - USER maneager - WORKDIR /home/maneager - RUN mkdir build - RUN mkdir software - COPY --chown=maneager:maneager ./project-source /home/maneager/source - COPY --chown=maneager:maneager ./software-dir /home/maneager/software - COPY --chown=maneager:maneager ./data-dir /home/maneager/data - RUN cd /home/maneager/source \ - && ./project configure --build-dir=/home/maneager/build \ - --software-dir=/home/maneager/software \ - --input-dir=/home/maneager/data - ``` - - 1. **Choose the base operating system:** The first step is to select the - operating system that will be used in the docker image. Note that your - choice of operating system also determines the commands of the next - step to install core software. - - ```shell - FROM debian:stable-slim - ``` - - 2. **Maneage dependencies:** By default the "slim" versions of the - operating systems don't contain a compiler (needed by Maneage to - compile precise versions of all the tools). You thus need to use the - selected operating system's package manager to import them (below is - the command for Debian). Optionally, if you don't have the project's - software tarballs, and want the project to download them automatically, - you also need a downloader. - - ```shell - # C and C++ compiler. - RUN apt update && apt install -y gcc g++ - - # Uncomment this if you don't have 'software-XXXX.tar.gz' (below). - #RUN apt install -y wget - ``` - - 3. **Define a user:** Some core software packages will complain if you try - to install them as the default (root) user. Generally, it is also good - practice to avoid being the root user. Hence with the commands below we - define a `maneager` user and activate it for the next steps. But just - in case root access is necessary temporarily, with the `passwd` - command, we are setting the root password to `123`. - - ```shell - RUN useradd -ms /bin/sh maneager - RUN printf '123\n123' | passwd root - USER maneager - WORKDIR /home/maneager - ``` - - 4. **Copy project files into the container:** these commands make the - assumptions listed below. IMPORTANT: you can also avoid copying over - all, and simply mount your host directories within the image, we have a - separate section on doing this below ("Only software environment in the - Docker image"). - - * The project's source is in the `maneaged/` sub-directory and this - directory is in the same directory as the `Dockerfile`. The source - can either be from cloned from Git (highly recommended!) or from a - tarball. Both are described above (note that arXiv's tarball needs to - be corrected as mentioned above). - - * (OPTIONAL) By default the project's necessary software source - tarballs will be downloaded when necessary during the `./project - configure` phase. But if you already have the sources, its better to - use them and not waste network traffic (and resulting carbon - footprint!). Maneaged projects usually come with a - `software-XXXX.tar.gz` file that is published on Zenodo (link above). - If you have this file, put it in the same directory as your - `Dockerfile` and include the relevant lines below. - - * (OPTIONAL) The project's input data. The `INPUT-FILES` depends on the - project, please look into the project's - `reproduce/analysis/config/INPUTS.conf` for the URLs and the file - names of input data. Similar to the software source files mentioned - above, if you don't have them, the project will attempt to download - its necessary data automatically in the `./project make` phase. - - ```shell - # Make the project's build directory and copy the project source - RUN mkdir build - COPY --chown=maneager:maneager ./maneaged /home/maneager/source - - # Optional (for software) - COPY --chown=maneager:maneager ./software-XXXX.tar.gz /home/maneager/ - RUN tar xf software-XXXX.tar.gz && mv software-XXXX software && rm software-XXXX.tar.gz - - # Optional (for data) - RUN mkdir data - COPY --chown=maneager:maneager ./INPUT-FILES /home/maneager/data - ``` - - 5. **Configure the project:** With this line, the Docker image will - configure the project (build all its necessary software). This will - usually take about an hour on an 8-core system. You can also optionally - avoid putting this step (and the next) in the `Dockerfile` and simply - execute them in the Docker image in interactive mode (as explained in - the sub-section below, in this case don't forget to preserve the build - container after you are done). - - ```shell - # Configure project (build full software environment). - RUN cd /home/maneager/source \ - && ./project configure --build-dir=/home/maneager/build \ - --software-dir=/home/maneager/software \ - --input-dir=/home/maneager/data - ``` - - 6. **Project's analysis:** With this line, the Docker image will do the - project's analysis and produce the final `paper.pdf`. The time it takes - for this step to finish, and the storage/memory requirements highly - depend on the particular project. - - ```shell - # Run the project's analysis - RUN cd /home/maneager/source && ./project make - ``` - - 7. **Build the Docker image:** The `Dockerfile` is now ready! In the - terminal, go to its directory and run the command below to build the - Docker image. We recommend to keep the `Dockerfile` in **an empty - directory** and run it from inside that directory too. This is because - Docker considers that directories contents to be part of the - environment. Finally, just set a `NAME` for your project and note that - Docker only runs as root. - - ```shell - sudo su - docker build -t NAME ./ - ``` - - - -#### Interactive tests on built container - -If you later want to start a container with the built image and enter it in -interactive mode (for example for temporary tests), please run the -following command. Just replace `NAME` with the same name you specified -when building the project. You can always exit the container with the -`exit` command (note that all your changes will be discarded once you exit, -see below if you want to preserve your changes after you exit). +### Building in containers -```shell -docker run -it NAME -``` - - - -#### Running your own project's shell for same analysis environment - -The default operating system only has minimal features: not having many of -the tools you are accustomed to in your daily command-line operations. But -your maneaged project has a very complete (for the project!) environment -which is fully built and ready to use interactively with the commands -below. For example the project also builds Git within itself, as well as -many other high-level tools that are used in your project and aren't -present in the container's operating system. - -```shell -# Once you are in the docker container -cd source -./project shell -``` - - - -#### Preserving the state of a built container - -All interactive changes in a container will be deleted as soon as you exit -it. THIS IS A VERY GOOD FEATURE IN GENERAL! If you want to make persistent -changes, you should do it in the project's plain-text source and commit -them into your project's online Git repository. As described in the Docker -introduction above, we strongly recommend to **not rely on a built container -for archival purposes**. - -But for temporary tests it is sometimes good to preserve the state of an -interactive container. To do this, you need to `commit` the container (and -thus save it as a Docker "image"). To do this, while the container is still -running, open another terminal and run these commands: - -```shell -# These two commands should be done in another terminal -docker container list - -# Get 'XXXXXXX' of your desired container from the first column above. -# Give the new image a name by replacing 'NEW-IMAGE-NAME'. -docker commit XXXXXXX NEW-IMAGE-NAME -``` - - - -#### Copying files from the Docker image to host operating system - -The Docker environment's file system is completely indepenent of your host -operating system. One easy way to copy files to and from an open container -is to use the `docker cp` command (very similar to the shell's `cp` -command). - -```shell -docker cp CONTAINER:/file/path/within/container /host/path/target -``` - - - - - -#### Only software environment in the Docker image - -You can set the docker image to only contain the software environment and -keep the project source and built analysis files (data and PDF) on your -host operating system. This enables you to keep the size of the Docker -image to a minimum (only containing the built software environment) to -easily move it from one computer to another. Below we'll summarize the -steps. - - 1. Get your user ID with this command: `id -u`. - - 2. Make a new (empty) directory called `docker` temporarily (will be - deleted later). - - ```shell - mkdir docker-tmp - cd docker-tmp - ``` - - 3. Make a `Dockerfile` (within the new/empty directory) with the - following contents. Just replace `UID` with your user ID (found in - step 1 above). Note that we are manually setting the `maneager` (user) - password to `123` and the root password to '456' (both should be - repeated because they must be confirmed by `passwd`). To install other - operating systems, just change the contents on the `FROM` line. For - example, for CentOS 7 you can use `FROM centos:centos7`, for the - latest CentOS, you can use `FROM centos:latest` (you may need to add - this line `RUN yum install -y passwd` before the `RUN useradd ...` - line.). - - ``` - FROM debian:stable-slim - RUN useradd -ms /bin/sh --uid UID maneager; \ - printf '123\n123' | passwd maneager; \ - printf '456\n456' | passwd root - USER maneager - WORKDIR /home/maneager - RUN mkdir build; mkdir build/analysis - ``` - - 4. Create a Docker image based on the `Dockerfile` above. Just replace - `MANEAGEBASE` with your desired name (this won't be your final image, - so you can safely use a name like `maneage-base`). Note that you need - to have root/administrator previlages when running it, so - - ```shell - sudo docker build -t MANEAGEBASE ./ - ``` - - 5. You don't need the temporary directory any more (the docker image is - saved in Docker's own location, and accessible from anywhere). - - ```shell - cd .. - rm -rf docker-tmp - ``` - - 6. Put the following contents into a newly created plain-text file called - `docker-run`, while setting the mandatory variables based on your - system. The name `docker-run` is already inside Maneage's `.gitignore` - file, so you don't have to worry about mistakenly commiting this file - (which contains private information: directories in this computer). +Containers are a common way to build projects in an independent filesystem +and an almost independent operating system without the overhead (in size +and speed) of a virtual machine. As a result, containers allow easy +movement of built projects from one system to another without +rebuilding. However, they are still large binary files (+1 Gigabytes) and +may not be usable in the future (for example with new software versions not +reading old images or old/new kernel issues). Containers are thus good for +execution/testing phases of a project, but shouldn't be what you archive +for the long term! - ``` - #!/bin/sh - # - # Create a Docker container from an existing image of the built - # software environment, but with the source, data and build (analysis) - # directories directly within the host file system. This script should - # be run in the top project source directory (that has 'README.md' and - # 'paper.tex'). If not, replace the '$(pwd)' part with the project - # source directory. - - # MANDATORY: Name of Docker container - docker_name=MANEAGEBASE - - # MANDATORY: Location of "build" directory on this system (to host the - # 'analysis' sub-directory for output data products and possibly others). - build_dir=/PATH/TO/THIS/PROJECT/S/BUILD/DIR - - # OPTIONAL: Location of project's input data in this system. If not - # present, a 'data' directory under the build directory will be created. - data_dir=/PATH/TO/THIS/PROJECT/S/DATA/DIR - - # OPTIONAL: Location of software tarballs to use in building Maneage's - # internal software environment. - software_dir=/PATH/TO/SOFTWARE/TARBALL/DIR - - - - - - # Internal proceessing - # -------------------- - # - # Sanity check: Make sure that the build directory actually exists. - if ! [ -d $build_dir ]; then - echo "ERROR: '$build_dir' doesn't exist"; exit 1; - fi - - # If the host operating system has '/dev/shm', then give Docker access - # to it also for improved speed in some scenarios (like configuration). - if [ -d /dev/shm ]; then shmopt="-v /dev/shm:/dev/shm"; - else shmopt=""; fi - - # If the 'analysis' and 'data' directories (that are mounted), don't exist, - # then create them (otherwise Docker will create them as 'root' before - # creating the container, and we won't have permission to write in them. - analysis_dir="$build_dir"/analysis - if ! [ -d $analysis_dir ]; then mkdir $analysis_dir; fi - - # If the data or software directories don't exist, put them in the build - # directory (they will remain empty, but this helps in simplifiying the - # mounting command!). - if ! [ x$data_dir = x ]; then - data_dir="$build_dir"/data - if ! [ -d $data_dir ]; then mkdir $data_dir; fi - fi - if ! [ x$software_dir = x ]; then - software_dir="$build_dir"/tarballs-software - if ! [ -d $software_dir ]; then mkdir $software_dir; fi - fi - - # Run the Docker image while setting up the directories. - sudo docker run -v "$software_dir":/home/maneager/tarballs-software \ - -v "$analysis_dir":/home/maneager/build/analysis \ - -v "$data_dir":/home/maneager/data \ - -v "$(pwd)":/home/maneager/source \ - $shmopt -it $docker_name - ``` - - 7. Make the `docker-run` script executable. - - ```shell - chmod +x docker-run - ``` - - 8. Start the Docker daemon (root permissions required). If the operating - system uses systemd you can use the command below. If you want the - Docker daemon to be available after a reboot also (so you don't have - to restart it after turning off your computer), run this command again - but replacing `start` with `enable`. +It is therefore very important that if you want to save and move your +maneaged project within containers, be sure to commit all your project's +source files and push them to your external Git repository (you can do +these within the container as explained below). This way, you can always +recreate the container with future technologies too. Generally, if you are +developing within a container, its good practice to recreate it from +scratch every once in a while, to make sure you haven't forgot to include +parts of your work in your project's version-controlled source. In the +sections below we also describe how you can use the container **only for +the software environment** and keep your data and project source on your +host. - ```shell - systemctl start docker - ``` - - 9. You can now start the Docker image by executing your newly added - script like below (it will ask for your root password). You will - notice that you are in the Docker container with the changed prompt. - - ```shell - ./docker-run - ``` - - 10. You are now within the container. First, we'll add the GNU C and C++ - compilers (which are necessary to build our own programs in Maneage) - and the GNU WGet downloader (which may be necessary if you don't have - a core software's tarball already). Maneage will build pre-defined - versions of both and will use them. But for the very first packages, - they are necessary. In the process, by setting the `PS1` environment - variable, we'll define a color-coding for the interactive shell prompt - (red for root and purple for the user). If you build another operating - system, replace the `apt` commands accordingly (for example on CentOS, - you don't need the `apt update` line and you should use `yum install - -y gcc gcc-c++ wget glibc-static` to install the three basic - dependencies). - - ```shell - su - echo 'export PS1="[\[\033[01;31m\]\u@\h \W\[\033[32m\]\[\033[00m\]]# "' >> ~/.bashrc - source ~/.bashrc - apt update - apt install -y gcc g++ wget - exit - echo 'export PS1="[\[\033[01;35m\]\u@\h \W\[\033[32m\]\[\033[00m\]]$ "' >> ~/.bashrc - source ~/.bashrc - ``` - - 11. Now that the compiler is ready, we can start Maneage's - configuration. So let's go into the project source directory and run - these commands to build the software environment. - - ```shell - cd source - ./project configure --input-dir=/home/maneager/data \ - --build-dir=/home/maneager/build \ - --software-dir=/home/maneager/tarballs-software - ``` - - 12. After the configuration finishes successfully, it will say so. It will - then ask you to run `./project make`. **But don't do that - yet**. Keep this Docker container open and don't exit the container or - terminal. Open a new terminal, and follow the steps described in the - sub-section above to preserve (or "commit") the built container as a - Docker image. Let's assume you call it `MY-PROJECT-ENV`. After the new - image is made, you should be able to see the new image in the list of - images with this command (in yet another terminal): - - ```shell - docker image list # In the other terminal. - ``` +If you have the necessary software tarballs and input data (optional +features described below) you can disable internet. In this situation, the +configuration and analysis will be exactly reproduced, the final LaTeX +macros will be created, and all results will be verified +successfully. However, no final `paper.pdf` will be created to +visualize/combine everything in one easy-to-read file. Until [task +15267](https://savannah.nongnu.org/task/?15267) is complete, Maneage only +needs internet to install TeXLive packages (using TeXLive's own package +manager `tlmgr`) in the `./project configure` phase. This won't stop the +configuration (since all the analysis can still be reproduced). We are +working on completing this task as soon as possible, but until then, if you +want to disable internet *and* you want to build the final PDF, please +disable internet after the configuration phase. Note that only the +necessary TeXLive packages are installed (~350 MB), not the full TeXLive +collection! - 13. Now that you have safely "committed" your current Docker container - into a separate Docker image, you can **exit the container** safely - with the `exit` command. Don't worry, you won't loose the built - software environment: it is all now saved separately within the Docker - image. +The container technologies that Maneage has a high-level interface for +(with the `reproduce/software/shell` directory) are listed below. Each has +a dedicated shell script in that directory with an (almost) identical +interface. See the respective `*-README.md` file in that directory for more +details, as well as running your desired script with `--help` or reading +its comments at the top of the file. - 14. Re-open your `docker-run` script and change `MANEAGEBASE` to - `MY-PROJECT-ENV` (or any other name you set for the environment you - committed above). - - ```shell - emacs docker-run - ``` - - 15. That is it! You can now always easily enter your container (only for - the software environemnt) with the command below. Within the - container, any file you save/edit in the `source` directory of the - docker container is the same file on your host OS and any file you - build in your `build/analysis` directory (within the Maneage'd - project) will be on your host OS. You can even use your container's - Git to store the history of your project in your host OS. See the next - step in case you want to move your built software environment to - another computer. - - ```shell - ./docker-run - ``` - - 16. In case you want to store the image as a single file as backup or to - move to another computer, you can run the commands below. They will - produce a single `project-env.tar.gz` file. - - ```shell - docker save -o my-project-env.tar MY-PROJECT-ENV - gzip --best project-env.tar - ``` - - 17. To load the tarball above into a clean docker environment (for example - on another system) copy the `my-project-env.tar.gz` file there and run - the command below. You can then create the `docker-run` script for - that system and run it to enter. Just don't forget that if your - `analysis_dir` directory is empty on the new/clean system. So you - should first run the same `./project configure ...` command above in - the docker image so it connects the environment to your source. Don't - worry, it won't build any software and should finish in a second or - two. Afterwards, you can safely run `./project make` and continue - working like you did on the old system. - - ```shell - docker load --input my-project-env.tar.gz - ``` - - - - - -#### Deleting all Docker images - -After doing your tests/work, you may no longer need the multi-gigabyte -files images, so its best to just delete them. To do this, just run the two -commands below to first stop all running containers and then to delete all -the images: - -```shell -docker ps -a -q | xargs docker rm -docker images -a -q | xargs docker rmi -f -``` + - [Apptainer](https://apptainer.org): useful in high performance + computing (HPC) facilities (where you do not have root + permissions). Apptainer is fully free and open source software. + Apptainer containers can only be created and used on GNU/Linux + operating systems, but are stored as a single file (very easy to + manage). + - [Docker](https://www.docker.com): requires root access, but useful on + virtual private servers (VPSs). Docker images are stored and managed by + a root-level daemon, so you can only manage them through its own + interface (making containers by all users visible and accessible to all + other users of a system by default). A docker container build on a + GNU/Linux host can also be executed on Windows or macOS. However, while + the Docker engine and its command-line interface on GNU/Linux are free + and open source software, its desktop application (with a GUI and + components necessary for Windows or macOS) is not (requires payment for + large companies). -### Copyright information -This file and `.file-metadata` (a binary file, used by Metastore to store -file dates when doing Git checkouts) are part of the reproducible project -mentioned above and share the same copyright notice (at the start of this -file) and license notice (below). +## Copyright information -This project is free software: you can redistribute it and/or modify it -under the terms of the GNU General Public License as published by the Free +This file is free software: you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. -This project is distributed in the hope that it will be useful, but WITHOUT +This file is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along -with this project. If not, see <https://www.gnu.org/licenses/>. +with this file. If not, see <https://www.gnu.org/licenses/>. @@ -1,4 +1,4 @@ -%% Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% See the end of the file for license conditions. \documentclass[10pt, twocolumn]{article} @@ -3,8 +3,8 @@ # High-level script to manage the project. # Run './project --help' for a description of how to use it. # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2021-2025 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2021-2026 Raul Infante-Sainz <infantesainz@gmail.com> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -33,8 +33,10 @@ set -e jobs=0 # 0 is for the default for the 'configure.sh' script. group= debug= -host_cc=0 +quiet=0 +timing=0 offline= +host_cc=0 operation= build_dir= input_dir= @@ -42,6 +44,7 @@ keep_going= check_config= make_targets= software_dir= +pauseformsg=1 clean_texdir=0 prepare_redo=0 highlightnew=0 @@ -89,7 +92,7 @@ RECOMMENDATION: If this is the first time you are configuring this template, please don't use the options and let the script explain each parameter in full detail by simply running './project configure'. -Project 'make' special features. +Project 'make' special tagets ./project make Build the project on one thread ./project make -jN Built the project in parallel on N threads. ./project make clean Clean all files generated by 'make' (not software). @@ -106,27 +109,30 @@ Project 'make' special features. With the options below you can modify the default behavior. Configure options: + --all-highlevel Build all high-level software (for development). -b, --build-dir=STR Top directory to build the project in. - -e, --existing-conf Use (possibly existing) local configuration. - --host-cc Use host system's C compiler, don't build GCC. - -i, --input-dir=STR Directory containing input datasets (optional). - -s, --software-dir=STR Directory containing necessary software tarballs. --check-config During configuration, show what is being built. --clean-texdir Remove possibly existing build-time subdirectories under the project's 'tex/' directory (can happen when source is from arXiv for example). - --all-highlevel Build all high-level software (for development). + -e, --existing-conf Use (possibly existing) local configuration. + -i, --input-dir=STR Directory containing input datasets (optional). + --host-cc Use host system's C compiler, don't build GCC. + --quiet Do not print basic info messages (with '-e'). + --no-pause Do not sleep/pause after basic info messages. + -s, --software-dir=STR Directory containing necessary software tarballs. Configure and Make options: - -d, --debug[=FLAGS] In configure: use -j1, no -k, and no Zenodo check. + -d, --debug[=FLAGS] In configure: use -j1 and stop on first crash. In make: 'FLAGS' will be directly passed to 'make'. - -k, --keep-going Keep going when some targets can't be made. -g, --group=STR Build and run with write permissions for a group. -j, --jobs=INT Number of threads to build/run the software. -?, --help Print this help list. Make (analysis) options: + -k, --keep-going Keep going when some targets can't be made. -p, --prepare-redo Re-do preparation (only done automatically once). + -t, --timing Starting and ending times written in 'timing.txt'. Make (final PDF) options: --refresh-bib Force refresh the bibliography. @@ -178,65 +184,71 @@ do shell) func_operation_set $1; shift;; # Configure options: - -b|--build-dir) build_dir="$2"; check_v "$1" "$build_dir"; shift;shift;; - -b=*|--build-dir=*) build_dir="${1#*=}"; check_v "$1" "$build_dir"; shift;; - -b*) build_dir=$(echo "$1" | sed -e's/-b//'); check_v "$1" "$build_dir"; shift;; - -e|--existing-conf) existing_conf=1; shift;; + -e|--existing-conf) existing_conf=1; shift;; -e*|--existing-conf=*) on_off_option_error --existing-conf -e;; - --host-cc) host_cc=1; shift;; + --host-cc) host_cc=1; shift;; --host-cc=*) on_off_option_error --host-cc;; - --offline) offline=1; shift;; + --offline) offline=1; shift;; --offline=*) on_off_option_error --offline;; - -i|--input-dir) input_dir="$2"; check_v "$1" "$input_dir"; shift;shift;; - -i=*|--input-dir=*) input_dir="${1#*=}"; check_v "$1" "$input_dir"; shift;; - -i*) input_dir=$(echo "$1" | sed -e's/-i//'); check_v "$1" "$input_dir"; shift;; - -s|--software-dir) software_dir="$2"; check_v "$1" "$software_dir"; shift;shift;; - -s=*|--software-dir=*) software_dir="${1#*=}"; check_v "$1" "$software_dir"; shift;; - -s*) software_dir=$(echo "$1" | sed -e's/-s//'); check_v "$1" "$software_dir"; shift;; - --check-config) check_config=1; shift;; + -i|--input-dir) input_dir="$2"; check_v "$1" "$input_dir"; shift;shift;; + -i=*|--input-dir=*) input_dir="${1#*=}"; check_v "$1" "$input_dir"; shift;; + -i*) input_dir=$(echo "$1" | sed -e's/-i//'); check_v "$1" "$input_dir"; shift;; + -s|--software-dir) software_dir="$2"; check_v "$1" "$software_dir"; shift;shift;; + -s=*|--software-dir=*) software_dir="${1#*=}"; check_v "$1" "$software_dir"; shift;; + -s*) software_dir=$(echo "$1" | sed -e's/-s//'); check_v "$1" "$software_dir"; shift;; + --check-config) check_config=1; shift;; --check-config=*) on_off_option_error --check-config;; - --clean-texdir) clean_texdir=1; shift;; + --clean-texdir) clean_texdir=1; shift;; --clean-texdir=*) on_off_option_error --clean-texdir;; - --all-highlevel) all_highlevel=1; shift;; + --all-highlevel) all_highlevel=1; shift;; --all-highlevel=*) on_off_option_error --all-highlevel;; + --no-pause) pauseformsg=0; shift;; + --no-pause=*) on_off_option_error --no-pause;; + --quiet) quiet=1; shift;; + --quiet=*) on_off_option_error --quiet;; # Configure and Make options: - -g|--group) group="$2"; check_v group "$group"; shift;shift;; - -g=*|--group=*) group="${1#*=}"; check_v group "$group"; shift;; - -g*) group=$(echo "$1" | sed -e's/-g//'); check_v group "$group"; shift;; - -j|--jobs) jobs="$2"; check_v jobs "$jobs"; shift;shift;; - -j=*|--jobs=*) jobs="${1#*=}"; check_v jobs "$jobs"; shift;; - -j*) jobs=$(echo "$1" | sed -e's/-j//'); check_v jobs "$jobs"; shift;; - -k|--keep-going) keep_going="--keep-going"; shift;; - -k=*|--keep-going=*) on_off_option_error --keep-going -k;; - -k*) on_off_option_error --keep-going -k;; - -'?'|--help) print_help; exit 0;; - -'?'*|--help=*) on_off_option_error --help -?;; + -b|--build-dir) build_dir="$2"; check_v "$1" "$build_dir";shift;shift;; + -b=*|--build-dir=*) build_dir="${1#*=}"; check_v "$1" "$build_dir";shift;; + -b*) build_dir=$(echo "$1" | sed -e's/-b//'); check_v "$1" "$build_dir";shift;; + -g|--group) group="$2"; check_v group "$group"; shift;shift;; + -g=*|--group=*) group="${1#*=}"; check_v group "$group"; shift;; + -g*) group=$(echo "$1" | sed -e's/-g//'); check_v group "$group"; shift;; + -j|--jobs) jobs="$2"; check_v jobs "$jobs"; shift;shift;; + -j=*|--jobs=*) jobs="${1#*=}"; check_v jobs "$jobs"; shift;; + -j*) jobs=$(echo "$1" | sed -e's/-j//'); check_v jobs "$jobs"; shift;; + -k|--keep-going) keep_going="--keep-going"; shift;; + -k=*|--keep-going=*) on_off_option_error --keep-going -k;; + -k*) on_off_option_error --keep-going -k;; + -'?'|--help) print_help; exit 0;; + -'?'*|--help=*) on_off_option_error --help -?;; # Make options (analysis): - -p|--prepare-redo) prepare_redo=1; shift;; - -p=*|--prepare-redo=*) on_off_option_error --prepare-redo; shift;; + -p|--prepare-redo) prepare_redo=1; shift;; + -p=*|--prepare-redo=*) on_off_option_error --prepare-redo; shift;; + -t|--timing) timing=1; shift;; + -t=*|--timing=*) on_off_option_error --timing; shift;; # Make options (final PDF): - --refresh-bib) [ -f tex/src/references.tex ] && touch tex/src/references.tex; shift;; - --highlight-all) highlightnew=1; highlightnotes=1; shift;; - --highlight-all=*) on_off_option_error --highlight-new;; - --highlight-new) highlightnew=1; shift;; - --highlight-new=*) on_off_option_error --highlight-new;; - --highlight-notes) highlightnotes=1; shift;; - --highlight-notes=*) on_off_option_error --highlight-notes;; - -d|--debug) if [ x$operation = x ]; then - echo "Please set the operation before calling '--debug'"; exit 1 - elif [ x$operation = xconfigure ]; then debug=a; shift; - elif [ x$operation = xmake ]; then - if [ x"$2" = x ]; then - echo "In make-mode, '--debug' needs a value; see GNU Make manual"; exit 1 - else debug="$2"; check_v debug "$debug"; shift;shift; fi - else - echo "Operation '$operation' not recognized, please use 'configure' or 'make'" - fi;; - -d=*|--debug=*) debug="${1#*=}"; check_v debug "$debug"; shift;; - -d*) debug=$(echo "$1" | sed -e's/-d//'); check_v debug "$debug"; shift;; + --refresh-bib) [ -f tex/src/references.tex ] && touch tex/src/references.tex; shift;; + --highlight-all) highlightnew=1; highlightnotes=1; shift;; + --highlight-all=*) on_off_option_error --highlight-new;; + --highlight-new) highlightnew=1; shift;; + --highlight-new=*) on_off_option_error --highlight-new;; + --highlight-notes) highlightnotes=1; shift;; + --highlight-notes=*) on_off_option_error --highlight-notes;; + -d|--debug) if [ x$operation = x ]; then + echo "Please set the operation before calling '--debug'"; exit 1 + elif [ x$operation = xconfigure ]; then debug=a; shift; + elif [ x$operation = xmake ]; then + if [ x"$2" = x ]; then + echo "In make-mode, '--debug' needs a value; see GNU Make manual"; exit 1 + else debug="$2"; check_v debug "$debug"; shift;shift; fi + else + echo "Operation '$operation' not recognized, please use 'configure' or 'make'" + fi;; + -d=*|--debug=*) debug="${1#*=}"; check_v debug "$debug"; shift;; + -d*) debug=$(echo "$1" | sed -e's/-d//'); check_v debug "$debug"; shift;; # Unrecognized option: -*) echo "$scriptname: unknown option '$1'"; exit 1;; @@ -290,8 +302,8 @@ EOF ls $coloropt .build/software/build-tmp || junk=1; fi - # Make the temporary directory, delete its contents, then put new - # links of all built software. + # Make the temporary directory, delete its contents, then put + # new links of all built software. if ! [ -d $checkdir ]; then mkdir $checkdir; fi rm -f $checkdir/* @@ -312,10 +324,11 @@ EOF if [ $printresults = 1 ]; then echo "--- Last 5 packages that were built:" - # Then sort all the links based on the most recent dates of the - # files they link to (with '-L'). + # Then sort all the links based on the most recent dates of + # the files they link to (with '-L'). ls -Llt $checkdir \ - | awk '/^-/ && c++<5 {printf "[at %s] %s\n", $(NF-1), $NF}' + | awk '/^-/ && c++<5 {printf "[at %s] %s\n", \ + $(NF-1), $NF}' fi else cat <<EOF @@ -339,14 +352,15 @@ fi -# Basic group settings -# -------------------- +# Group check +# ----------- if ! [ x$group = x ]; then # Check if group is usable. if ! sg "$group" "echo Group \'$group\' exists"; then - echo "$scriptname: '$group' is not a usable group name on this system."; - echo "(TIP: you can use the 'groups' command to see your groups)" + printf "$scriptname: '$group' is not a usable group name on " + printf "this system. TIP: you can use the 'groups' command " + printf "to see your groups)\n" exit 1 fi @@ -358,9 +372,45 @@ fi -# Error when configuration isn't run -configuration_necessary() { - cat <<EOF +# Build directory symbolic links +# ------------------------------ +# +# The source directory will contain two symbolic links that point to the +# build directory: + +# - .build: the top build directory. +# +# - .local: the second to the top software installed directory. They are +# used during the configuration phase to simplify commands and are also +# very useful during the development of a maneage'd project (to easily +# get to the build directory or execut Maneage'd software). +# +# This needs to be done on every run because: +# - './project configure' can be run with a new build directory, and +# keeping the old '.build' conflicts with the new build directory +# that the user gave. +# - './project make' or './project shell' (within a newly cloned source +# directory from inside a container): the links do not exist but have +# to be set to the container's build directory. +# - This is not an expensive operation. +if ! [ x"$build_dir" = x ]; then + rm -f .build .local + ln -s $build_dir .build + ln -s $build_dir/software/installed .local +fi + + + + + +# Function to validate configuration +# ---------------------------------- +# +# Check if the configuration is missing/incomplete +configuration_check() { + confdone=software/config/hardware-parameters.tex + if ! [ -f .build/$confdone ]; then + cat <<EOF The project is either (1) not configured on this system, or (2) the configuration wasn't successful. @@ -380,15 +430,70 @@ If there was a problem, please let us know by filling this online form: http://savannah.nongnu.org/support/?func=additem&group=reproduce EOF - exit 1 + exit 1 + fi +} + + + + + +# Function for TeX Preparations +# ----------------------------- +# +# Make sure that the necessary analysis directories directory exist in the +# build directory. These will be necessary in various phases of hte +# analysis and having them inside the lower-level Make steps will require +# setting them as prerequisites for many basic jobs (thus making the +# Makefiles harder to read and add potentials for bugs: forgetting to add +# them for example). Also, we don't want the configure phase to make any +# edits in the analysis directory, so they are not built there. +tex_preparations () { + + # Extract the location of the build directory. + bdir=$(.local/bin/realpath .build) + + # We are using our custom-built 'mkdir' which is guaranteed to have the + # '-p' option (that will also build intermediate directories) and won't + # complain if the directory already exists. + badir=$bdir/analysis + texdir=$badir/tex + btexdir=$texdir/build + tikzdir=$btexdir/tikz + mtexdir=$texdir/macros + .local/bin/mkdir -p $mtexdir $btexdir $tikzdir + + # If 'tex/build' and 'tex/tikz' are symbolic links then 'rm -f' + # will delete them and we can continue. However, when the project + # is being built from the tarball (from arXiv for example), these + # two are not symbolic links but actual directories with the + # necessary built-components to build the PDF in them. In this + # case, because 'tex/build' is a directory, 'rm -f' will fail, so + # we'll just rename the two directories (as backup) and let the + # project build the proper symbolic links afterwards. + if rm -f tex/build; then + rm -f tex/tikz + else + mv tex/tikz tex/tikz-from-tarball + mv tex/build tex/build-from-tarball + fi + + # Build the symbolic links. + if ! [ -L tex/tikz ]; then ln -s "$tikzdir" tex/tikz; fi + if ! [ -L tex/build ]; then ln -s "$texdir" tex/build; fi } -# Run operations in controlled environment -# ---------------------------------------- +# Function to run in controlled environment +# ----------------------------------------- +# +# Controlling the environment is necessary for running the analysis. Like +# the other funcitons here, this is defined to simplify the high-level code +# within the 'make)' switch statement. +perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x" controlled_env() { # Get the full address of the build directory: @@ -423,7 +528,6 @@ controlled_env() { # Do requested operation # ---------------------- -perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x" configscript=./reproduce/software/shell/configure.sh case $operation in @@ -444,8 +548,11 @@ case $operation in # to make sure they have them, we are activating the executable # flags by default here every time './project configure' is run. If # any other file in your project needs such flags, add them here. - chmod +x reproduce/software/shell/* reproduce/software/config/*.sh \ - reproduce/analysis/bash/* + if ! [ -x reproduce/software/shell/configure.sh ]; then + chmod +x reproduce/analysis/bash/* \ + reproduce/software/shell/* \ + reproduce/software/config/*.sh + fi # If the user requested, clean the TeX directory from the extra # (to-be-built) directories that may already be there (and will not @@ -457,11 +564,13 @@ case $operation in # Variables to pass to the configuration script. export jobs=$jobs export debug=$debug + export quiet=$quiet export host_cc=$host_cc export offline=$offline export build_dir=$build_dir export input_dir=$input_dir export scriptname=$scriptname + export pauseformsg=$pauseformsg export maneage_group_name=$group export software_dir=$software_dir export existing_conf=$existing_conf @@ -481,7 +590,8 @@ case $operation in # creates problems when another group member wants to update # the software for example. We thus need to manually add the # group writing flag to all installed software files. - echo "Enabling group writing permission on all installed software..." + printf "Enabling group writing permission on all installed " + printf "software...\n" .local/bin/chmod -R g+w .local/; fi ;; @@ -493,63 +603,75 @@ case $operation in # Batch execution of the project. make) - # Make sure the configure script has been completed properly - # ('configuration-done.txt' exists). - if ! [ -f .build/software/configuration-done.txt ]; then - configuration_necessary - fi + # Make sure the configure script is complete and necessary LaTeX + # directories are in place. + configuration_check + tex_preparations + + # Register the start of this run if requested (we are appending the + # new information so previous information is preserved until the + # user intentionally deletes/cleans it). + if [ $timing = 1 ]; then echo "start: $(date)" >> timing.txt; fi # Run data preparation phase (optionally build Makefiles with # special values for optimizing the main 'top-make.mk'). But note # that data preparation is only done automatically the first time - # the project is built (when '.build/software/preparation-done.mk' + # the project is built (when '.build/analysis/preparation-done.mk' # doesn't yet exist). After that, if the user wants to re-do the # preparation they have to use the '--prepare-redo' option. - if ! [ -f .build/software/preparation-done.mk ] \ + if ! [ -f .build/analysis/preparation-done.mk ] \ || [ x"$prepare_redo" = x1 ]; then controlled_env reproduce/analysis/make/top-prepare.mk fi - # Run the actual project. + # Call top-make (highest level analysis Makefile). controlled_env reproduce/analysis/make/top-make.mk + + # Register the time of the project's ending. + if [ $timing = 1 ]; then echo "end: $(date)" >> timing.txt; fi ;; + + + + # Interactive shell of Maneage. shell) - # Make sure the configure script has been completed properly - # ('configuration-done.txt' exists). - if ! [ -f .build/software/configuration-done.txt ]; then - configuration_necessary - fi + # Make sure the configure script has been completed properly. + configuration_check # Run the project's own shell without inheriting any environment # from the host. The 'TERM' environment variable is necessary for # tools like some text editors. - bdir=`.local/bin/realpath .build` + bdir=$(.local/bin/realpath .build) instdir="$bdir"/software/installed bindir="$bdir"/software/installed/bin rcfile=$(pwd)/reproduce/software/shell/bashrc.sh .local/bin/env -i \ - HOME="$bdir" \ - TERM="$TERM" \ - PATH="$bindir" \ - CCACHE_DISABLE=1 \ - PROJECT_STATUS=shell \ - SHELL="$bindir"/bash \ - COLORTERM="$COLORTERM" \ - PROJECT_RCFILE="$rcfile" \ - LDFLAGS=-L"$instdir"/lib \ - CPPFLAGS=-I"$instdir"/include \ - LD_LIBRARY_PATH="$instdir"/lib \ - OMPI_MCA_plm_rsh_agent=/bin/false \ - PYTHONPATH="$instdir"/lib/python/site-packages \ - PYTHONPATH3="$instdir"/lib/python/site-packages \ - PS1="[\[\033[01;35m\]maneage@\h \W\[\033[32m\]\[\033[00m\]]$ " \ - "$bindir"/bash --noprofile --rcfile "$rcfile" + LC_ALL=C \ + HOME="$bdir" \ + TERM="$TERM" \ + PATH="$bindir" \ + CCACHE_DISABLE=1 \ + PROJECT_STATUS=shell \ + SHELL="$bindir"/bash \ + COLORTERM="$COLORTERM" \ + PROJECT_RCFILE="$rcfile" \ + LDFLAGS=-L"$instdir"/lib \ + CPPFLAGS=-I"$instdir"/include \ + LD_LIBRARY_PATH="$instdir"/lib \ + OMPI_MCA_plm_rsh_agent=/bin/false \ + PYTHONPATH="$instdir"/lib/python/site-packages \ + PYTHONPATH3="$instdir"/lib/python/site-packages \ + PS1="[\[\033[01;35m\]maneage@\h \W\[\033[32m\]\[\033[00m\]]$ " \ + "$bindir"/bash --noprofile --rcfile "$rcfile" ;; + + + # Operation not specified. *) cat <<EOF @@ -557,7 +679,7 @@ case $operation in No operation defined! Please run with '--help' for more information. -(TIP: available operations are: 'configure', 'make', or 'shell'). +(TIP: available modes are: 'configure', 'make', or 'shell'). EOF exit 1 diff --git a/reproduce/analysis/bash/download-multi-try.sh b/reproduce/analysis/bash/download-multi-try.sh index bea88d5..2df8aab 100755 --- a/reproduce/analysis/bash/download-multi-try.sh +++ b/reproduce/analysis/bash/download-multi-try.sh @@ -31,7 +31,7 @@ # reason, you don't want to use a lock file, set the 'lockfile' name to # 'nolock'. # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -90,6 +90,61 @@ urlfile=$(echo "$inurl" | awk -F "/" '{print $NF}') +# Function for downloading +download_func () { + + # Set the arguments. + inurl="$1" + + # During the installation of basic software, the dependencies of the + # downloaders that are installed in Maneage can conflict with the + # downloader that is not yet installed in Maneage. To avoid this, we + # check if the downloader works (with '--version') and remove the + # Maneage library temporarily in case it does not. + lpathorig="" + dprog=$(echo $downloader | awk '{print $1}') + if ! $dprog --version > /dev/null 2> /dev/null; then + echo "NOTE: the segmentation fault is expected, is not a problem" + basedir=$(echo $outname \ + | sed -e's|\/tarballs\/| |' \ + | awk '{print $1}') + libdir=$basedir/installed/lib + lpathorig=$LD_LIBRARY_PATH + lpath=$(echo $LD_LIBRARY_PATH \ + | sed -e's|'$libdir':||') + export LD_LIBRARY_PATH=$lpath + fi + + # Attempt downloading the file. Note that the 'downloader' ends with + # the respective option to specify the output name. For example "wget + # -O" (so 'outname', that comes after it) will be the name of the + # downloaded file. + if [ x"$lockfile" = xnolock ]; then + if ! $downloader $outname $inurl; then rm -f $outname; fi + else + flock "$lockfile" sh -c \ + "if ! $downloader $outname \"$inurl\"; then rm -f $outname; fi" + fi + + # In case the LD_LIBRARY_PATH was changed, set it back to normal. + if ! [ x"$lpathorig" = x ]; then + export LD_LIBRARY_PATH=$lpathorig + fi + + # Some servers return HTTP 4xx/5xx errors as an HTML page with a 200 + # status, so the downloader exits 0 and saves the HTML body to disk. + # Detect this by checking whether the response starts with an HTML tag + # and treat it as a failed download so backup servers will be tried. + if [ -f "$outname" ] \ + && head -c 500 "$outname" 2>/dev/null | grep -qi '<html'; then + rm -f "$outname" + fi +} + + + + + # Try downloading multiple times before crashing. counter=0 maxcounter=10 @@ -119,30 +174,22 @@ while [ ! -f "$outname" ]; do sleep $tstep fi - # Attempt downloading the file. Note that the 'downloader' ends with - # the respective option to specify the output name. For example "wget - # -O" (so 'outname', that comes after it) will be the name of the - # downloaded file. - if [ x"$lockfile" = xnolock ]; then - if ! $downloader $outname $inurl; then rm -f $outname; fi - else - # Try downloading from the requested URL. - flock "$lockfile" sh -c \ - "if ! $downloader $outname \"$inurl\"; then rm -f $outname; fi" - fi + # First attempt at the download. + download_func "$inurl" # If the download failed, try the backup server(s). if [ ! -f "$outname" ]; then if [ x"$backupservers" != x ]; then for bs in $backupservers; do - # Use this backup server. - if [ x"$lockfile" = xnolock ]; then - if ! $downloader $outname $bs/$urlfile; then rm -f $outname; fi - else - flock "$lockfile" sh -c \ - "if ! $downloader $outname $bs/$urlfile; then rm -f $outname; fi" - fi + # For Zenodo backup servers, append '?download=1' as well. + bsurl="$bs/$urlfile" + case "$bsurl" in + *zenodo.org*) bsurl="${bsurl}?download=1" ;; + esac + + # Download the file. + download_func "$bsurl" # If the file was downloaded, break out of the loop that # parses over the backup servers. diff --git a/reproduce/analysis/config/INPUTS.conf b/reproduce/analysis/config/INPUTS.conf index 5860806..7373503 100644 --- a/reproduce/analysis/config/INPUTS.conf +++ b/reproduce/analysis/config/INPUTS.conf @@ -105,7 +105,7 @@ # also called '%' (if your local copy of the input dataset and the only # repository names are the same, be sure to set '%' accordingly). # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/analysis/config/delete-me-squared-num.conf b/reproduce/analysis/config/delete-me-squared-num.conf index ba8c960..80eb60a 100644 --- a/reproduce/analysis/config/delete-me-squared-num.conf +++ b/reproduce/analysis/config/delete-me-squared-num.conf @@ -1,6 +1,6 @@ # Number of samples in the demonstration analysis (to be deleted). # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/analysis/config/metadata.conf b/reproduce/analysis/config/metadata.conf index 1ab7de1..4c95c2f 100644 --- a/reproduce/analysis/config/metadata.conf +++ b/reproduce/analysis/config/metadata.conf @@ -15,7 +15,7 @@ # and the copyright license name and standard link to the fully copyright # license. # -# Copyright (C) 2020-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2020-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/analysis/config/pdf-build.conf b/reproduce/analysis/config/pdf-build.conf index 7821306..e0400d3 100644 --- a/reproduce/analysis/config/pdf-build.conf +++ b/reproduce/analysis/config/pdf-build.conf @@ -12,7 +12,7 @@ # LaTeX. Otherwise, a notice will just printed that, no PDF will be # created. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/analysis/config/verify-outputs.conf b/reproduce/analysis/config/verify-outputs.conf index 031085d..09cca90 100644 --- a/reproduce/analysis/config/verify-outputs.conf +++ b/reproduce/analysis/config/verify-outputs.conf @@ -1,6 +1,6 @@ # To enable verification of output datasets set this variable to 'yes'. # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/analysis/make/delete-me.mk b/reproduce/analysis/make/delete-me.mk index a20abc6..6bba8ba 100644 --- a/reproduce/analysis/make/delete-me.mk +++ b/reproduce/analysis/make/delete-me.mk @@ -1,6 +1,6 @@ # Dummy Makefile to create a random dataset for plotting. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -26,9 +26,9 @@ # and X^2 and draw its plot. # # Note that this dataset is directly read by LaTeX to generate a plot, so -# we need to put it in the $(tex-publish-dir) directory. -dm-squared = $(tex-publish-dir)/squared.txt -$(dm-squared): $(pconfdir)/delete-me-squared-num.conf | $(tex-publish-dir) +# we need to put it in the $(figdir) directory. +dm-squared = $(figdir)/squared.txt +$(dm-squared): $(pconfdir)/delete-me-squared-num.conf | $(figdir) # When the plotted values are re-made, it is necessary to also delete # the TiKZ externalized files so the plot is also re-made by @@ -87,10 +87,10 @@ $(dm-img-pdf): $(dm-histdir)/%.pdf: $(indir)/%.fits | $(dm-histdir) # # For an example plot, we'll show the pixel value histogram also. IMPORTANT # NOTE: because this histogram contains data that is included in a plot, we -# should publish it, so it will go into the $(tex-publish-dir). -dm-img-histogram = $(tex-publish-dir)/wfpc2-histogram.txt -$(dm-img-histogram): $(tex-publish-dir)/%-histogram.txt: $(indir)/%.fits \ - | $(tex-publish-dir) +# should publish it, so it will go into the $(figdir). +dm-img-histogram = $(figdir)/wfpc2-histogram.txt +$(dm-img-histogram): $(figdir)/%-histogram.txt: $(indir)/%.fits \ + | $(figdir) # When the plotted values are re-made, it is necessary to also delete # the TiKZ externalized files so the plot is also re-made. diff --git a/reproduce/analysis/make/initialize.mk b/reproduce/analysis/make/initialize.mk index 92e5eff..250cb25 100644 --- a/reproduce/analysis/make/initialize.mk +++ b/reproduce/analysis/make/initialize.mk @@ -1,6 +1,6 @@ # Project initialization. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -41,7 +41,7 @@ bsdir=$(BDIR)/software # Derived directories (the locks directory can be shared with software # which already has this directory.). texdir = $(badir)/tex -lockdir = $(bsdir)/locks +lockdir = $(badir)/.locks indir = $(badir)/inputs prepdir = $(badir)/prepare mtexdir = $(texdir)/macros @@ -53,6 +53,45 @@ pconfdir = reproduce/analysis/config +# Directories for to-be-published data +# ------------------------------------ +# +# Its good practice (so you don't forget in the last moment!) to have all +# the plot/figure/table data that you ultimately want to publish in a +# single directory. +# +# There are two types of to-publish data in the project. +# +# 1. Those data that also go into your figures/plots (for example to give +# to LateX's PGFPlots package to create the plot internally) should be +# under the '$(texdir)' directory (because other LaTeX producers may +# also need it for example when using './project make dist', or you may +# want to publish the raw data behind the plots, for example see +# Appendix E of [1]). The contents of this directory are also directly +# taken into the tarball. As you see in [1], it is best to keep these +# files in plain-text format. +# +# 2. The data that are not included directly in the paper's figures and +# can be seen as supplements, or a "data release" associated to the +# report. A good place to keep them is under your build-directory +# (since the data may be large). These files can be custom formats like +# FITS files. +# +# RECOMMENDATION: don't put the figure/plot/table number in the names of +# your to-be-published datasets! Given them a descriptive/short name that +# would be clear to anyone who has read the paper. Later, in the caption +# (or paper's tex/appendix), you will put links to the dataset on servers +# like Zenodo (see the "Publication checklist" in 'README-hacking.md'). +# +# [1] https://scixplorer.org/abs/2026A&A...709A.210E +drdir = $(badir)/data-release +figdir = $(texdir)/figures-data +$(figdir) $(drdir):; mkdir $@ + + + + + # Preparation phase # ----------------- # @@ -73,7 +112,7 @@ pconfdir = reproduce/analysis/config ifeq (x$(project-phase),xprepare) $(prepdir):; mkdir $@ else --include $(bsdir)/preparation-done.mk +-include $(badir)/preparation-done.mk ifeq (x$(include-prepare-results),xyes) -include $(prepdir)/*.mk $(prepdir)/*.conf endif @@ -145,6 +184,7 @@ curdir := $(shell echo $$(pwd)) # avoid conflicts with existing TeX Live solutions. Later (in 'paper.mk'), # we are also going to overwrite 'TEXINPUTS' just before 'pdflatex'. .ONESHELL: +export LC_ALL := C export TEXINPUTS := export CCACHE_DISABLE := 1 export PATH := $(installdir)/bin @@ -265,16 +305,8 @@ clean: # executing 'build'. rm -f *.aux *.log *.synctex *.auxlock *.dvi *.out *.run.xml *.bcf -# Delete all the built outputs except the dependency programs. We'll -# use Bash's extended options builtin ('shopt') to enable "extended -# glob" (for listing of files). It allows extended features like -# ignoring the listing of a file with '!()' that we are using -# afterwards. - shopt -s extglob - rm -rf $(texdir)/macros/!(dependencies.tex|dependencies-bib.tex|hardware-parameters.tex) - rm -rf $(badir)/!(tex) $(texdir)/!(macros|$(texbtopdir)) - rm -rf $(texdir)/build/!(tikz) $(texdir)/build/tikz/* - rm -rf $(bsdir)/preparation-done.mk +# Delete the full 'badir' (containing all analysis outputs). + rm -rf $(badir) distclean: clean # Without cleaning the Git hooks, we won't be able to easily commit @@ -285,7 +317,7 @@ distclean: clean # 'rm' program. So for this recipe, we'll use the host system's 'rm', # not our own. $$sys_rm -rf $(BDIR) - $$sys_rm -f .local .build $(pconfdir)/LOCAL.conf + $$sys_rm -f .local .build @@ -329,12 +361,11 @@ $(project-package-contents): paper.pdf | $(texdir) paper.tex > $$dir/paper.tex # Copy ONLY the version-controlled files in 'reproduce' and -# 'tex/src'. This is important because files like 'LOCAL.conf' (in -# 'reproduce/software/config') should not be archived, they contain -# information about the host computer and are irrelevant for -# others. Also some project authors may have temporary files here -# that are not under version control and thus shouldn't be archived -# (although this is bad practice, but that is up to the user). +# 'tex/src'. This is important because the git commit hash goes in +# the tarball name (should correspond to it) and some project authors +# may have temporary files here that are not under version control +# and thus shouldn't be archived (although this is bad practice, but +# that is up to the user). # # To keep the sub-directory structure, we are packaging the files # with Tar, piping it, and unpacking it in the archive directory. So @@ -362,17 +393,20 @@ $(project-package-contents): paper.pdf | $(texdir) rm -rf $$dir/tex/build/build* # If the project has any PDFs in its 'tex/tikz' directory (TiKZ or -# PGFPlots was used to generate them), copy them too. +# PGFPlots was used to generate them), copy them too. Note that in +# the main project source, 'tex/tikz' is just a symbolic link to +# 'tex/build/tikz'. But inside the tarball we do not want to have +# symbolic links and they should be independent. if ls tex/tikz/*.pdf &> /dev/null; then cp tex/tikz/*.pdf $$dir/tex/tikz fi # When submitting to places like arXiv, they will just run LaTeX once -# and won't run 'biber'. So we need to also keep the '.bbl' file into -# the distributing tarball. However, BibLaTeX is particularly -# sensitive to versioning (a '.bbl' file has to be read by the same -# BibLaTeX version that created it). This is hard to do with -# non-up-to-date places like arXiv. Therefore, we thus just copy the +# and won't run 'biber' or 'biblatex'. So we need to also keep the +# '.bbl' file into the distributing tarball. However, BibLaTeX is +# particularly sensitive to versioning (a '.bbl' file has to be read +# by the same BibLaTeX version that created it). This is hard to do +# with non-up-to-date places like arXiv. Therefore, we just copy the # whole of BibLaTeX's source (the version we are using) into the top # tarball directory. In this way, arXiv's LaTeX engine will use the # same BibLaTeX version to interpret the '.bbl' file. TIP: you can @@ -521,7 +555,10 @@ $(inputdatasets): $(indir)/%: | $(indir) $(lockdir) # Unrecognized format. *) - echo "Maneage: 'DATABASEAUTHTYPE' format not recognized! Please see the description of this variable in 'reproduce/software/config/LOCAL.conf' for the acceptable values."; exit 1;; + printf "Maneage: 'DATABASEAUTHTYPE' format not recognized! " + printf "Please see the description of this variable in " + printf "'$(bsdir)/config/LOCAL.conf' for the acceptable " + printf "values."; exit 1;; esac fi @@ -597,41 +634,6 @@ $(inputdatasets): $(indir)/%: | $(indir) $(lockdir) -# Directory containing to-be-published datasets -# --------------------------------------------- -# -# Its good practice (so you don't forget in the last moment!) to have all -# the plot/figure/table data that you ultimately want to publish in a -# single directory. -# -# There are two types of to-publish data in the project. -# -# 1. Those data that also go into LaTeX (for example to give to LateX's -# PGFPlots package to create the plot internally) should be under the -# '$(texdir)' directory (because other LaTeX producers may also need it -# for example when using './project make dist', or you may want to -# publish the raw data behind the plots, like: -# https://zenodo.org/record/4291207/files/tools-per-year.txt). The -# contents of this directory are also directly taken into the tarball. -# -# 2. The data that aren't included directly in the LaTeX run of the paper, -# can be seen as supplements. A good place to keep them is under your -# build-directory. -# -# RECOMMENDATION: don't put the figure/plot/table number in the names of -# your to-be-published datasets! Given them a descriptive/short name that -# would be clear to anyone who has read the paper. Later, in the caption -# (or paper's tex/appendix), you will put links to the dataset on servers -# like Zenodo (see the "Publication checklist" in 'README-hacking.md'). -tex-publish-dir = $(texdir)/to-publish -data-publish-dir = $(badir)/data-to-publish -$(tex-publish-dir):; mkdir $@ -$(data-publish-dir):; mkdir $@ - - - - - # Print Copyright statement # ------------------------- # @@ -688,27 +690,40 @@ $(mtexdir)/initialize.tex: # Calculate the latest Maneage commit used to build this project: # - The project may not have the 'maneage' branch (for example # after cloning from a fork that didn't include it!). In this -# case, we'll print a descriptive warning, telling the user what -# should be done (reporting the last merged commit and its date -# is very useful for the future). +# case, if a PDF is to be created, print a descriptive warning, +# telling the user what should be done (reporting the last merged +# commit and its date is very useful for the future). +# - This is not relevant when no PDF is requested because we only +# use this to print the latest Maneage commit as a LaTeX macro. # - The '--dirty' option (used in 'project-commit-hash') isn't # applicable to "commit-ishes" (direct quote from Git's error -# message!). +# message!), so no need to include it here. if git log maneage -1 &> /dev/null; then c=$$(git merge-base HEAD maneage) v=$$(git describe --always --long $$c) d=$$(git show -s --format=%aD $$v | awk '{print $$2, $$3, $$4}') + +# No 'maneage' branch found else - echo - echo "WARNING: no 'maneage' branch found! Without it, the latest merge of " - echo "this project with Maneage can't be reported in the paper (which is bad " - echo "for your readers; that includes yourself in a few years). Please run " - echo "the commands below to fetch the 'maneage' branch from its own server " - echo "and remove this warning (these commands will not affect your project):" - echo " $ git remote add origin-maneage http://git.maneage.org/project.git" - echo " $ git fetch origin-maneage" - echo " $ git branch maneage --track origin-maneage/maneage" - echo + +# Only print a warning if a PDF is to be created. + if [ x$(pdf-build-final) = xyes ]; then + echo + printf "WARNING: no 'maneage' branch found! Without it, " + printf "the latest merge of this project with Maneage can't " + printf "be reported in the paper (which is bad for your " + printf "readers; that includes yourself in a few years). " + printf "Please run the commands below to fetch the " + printf "'maneage' branch from its own server and remove " + printf "this warning (these commands will not affect your " + printf "project):\n" + printf " $ git remote add origin-maneage " + printf "http://git.maneage.org/project.git\n" + printf " $ git fetch origin-maneage\n" + printf " $ git branch maneage --track " + printf "origin-maneage/maneage\n" + echo + fi v="\textcolor{red}{NO-MANEAGE-BRANCH (see printed warning to fix this)}" d="\textcolor{red}{NO-MANEAGE-DATE}" fi diff --git a/reproduce/analysis/make/paper.mk b/reproduce/analysis/make/paper.mk index 66c6859..c218cdd 100644 --- a/reproduce/analysis/make/paper.mk +++ b/reproduce/analysis/make/paper.mk @@ -1,6 +1,6 @@ # Build the final PDF paper/report. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -18,6 +18,22 @@ + +# Software info in TeX +# -------------------- +# +# The information of the installed software is placed in the +# '.build/software' directory (which the TeX build should not depend +# on). Therefore, we should copy those macros here in the LaTeX build +# directory, so the TeX directory is completely independent from each +# other. +$(mtexdir)/dependencies.tex: $(bsdir)/config/dependencies.tex + cp $(bsdir)/config/*.tex $(mtexdir)/ + + + + + # LaTeX macros for paper # ---------------------- # @@ -39,19 +55,36 @@ # Note that if you don't want the final PDF and just want the processing # and file outputs, you can give any value other than 'yes' to # 'pdf-build-final' in 'reproduce/analysis/config/pdf-build.conf'. -$(mtexdir)/project.tex: $(mtexdir)/verify.tex +$(mtexdir)/project.tex: $(mtexdir)/verify.tex $(mtexdir)/dependencies.tex # If no PDF is requested, or if LaTeX isn't available, don't continue # to building the final PDF. Otherwise, merge all the TeX macros into # one for building the PDF. - @if [ -f .local/bin/pdflatex ] && [ x"$(pdf-build-final)" = xyes ]; then + @if [ -f .local/bin/pdflatex ] && [ x"$(pdf-build-final)" = xyes ]; + then # Put a LaTeX input command for all the necessary macro files. -# 'hardware-parameters.tex' is created in 'configure.sh'. +# Note that 'hardware-parameters.tex' is created in 'configure.sh' +# (which is why it is first in the list). projecttex=$(mtexdir)/project.tex rm -f $$projecttex - for t in $(subst paper,,$(makesrc)) hardware-parameters; do - echo "\input{tex/build/macros/$$t.tex}" >> $$projecttex + for t in hardware-parameters $(subst paper,,$(makesrc)); do + +# Only add the macro file to 'project.tex' if the macro file is +# not empty. Empty macro files are not a bug in Maneage alone: 1) +# the processing of a sub-Makefile may not need to produce any +# written macro for the PDF, and that is perfectly fine. 2) LaTeX +# can also '\input' an empty file. However, it is important for +# for publication servers (like arXiv) that do not allow +# uploading empty files. Therefore, when 'project.tex' contains +# empty files, such severs will not be able compile the PDF +# because the file name is mentioned in 'project.tex', but cannot +# be found in the uploaded files. + tfile=tex/build/macros/$$t.tex + if [ -s $$tfile ]; then + echo "\input{$$tfile}" >> $$projecttex + fi + done # Possibly highlight the '\new' parts of the text. @@ -92,6 +125,23 @@ $(mtexdir)/project.tex: $(mtexdir)/verify.tex +# TeX build directory +# ------------------- +# +# If built in a group scenario, the TeX build directory must be separate +# for each member (so they can work on their relevant parts of the paper +# without conflicting with each other). +ifeq ($(strip $(maneage_group_name)),) +texbdir:=$(texdir)/build +else +texbdir:=$(texdir)/build-$(shell whoami) +endif +tikzdir:=$(texbdir)/tikz + + + + + # The bibliography # ---------------- # @@ -104,8 +154,8 @@ $(mtexdir)/project.tex: $(mtexdir)/verify.tex # recipe and the 'paper.pdf' recipe. But if 'tex/src/references.tex' hasn't # been modified, we don't want to re-build the bibliography, only the final # PDF. -$(texbdir)/paper.bbl: tex/src/references.tex $(mtexdir)/dependencies-bib.tex \ - | $(mtexdir)/project.tex +$(texbdir)/paper.bbl: tex/src/references.tex | $(mtexdir)/project.tex + # If '$(mtexdir)/project.tex' is empty, don't build PDF. @macros=$$(cat $(mtexdir)/project.tex) if [ x"$$macros" != x ]; then @@ -130,12 +180,11 @@ $(texbdir)/paper.bbl: tex/src/references.tex $(mtexdir)/dependencies-bib.tex \ # for details. # # We need the modification to 'LD_LIBRARY_PATH' because we do not -# build LaTeX from source and it uses '/bin/sh' (among other -# possible system-wide things). - export LD_LIBRARY_PATH="$(sys_library_sh_path):$$LD_LIBRARY_PATH" +# build LaTeX from source and it (or its packages) may use +# '/bin/sh' (among other possible system-wide things). + export LD_LIBRARY_PATH="$(SYS_LIBRARY_SH_PATH):$$LD_LIBRARY_PATH" pdflatex -shell-escape -halt-on-error "$$p"/paper.tex biber paper - fi @@ -165,9 +214,9 @@ paper.pdf: $(mtexdir)/project.tex paper.tex $(texbdir)/paper.bbl # option '-shell-escape'. # # We need the modification to 'LD_LIBRARY_PATH' because we do not -# build LaTeX from source and it uses '/bin/sh' (among other -# possible system-wide things). - export LD_LIBRARY_PATH="$(sys_library_sh_path):$$LD_LIBRARY_PATH" +# build LaTeX from source and it (or its packages) may use +# '/bin/sh' (among other possible system-wide things). + export LD_LIBRARY_PATH="$(SYS_LIBRARY_SH_PATH):$$LD_LIBRARY_PATH" pdflatex -shell-escape -halt-on-error "$$p"/paper.tex # Come back to the top project directory and copy the built PDF diff --git a/reproduce/analysis/make/prepare.mk b/reproduce/analysis/make/prepare.mk index 2cc1187..352fd00 100644 --- a/reproduce/analysis/make/prepare.mk +++ b/reproduce/analysis/make/prepare.mk @@ -1,6 +1,6 @@ # Basic preparations, called by './project make'. # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -25,7 +25,7 @@ # # We need to remove the 'prepare' word from the list of 'makesrc'. prepare-dep = $(filter-out prepare, $(makesrc)) -$(bsdir)/preparation-done.mk: \ +$(badir)/preparation-done.mk: \ $(foreach s, $(prepare-dep), $(mtexdir)/$(s).tex) # If you need to add preparations (mainly automatically generated diff --git a/reproduce/analysis/make/top-make.mk b/reproduce/analysis/make/top-make.mk index 2689e64..7286ca6 100644 --- a/reproduce/analysis/make/top-make.mk +++ b/reproduce/analysis/make/top-make.mk @@ -1,6 +1,6 @@ # Top-level Makefile (first to be loaded). # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -19,9 +19,9 @@ -# Load the local configuration (created after running -# './project configure'). -include reproduce/software/config/LOCAL.conf +# Load the local configuration (created after running './project +# configure'). +include .build/software/config/LOCAL.conf diff --git a/reproduce/analysis/make/top-prepare.mk b/reproduce/analysis/make/top-prepare.mk index 7d92d72..913a062 100644 --- a/reproduce/analysis/make/top-prepare.mk +++ b/reproduce/analysis/make/top-prepare.mk @@ -4,7 +4,7 @@ # are not included here. Please see that file for thorough comments on each # step. # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -23,9 +23,9 @@ -# Load the local configuration (created after running -# './project configure'). -include reproduce/software/config/LOCAL.conf +# Load the local configuration (created after running './project +# configure'). +include .build/software/config/LOCAL.conf @@ -36,16 +36,16 @@ include reproduce/software/config/LOCAL.conf # # See 'top-make.mk' for complete explanation. ifeq (x$(maneage_group_name),x$(GROUP-NAME)) -all: $(BDIR)/software/preparation-done.mk +all: $(BDIR)/analysis/preparation-done.mk @echo "Project preparation is complete."; else all: @if [ "x$(GROUP-NAME)" = x ]; then \ echo "Project is NOT configured for groups, please run"; \ - echo " $$ ./project prepare"; \ + echo " $$ ./project configure"; \ else \ echo "Project is configured for groups, please run"; \ - echo " $$ ./project prepare --group=$(GROUP-NAME) -j8"; \ + echo " $$ ./project configure --group=$(GROUP-NAME) -j8"; \ fi exit 1 endif @@ -61,7 +61,7 @@ endif # # To ensure that 'prepare' and 'make' have the same basic definitions and # environment and that all 'downloads' are managed in one place, both -# './project prepare' and './project make' will first read 'initialize.mk' +# './project configure' and './project make' will first read 'initialize.mk' # and 'downloads.mk'. makesrc = initialize \ prepare diff --git a/reproduce/analysis/make/verify.mk b/reproduce/analysis/make/verify.mk index c74f8ca..91c4b77 100644 --- a/reproduce/analysis/make/verify.mk +++ b/reproduce/analysis/make/verify.mk @@ -1,6 +1,6 @@ # Verify the project outputs before building the paper. # -# Copyright (C) 2020-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2020-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -138,11 +138,44 @@ $(mtexdir)/verify.tex: $(foreach s, $(verify-dep), $(mtexdir)/$(s).tex) # Verify TeX macros (the values that go into the PDF text). for m in $(verify-check); do - file=$(mtexdir)/$$m.tex - if [ $$m == delete-me ]; then s=711e2f7fa1f16ecbeeb3df6bcb4ec705 - else echo; echo "'$$m' not recognized."; exit 1 + +# Set the macro file name, make sure it exists (the user may have +# forgot to create it in the respective sub-Makefile). + tfile=$(mtexdir)/$$m.tex + if ! [ -f $$tfile ]; then + printf "$$tfile: ERROR: does not exist! If you do not " + printf "(currently) plan macros to be added to the final " + printf "PDF from that sub-Makefile, do the following to " + printf "avoid this error: put a 'touch \$$@' as the recipe " + printf "of the '\$$(mtexdir)/$$m.tex' target (at the bottom " + printf "of 'reproduce/analysis/make/$$m.mk'). If you later " + printf "decide to define macros there, simply replace the " + printf "temporary 'touch \$$@' command with the command(s) " + printf "to prepare/write the macro (see 'initialize.mk' " + printf "as an example)\n"; + exit 1 fi - $(call verify-txt-no-comments-no-space, $$file, $$s, $@.tmp) + +# Check the hash for this file and verify it. + case $$m in + delete-me) s=711e2f7fa1f16ecbeeb3df6bcb4ec705;; + *) # This file's hash is not in the list above. + +# If the file is empty, we can skip it (with the 'continue' +# command). Otherwise, the user should be informed with an +# error. + if [ -s $$tfile ]; then + printf "$$tfile: ERROR: please add the hash of this " + printf "macro file to the list of hashes above this " + printf "error message (in " + printf "'reproduce/analysis/make/verify.mk')\n"; exit 1 + else + continue + fi + ;; + esac + $(call verify-txt-no-comments-no-space, $$tfile, $$s, $@.tmp) + done # Move temporary file to final target. diff --git a/reproduce/software/bibtex/astrometrynet.tex b/reproduce/software/bibtex/astrometrynet.tex index 5697885..3c012cf 100644 --- a/reproduce/software/bibtex/astrometrynet.tex +++ b/reproduce/software/bibtex/astrometrynet.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/astropy.tex b/reproduce/software/bibtex/astropy.tex index 3df1aa1..aa9d162 100644 --- a/reproduce/software/bibtex/astropy.tex +++ b/reproduce/software/bibtex/astropy.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/corner.tex b/reproduce/software/bibtex/corner.tex index fd7ea76..164adda 100644 --- a/reproduce/software/bibtex/corner.tex +++ b/reproduce/software/bibtex/corner.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/cython.tex b/reproduce/software/bibtex/cython.tex index 311072d..0c761ce 100644 --- a/reproduce/software/bibtex/cython.tex +++ b/reproduce/software/bibtex/cython.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/fftw.tex b/reproduce/software/bibtex/fftw.tex index 15c5c62..f5f4069 100644 --- a/reproduce/software/bibtex/fftw.tex +++ b/reproduce/software/bibtex/fftw.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/galsim.tex b/reproduce/software/bibtex/galsim.tex index bbfdec1..c3cd7d6 100644 --- a/reproduce/software/bibtex/galsim.tex +++ b/reproduce/software/bibtex/galsim.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/gnuastro.tex b/reproduce/software/bibtex/gnuastro.tex index 7d56e3e..8f97305 100644 --- a/reproduce/software/bibtex/gnuastro.tex +++ b/reproduce/software/bibtex/gnuastro.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/healpix.tex b/reproduce/software/bibtex/healpix.tex index afdd6a6..6063722 100644 --- a/reproduce/software/bibtex/healpix.tex +++ b/reproduce/software/bibtex/healpix.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/imfit.tex b/reproduce/software/bibtex/imfit.tex index 1d5270b..a1f8b4d 100644 --- a/reproduce/software/bibtex/imfit.tex +++ b/reproduce/software/bibtex/imfit.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/kiwisolver.tex b/reproduce/software/bibtex/kiwisolver.tex index 47d3f56..62df2a2 100644 --- a/reproduce/software/bibtex/kiwisolver.tex +++ b/reproduce/software/bibtex/kiwisolver.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2025-2025 Boud Roukema <boud@cosmo.torun.pl> +%% Copyright (C) 2025-2026 Boud Roukema <boud@cosmo.torun.pl> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/matplotlib.tex b/reproduce/software/bibtex/matplotlib.tex index ec7bab4..68d648a 100644 --- a/reproduce/software/bibtex/matplotlib.tex +++ b/reproduce/software/bibtex/matplotlib.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/missfits.tex b/reproduce/software/bibtex/missfits.tex index 83510dd..b973248 100644 --- a/reproduce/software/bibtex/missfits.tex +++ b/reproduce/software/bibtex/missfits.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2020-2025 Surena Fatemi <surena.fatemi@ipm.ir> +%% Copyright (C) 2020-2026 Surena Fatemi <surena.fatemi@ipm.ir> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/mpi4py.tex b/reproduce/software/bibtex/mpi4py.tex index 10e0e74..6125251 100644 --- a/reproduce/software/bibtex/mpi4py.tex +++ b/reproduce/software/bibtex/mpi4py.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/numpy.tex b/reproduce/software/bibtex/numpy.tex index fb017c4..8955d3d 100644 --- a/reproduce/software/bibtex/numpy.tex +++ b/reproduce/software/bibtex/numpy.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/parallel.tex b/reproduce/software/bibtex/parallel.tex new file mode 100644 index 0000000..1ba1018 --- /dev/null +++ b/reproduce/software/bibtex/parallel.tex @@ -0,0 +1,15 @@ +%% Copyright (C) 2026-2026 Boud Roukema <boud@cosmo.torun.pl> +%% +%% Copying and distribution of this file, with or without modification, +%% are permitted in any medium without royalty provided the copyright +%% notice and this notice are preserved. This file is offered as-is, +%% without any warranty. + +@ARTICLE{parallel, + author = {{Tange}, Ole}, + title = "{GNU Parallel 20260322, https://doi.org/10.5281/zenodo.19321428}", + journal = {Zenodo.19321428}, + year = 2026, + month = mar, + doi = {10.5281/zenodo.19321428}, +} diff --git a/reproduce/software/bibtex/r-cran.tex b/reproduce/software/bibtex/r-cran.tex index c58eeeb..f17fc3a 100644 --- a/reproduce/software/bibtex/r-cran.tex +++ b/reproduce/software/bibtex/r-cran.tex @@ -1,5 +1,5 @@ -%% Copyright (C) 2022-2025 Boud Roukema <boud@cosmo.torun.pl> -%% Copyright (C) 2022-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2022-2026 Boud Roukema <boud@cosmo.torun.pl> +%% Copyright (C) 2022-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/scamp.tex b/reproduce/software/bibtex/scamp.tex index b5c4da9..271a09e 100644 --- a/reproduce/software/bibtex/scamp.tex +++ b/reproduce/software/bibtex/scamp.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/scipy.tex b/reproduce/software/bibtex/scipy.tex index 1296393..0574891 100644 --- a/reproduce/software/bibtex/scipy.tex +++ b/reproduce/software/bibtex/scipy.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/sextractor.tex b/reproduce/software/bibtex/sextractor.tex index 818f04a..33ec1e8 100644 --- a/reproduce/software/bibtex/sextractor.tex +++ b/reproduce/software/bibtex/sextractor.tex @@ -1,5 +1,5 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/sip_tpv.tex b/reproduce/software/bibtex/sip_tpv.tex index 0d52e33..d9b7cdd 100644 --- a/reproduce/software/bibtex/sip_tpv.tex +++ b/reproduce/software/bibtex/sip_tpv.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/swarp.tex b/reproduce/software/bibtex/swarp.tex index e0989f9..813942d 100644 --- a/reproduce/software/bibtex/swarp.tex +++ b/reproduce/software/bibtex/swarp.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/sympy.tex b/reproduce/software/bibtex/sympy.tex index 01e481b..18e4254 100644 --- a/reproduce/software/bibtex/sympy.tex +++ b/reproduce/software/bibtex/sympy.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +%% Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/bibtex/tides.tex b/reproduce/software/bibtex/tides.tex index caf1500..00d8195 100644 --- a/reproduce/software/bibtex/tides.tex +++ b/reproduce/software/bibtex/tides.tex @@ -1,4 +1,4 @@ -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> %% %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright diff --git a/reproduce/software/config/LOCAL.conf.in b/reproduce/software/config/LOCAL.conf.in index 341a78e..7d45d06 100644 --- a/reproduce/software/config/LOCAL.conf.in +++ b/reproduce/software/config/LOCAL.conf.in @@ -3,18 +3,37 @@ # This is just a template for the './project configure' script to fill # in. Please don't make any change to this file. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and # this notice are preserved. This file is offered as-is, without any # warranty. -BDIR = @bdir@ -INDIR = @indir@ + + + + +# Local system settings +# --------------------- +# +# Build directory (mandatory). All the created files of the project will be +# within this directory. +BDIR = @bdir@ + +# Input data directory. This can be empty or a non-existant location. If +# so, then the inputs will be downloaded from the 'INPUTS.conf' into the +# build directory. +INDIR = @indir@ + +# Software source code directory. This can be empty or a non-existant +# location. If so, the software tarballs will be downloaded. DEPENDENCIES-DIR = @ddir@ -SYS_CPATH = @sys_cpath@ -DOWNLOADER = @downloader@ -GROUP-NAME = @groupname@ + +# Other local settings (compiler, downloader and user). +SYS_CPATH = @sys_cpath@ +GROUP-NAME = @groupname@ +DOWNLOADER = @downloader@ +SYS_LIBRARY_SH_PATH = @sys_library_sh_path@ diff --git a/reproduce/software/config/TARGETS.conf b/reproduce/software/config/TARGETS.conf index 716fd41..51791a5 100644 --- a/reproduce/software/config/TARGETS.conf +++ b/reproduce/software/config/TARGETS.conf @@ -1,7 +1,7 @@ # Necessary high-level software to build in this project. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and @@ -42,5 +42,7 @@ top-level-programs = gnuastro top-level-python = # R libraries/modules -# [For developers 2022-01-02: 'r-cran-cowplot r-cran-gridExtra' for all] +# +# As of 2022-01-02: to have all of the R packages, select 'r-cran-cowplot +# r-cran-gridExtra'. top-level-r-cran = diff --git a/reproduce/software/config/checksums.conf b/reproduce/software/config/checksums.conf index 89738ce..d3717cb 100644 --- a/reproduce/software/config/checksums.conf +++ b/reproduce/software/config/checksums.conf @@ -1,8 +1,8 @@ # sha512 checksums of all the necessary software tarballs. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> -# Copyright (C) 2022-2025 Pedram Ashofteh Ardakani <pedramardakani@pm.me> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2022-2026 Pedram Ashofteh Ardakani <pedramardakani@pm.me> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and @@ -15,43 +15,42 @@ # Basic/low-level programs and libraires (installed in any case) # -------------------------------------------------------------- -bash-checksum = 52fa7d14e5f05d7b1b5f64ca110388db119fd6b8d4cc25011e0e25848491507e610259091c56f3ca7a7688ee2547f8a98de2005626b35c29558e71f28e29b848 -binutils-checksum = efbc5957bd22804a860261886598778347e82f9cbc0728d21d26748b7296e34328fdc16252868169594a001a301babbcd0964d9c508523736bc79cac2aff8efc +bash-checksum = b905e107403ceb25ad92f26ab71d6a7f12b1cadbcb27de9f5fefa7da31e96de0be281f8095170ed36e3fc22237dcfc252ff674925e6123638f6a3e11f3c5ecfd +binutils-checksum = ad7bbebc0c4e537c15d7537027246e2c1c0d355efcd6a369c7ef91c6491ec6a6e9af24c60b7df7508213abeedbd2e53c41549889073ac1f629b0746b78ee894b bzip2-checksum = 929cb1b2d0db8a505e2bafe3ce2c893c8f132dd617c0110e86e19f116b570a85de3c8a635483b530456fc8f8b6698c8190d21313e47607f442807b10489ac86e certpem-checksum = d1198511bf1f61a624691c182ecdd57ff468030a4af8b0f0ef248a5ce1f42ed883c684fc5aafdd791d5d3aa24504fbfbab11043231594b1c4e21fdb7247e5a33 -coreutils-checksum = 2afaee98b305f000ab1c9e25b5fec6413c3e385c685f2afe94e3676593c09efca39d007cfda2b3b122e68e94204d2ebe7f50b39300899d8720518d6f74bce019 -curl-checksum = f391f696d725ace63271fe614ab3067b8d55d04be3a160c70bbca388de93670e5fb986dd5dd45e1b62c0327b2b1374ab229bdb5c9df746f70823e0721c9f048d +coreutils-checksum = 6f761311e673ded32c0b8d29605630bc703920718b305937a057f992fb344106e57ea96b51bba5ac674a815acc592fb127eb38a4fba6c40f07ff6b123c432e87 +curl-checksum = 5b0213eb1d1c2146c591a72ba0d763c40cf707c7fcc43f4807cccec12ddd3cc63127da0f18f96c2450efc8607812519dbccbc924b5b629712fbc7b02b4976aee dash-checksum = f21a2ac71606a5ae8dbda5639d111f385cc44c357c580a83e453b33faa32c92fd83f8ca6c5d2cf720343f8ab1cfec247ca499596f78ee19f62d67cd1d7ae0c69 -diffutils-checksum = 52582c860a4ce544f0c1a3e61bd9a1ce5f13e1c677b67d3713982439760e40b60267b222de3ef0a27065652822d9880a206899528d7ff8d60c7e64b64721b3e1 +diffutils-checksum = 9d42c57a5ae4770b6d84b76f324c7c09c62792cac063048793037281972983dd4c409868545f2460c281f761adba6c49fab3fa227ab5231397809474cedbdfc1 file-checksum = b843b3c25656e8dec52e64eed6f581b29faf36540e0604a803d61c0f0eca830a01a947b81970b0d8a24a1336a37ae5c96bb2ade59daa16c544b1e82fc7db24e8 findutils-checksum = 826c643e7f5c5d6976a47eabcd9807e51350d09ee8fc7dc931f2d9276f938f65aa0bd97e6213aa979742234784c120e1a6850a52207c327e1c1a465feb374053 flock-checksum = f711815035e21b46572bf80e730a55822e5abf4cb29749e476ee6cf4d5027e9a7deeacf5f6b8c37f18f17a0cc7a6d98fb0be3936e97b122707f1cb2306d1e1d9 -gawk-checksum = 13cb59a4ef43e5b0d10f13263dcf5ccd72d9344f3ecc512589ed7f6d059baf8a5ae375d38f7654695e29f0694fc33deba3c7c008d61f09e67df57bf81e573f0c -gcc-checksum = 40f239730fc0165a0ef48f252a886e9cbe7a95ed53fe2c7a16531d7f3d6faabecb2d868cc895a25f7561e7bb90892e3372652953efd8ac35359a6ac2e18a4447 -gettext-checksum = c53f3a09cbb62a3ba8870f1310da336c084bfc83534d7e9c236f7660f503d638654a91301558f1e4880ead2d629dcd92f7c3dd7e0d981d445b74215d836d19ea -git-checksum = 6e7ac2188cf3204fceeebffc46f34a8180e53a887987a8e1b9d6c90f84073e4b86d45f39488f69432b1cd0962cd8fbd434c07128fc2e430cd2ec20f1a1fc8b61 +gawk-checksum = 9013f5cb4c08aa13029ecf0c6b5e02ac93a3b500683b07ae8d34d698f0451dd5146a0a5aef96a249112834e27b379869d8e541f6a1b99accf701ba6a48f58f33 +gcc-checksum = b2191bf7005c21dee7675cc1f9ed615de418eac6afaf8dd6b3fc6022660d2ae63aec66a3fd072f227fa05da1654e04c2d06fb331fd50798c768de7ef1a7d9590 +gettext-checksum = dbdc5016a776f12feb7134c05e2568ba9134bfa66b824967b4cc81d2093328c0f3bbcf269ad5a60ff2257625c763ac96c071685ee2b1d397730b06a058f565aa +git-checksum = e2a3d2cf272554c0e9fa2abc4cf7b1eff17884911859f17567b76f170996802caa01675a42dbf29af7fc283a07e8dc4a11382ffbd2a03d7d62238e8c598f0b56 gmp-checksum = ad65de00ecb46cf454ed6c40d2a57ce2528f5fa64df1284dfa15036f1e8cf27760a09a4ecdfcc39048faffb71339bba30d99dd365c54173dbc2ba629bee2fad9 -grep-checksum = b1ad850671290120ca0ced6d1744622d99a903c6e638dd231e89a70a2ab48c382100e218697b6fbbbda8c99316679a7c681383a5e954790bbffb9719bd0d6d8e -gzip-checksum = 19ef6f51ee8274c08aa188ebd26634a40f91c057ff1ad3253719a7fa0c7ac857de5cf289990e0b9fc67d89ca3abb50db6dbaa7664d2c58275c844712834d5e10 +grep-checksum = 4e39935e26a0d742c0e854135940693166ea142673f50dfd589eedc96dded0e8dcad425e6519ee85d59fdf9d418381ba775a7c560cbeebcdf8a1848339243145 +gzip-checksum = 3c61b50f3a9efabf69b4b91410ffd83e0f3a54f1d465a5326e66972e18fe829eaabc77bf47b89340d59f6c5b57fb4c03e293cb7044c57ad7526c8de9bbc80ed7 isl-checksum = 615827a86e809c9645e090786008f8b37d7efc7fee6501ff81e937a299a5a72e7998a3965457d1380cb04f7e765b600276d87b7b59cae22d82aa65e71294f499 -less-checksum = 3b52347e1b779a52c89cb18da85a0963eed500e1e94cd3952c0693c56fd0eec4e8606eb02053d451ac92ec8c6e196edd1341eac54515ed84483112513f41f210 +less-checksum = f02557d9b85d4144e530ef8f2e95329b7bd2eba1be2513558cd7359cf520e1e6723c4d62b6dd42082084003ac9cb23c3d1d9ad7ca73d58597f7fedf31e9d08fd libiconv-checksum = 1f33fcf7f617990812a26e15092dbb2f9e81400cee117eefd40ade813f7ca17c0b9a50a7a9881b4ab0b9c0be8475528204c347d940cbc667a918570e0bde051a libtool-checksum = 2aa6d535b4d644393d7ca5c8f174b09923dbdb50d947ba40917aad75c8cecf957fec7a54717bd3670c44130331b1f08fae3694e32c79c8e187c31e909b3d401b -libunistring-checksum = 6ca6a2dea2c09b6e8e63eb7ff73ab62b2e2f7c412766209c9f6da5b13a109e3a7cb41e67aaee1fb2b46549965b7df13c508000e40594570abe71819dc9cc2eaf -libxml2-checksum = 7c2c65ae5017be5d695ec5a0e9bf443fe130d33beec97f31bcdbe74c22ac7745ce02524ca603e701035576bdca91c4cb1cbe67ccad5525f9a4566451a8f0c935 +libunistring-checksum = 6621b23b89e68f38a88bd0f0d017c1f20c0b81aa35901ce0e2620054de41a89194c71e55c27de5b9aa0dbbffdb9479c2411b9069cdad09ed67ecf4b1611af525 +libxml2-checksum = aee3946beba88dba0a4ba573704da0c48de3233e5a1717ac6d8e647f3c8b65131c1b4d320604adedd8eff3364af11a3489a2c26b57b8151b59fca60da3f2a1c3 lzip-checksum = 513b7ecdee1d6f12298cbbb025286c887c5e6588e1268b489735955f822a8e305e7a8d4d06054a77b2f4c40f8d82e9e6d779cd73d4227ca3cdfe4002b9e42a98 -m4-checksum = 7f8845f99e64d6a45859b9d80b03352a5526b3de0311ca4d6dd6850e504d26dfc90cd21d1640b10382f786213f8fdf20183bff424b3c41ea11432315993ab829 +m4-checksum = 3025f1bc2f645d0fb7ea3be4788d4670c8aa7901c8afcd90f3f04454b80fe855ce5bb036061ea3d6e800f58cb3721efd72cbf00a7598ecf931377c43e41d1508 make-checksum = 154a0247297a7b0131ff63fa8636d651b33aacc8f4a0dad1db176995b20c2039f4fd58eeb2ec27cc0ca7a95c1853199a4ee35a14afc5084995ecc1d694203412 mpc-checksum = 76e0720e6287d8b7a3eaa09dc5baa5a2b61dde88198c9e1bc9c458a268a44035d11fbab1c8019501a7ee2d5745c7eddfb0bb3c7297110a0fd9e027acb42d0fc3 -mpfr-checksum = c6013b0f573271a3bc41bc3eb28c61f099c6fc5409096a27624b5ec049e6305a1c48275c6f244d0cbe1fe65f649704c7d5f6487ad24308f6b3becc17032fc222 -nano-checksum = e559c09057ff75cd650f02744dff801750a159785234189e48f976bb4ab90142aca2577a283f80d8eecdc4f2b6c0a55bfe6da69e6aaecc5812f32f923075ca9c +mpfr-checksum = 40066bbbfb394ad4fc2efcbf633542c43fc6e31e3503d0f8f425649c95cfd6aeed78e889852bdfe23f783b5b13766ebdada9c92e6a921d84c6a8703123e9c165 +nano-checksum = 964ffb246e186ed865898fa29c4d39e869092c84685312dc00feeb4bc72ad019053384a79d7155a6fc396fb5f8a5d16fb894d1dd35fbc71563d3a52e79351f27 ncurses-checksum = c94eccc1b23a9c6ca9b27881674b19802942802cf21084bd80e45e0b9e50a2d99e0e18a1ecd5cf3e686949982ca93132a7fe2f117a7c4307283db5012b11a2d5 -openssl-checksum = c4c44ab8e4a6d39e6aa69b096a831645c33d5675619d31da6e441452f67e487489b33041a34038a0f8ad596506e17ce8695a8f87bae861700481ecf6b9742b76 +openssl-checksum = 7cf3cfa3a190a8944366393942faab0813c8443dce5bdb8bdf11b7cae3627f9a3ed2698c6785ef4688c42df12983c2a9ce932e58f2e3691e752589dbae5ea363 patchelf-checksum = f74409c00e7e50a88590267fa0c173f71239f1471985792f7bcdab750d202a9a25383c015987608bbcd6ace5b3642645d46dba63199ba54fabbcb51ae7fca8f6 -perl-checksum = d53da403f2232b487f11df4b0b889babf58350715c7430515ba32e2e7d9996308c5d75ee626906a9f3c56872672c9cfe3880e10c7bb084c178c7c76bb2a5346f +perl-checksum = d5c782948b97b706fe9650f6a9995591d9b90da3a98866d1db4d20025262d5606f4133222be4b96cc06578323db03759910e96012d8931033feaddca0e156e31 pkgconfig-checksum = c37ab9336ac15a73bf5a23101f3fd8a04810f39b0679634e420e5d7bbd60a80cfc919122694cf83965beb071d2780681efc8d954ce99d77eb235a4dc7497e250 -podlators-checksum = 2e1e8547b0c890101a9fb8d80807b164a0a708820617a6826127930f85115285a0e4d62ddf8ce2401df14c021521a1b3ce0cad0d308e08f7b3d0c49d1b0dd36f -readline-checksum = ca08305c5a6d5eb6656a7cdff73aab65c1234de62db8da90335e9ea140b1d4f8a867204bdc49ead3acc5e5236358130e73032bd73de5dfe1eee5bb2c76be5cb2 +readline-checksum = 349248d78cfc1deebdaaefc70a044835139cb8775d74bb33ae2a2600d9daef2b169608a8f706fc803a8d95fe9b309da1db0c6d19fa78eaa67430aff5b13feaa2 sed-checksum = e801dacce45a08671e65897f85be0a37a29d97f61e846ca17ff870d3ab9f5a639dba73457491087ff637df4a81099b864d2496d59acf17a07916b43bfe0ae480 tar-checksum = f8d0b543dc4a2a57d61d8a183d7a64d611d11f033c64683f022a7f1f956d352d364a959903a2549ab1256c1a068de51052960cf34a70f355b80e237f654bfac7 texinfo-checksum = 06e7d95d73380c16f064e119bd8717942fb464c1cbc532fb1833d2d5dfc3cbbcab2a9001a1d03721c347f676e1db22c2a0f78d19d22df379d0393527be411ef3 @@ -60,7 +59,6 @@ valgrind-checksum = a99e09e6d957ce435e64f4ce7b1a14e7e266282578171ce4e3bb3f405ce3 wget-checksum = 75ba60150baf673def8ff5f2688af57d89d699f1b5105cb1a6bc0ce88778be191c74757665d773ac84fdab975851ca9015b30f12f31877fd752500efb49e0034 which-checksum = cbda59450d35c2fa04f9072db43584ff745844cdb129bce1a382eb50bcd19815f0bbf12f2b03f748cb3f585acf0a16eb2e661b71f9066469236d28e9fa40f168 xz-checksum = d897a55ae5ee7d778c68edff83c53e6624619f233acd529439fade305681d07b7665ba1d370cf84abb38c7f3f8f88c3d127974c4f59667843814211802501080 -zip-checksum = 433eda6a27074746a960952fa3b08028d5ba43fe976a1306dbace9209be5f89aa0554b17bba5815cd00984c0f64559c5fa0b754b620f6d646a2b145a8b599acc zlib-checksum = 8f0b28c7d0d5d1906c2c03586f4ad3188a7c7eb73d25295a7acaa88cdf87b7c82ac609282b9b12d2b4d5a368169e4bb5f88ba3b840b57e1cf32c78f2ee3ecd86 @@ -85,27 +83,30 @@ bison-checksum = 08c3e9bcf9fc01bc8b3a6c5e5f8ecdf628e07d6ce0874341e9df6d7b2925db7 boost-checksum = df252f6aabd9b1d5421afc478fa288e947a7c5e11f26e54f0a8619c9ac5e16688c20f44106a429ae798b20a4c97cbf800ee0ac9b2aa7febe1bd2755c88802ac1 cairo-checksum = 5de8528196b450d01da5cd4562dec0dd8c31028b77640b6c611172f7c5ad1dc9a4d30bd4e38259344c3ee42b979e29be702a3190aae5fbf47c82c79ee0c14a8e cdsclient-checksum = 24584eedeb84ab4666cbd2ed0b7264c92c63c9cf090595b75b01dd46f848419d6a7e5cff4db3946695f879d0ee2bfbeb527e88c2be85e769b577302d9b2ad2b7 -cfitsio-checksum = fd40e0cec22bf3a35a7c48ad61a6ebad5a54d3230606c7492161c067b7ccd303ed34d149427bf1d3365e7a3082d89b51df46fc8a9c3d3c0b1eef8756374d711e +cfitsio-checksum = 60e10544bad66812dca3ff04620051c2c99d047952b5fa6280f4ca443affcbe4806b84f4cb439036b9affd42ab82cf723ae7866aabd33629b25e8edaaba63c72 cmake-checksum = 3904964eeb256ae4443d000a24ebdbfd76c677aec08d006f8665020f1ebf2008c65b6a1cffd6fb5b8a29cdb52420415ed2e0e2ea258db2e9888dc48068fc3941 eigen-checksum = 39c1944e0daca50bb01e929edc98e2bfc234accb30ca019dcccfc7b02db5cc81035592be219ae93cc3ec16fe0255e7e4d1f29ca2ad3dfee7e63546c47cb2f807 emacs-checksum = 91313dced8cd74f9e4a996eef42806aad00fb2576fc77b3694bb7b670624b6a1551ff49c2a3326dbeeba916e5732ca5975b5dc40b3b063f6e73a22839009abe9 expat-checksum = c006dff4ee90ee3722df94ef227cfaab3ee4ebf7ee8193a65b904798b4d185e89c863c75a1e8eae9c120bfbf79336c1e31a8b6aa3042f4131b9a1eb0e97c6643 fftw-checksum = 28bd2f620399a415181027d30d7ea193aa487c7a277c9943d0051488908fd87e2731de0dfc3bebd22a6121d1deaa46037be8296a8a9cdb711f9fde4510c3d368 flex-checksum = a18e0dcdd21bd51779fd70e5efc2d77fa7b2a56d4d0e929d8abb437f86d7fcfa2cd09b8268185698200866dac8bec4444c5a7cff3140c515eaa49fc77c5aea66 +framel-checksum = f93203bc7cebb35f2ef356fef7190ae6874345643ec35ebe7db22228fb1f0182656fa03339c1f749381ff875e7955f9b98240bc5b55889f0d97cb5db48ac5456 freetype-checksum = fb2d89421633a56d4276b192106d930c9f29cf2b70c52d9e13ba0c6a2679b6f69c20ab9f14480522992c263b493f198f24593bdeddb5806a4d801ab7604a5926 gdb-checksum = 4afd6660d2bbc4c48ce726062fb42217bdaa5974774407358b2624036a2ace48ed853746e4d6b83b9a409cd8fefc622b0727c79da8826e28e3d1444fa3a9d148 -ghostscript-checksum = 2d6422d434cbe6612d4e995022322e17d0167d0d92fe1f6fe122da73558a81fa43025458411e38363a0e30a573cbc6cf997ae7aa562501787d0af88bf1b5b387 +ghostscript-checksum = 516f7f3bf8650ff5cf5a913ac1d69c196dbf62de97b77fba75fad7cfbace5c978a4aff7da0ede3590f56d68b240398ee17a1a504bd8e8ee76a4879b0b3b6a8df ghostscript-fonts-gnu-checksum = d8de17e5d920bd3803ecdc07aca224fa5b9a26db847ddae0ecc79488dc3e867ab3155a8565f20df7855e5228bc7f36a549b082ba0b8b56c9b948d6cfb655fdc0 ghostscript-fonts-std-checksum = 40e01f88dd113c3120686c11da1c4cf4ee77c8db315b21cc936c3ccfb42cad4949e207298ec6d78d03327ff05122fcd75ac82d5e7cdc9698d4bbf1f72b19756a -gnuastro-checksum = d27399993ab65d1864c915fe107efb298c758e1f24e3f300830f8f120768dd3a54daea1d2af5537e9dd3b50a13a51edd237f434bfbb442ab992828b2dab1386a +gnuastro-checksum = f777f84d03cc106a12373416ec3617e4c598f7642f6ebf881fd59a3068c8b4cd5ba98195c1e154a4a37d3b22828b618e5ac70157c2775d11032b7e061880aa2d gperf-checksum = 854cdb24337f14b0d4199cc97c4a1d2ec7f953ad8125a47a932d93f79bdc067e2de231859ac0e842f14b1a8d80c5606fdc2c295a370df5e85ea15023b478805b gsl-checksum = 9a46b431eed536182f31e1ecad781ade214ac8bd885ac88bbfd02b6e08482c294775a529e561d358c1f0211d8d20db4ec99b448b9006c3bf7599c076d3fbbcc3 hdf5-checksum = 58d6d9a1c201efa0863d9c31d189b4d40b1a9b1fb1ab9cb3fdeb4463f508c043e467aa4d44484c2af111d4e2fb775223286da69568774e6edef285208290c67b -healpix-checksum = c11949e92aae9919fd41de91cbab72beffe2519c1203ab153ba217c6b6c81f2c10ceb07b683da0d04fb53ef5c80a9b208bdbb5379f379b7da9d0611d2430cc8e +healpix-checksum = eae764ede6b540d661619f4a1eacfb3cb4c6e3115bf696fdf45734ffa5c78bac1469dba7ad960bc6f082952dce8bbad54bc630ee7674afee1647cab400a750fb help2man-checksum = 83dca38c2020c85a66da882cd994b4e291eb6a0584149b7b3a74fec1444399ec5ecfc6296080fbc34071cd9f8ec5c931e249fde1f755c7f40930cecafb3efab0 icu-checksum = ab14b52a3fdf2dcde6b5160ab7218eac381b850d3c278324379741c49d71fa6040fbacca94c6937e6c9fc15843761121deff302ca6854da5ca1cd5b26a34e839 imagemagick-checksum = 2132614540b7422c9772fcebe7e8e358994efcfb53d8e48fa52992313b09b191847e395bad305322c377a4697014353bb8c15adc4edfd712e038504fc7f17c5e imfit-checksum = 15edd2349232c1c8e611b31d3a46b0700112d274515f54d0a0085bb4bfa6d3d5f8a15cd926516e043a29ce841accf3534ae58dbfb952d858dc9445199c957096 +jq-checksum = f83613b26ed6f4f963865bbc28fdc5246644e0e21287d0707ce62c5d10e84d70e622c7ca7f83147682e6052d96892697a2fe32a47d47bd3f32e3d74548877743 +lalsuite-checksum = 330f2576698bd3107621d49596da66c7629d314e5744b3a267b3fa346e7926bb255979506627abaeeba62ff9a776c6091e9150ab5c471c3b4a43bec40a1a5de1 lapack-checksum = ff670e194a1d8c998f05e6143e01a09e6b43176c511217ea3c77742afd9f2566251c50fc23aeb916442401f7118c1d1fe21f0172382a7f4f2c516c1d7d873e24 libbsd-checksum = 5c7d98474000af1271a36ab769e54aba41578e0b0f06e47af2986d6821b6586ac430ec04cc51b7836823834dd9d0aec9f4ab3af088b94f963b89729fa2cc95d8 libffi-checksum = 027416da7066bd36ac0c9ed8228693e5eeeb5e11dc17afcd7ed7faa66a9e57f505e94dffaf18f8978e6b40964938d4289aa3b5fe8778abe8fb97a68138b8b120 @@ -118,15 +119,18 @@ libpaper-checksum = 8532e5e95b53e0dcb2b2c4c161d3840a34087b5870d449950a853dc312a4 libpng-checksum = c4eff9a427302c6f228b93a5a6b74d1e667439ce77f20c086d91ec6efa932a0e7f5752b976f4af5cd07574c5e4999c86e2b9cae3e82cc448ee61f3d0dde9253d libtiff-checksum = 72ce12ce317ab2a5c188d3aeb575004eacf890dcb2729e9cb46f73c15cde2132624b4e229539af01626e59ea6a4a7380b4b5f90d5fecad787197b8b901615d20 libtirpc-checksum = bcb6b5c062c1301aa1246ec93ae0a5c1d221b8421126d020863517cb814b43ed038fb6c0c2faf4e68ff133b69abefe4f4d42bfc870671da6c27ca941a30b155a +metaio-checksum = efc7f052c550d1566123a8842d5f6237b330d8fa7b012b276117bb2587e4df08a603cdbf22149cad70e7725bacef57f5f396ab3a2965446b940a0665c12763fa metastore-checksum = ac1230686535a652e95024abaf6c5585bdab36f4e092bee5fa4deff2a913cd60f3a6bd8020c6887ccab97f0da3a284a0d4619fad5464a269e7b2040d6f7e6aaf missfits-checksum = 32727f5eb30573a1cedacb8900e2536867e4815059eee32e64e3db65be9291b8a91b9f45b2c9f3cf6fc2a8cc448012ea3d502bdd9dee516008e17d5086aee795 netpbm-checksum = aef81d2e46850fae1fb34a38fa9f634be3c47c7c4e80f300c61aadd5532bcbccdec636e6c701e54caca67ac2d6ce3a7b6a9f090deb00cd4db718439986f89d8e -ninjabuild-checksum = 3a6bfeef3ff1143d427a658d2d4119e4c9d85ebba37393760dd7a6dcce1c4a9e217009b6600b5a72338edb34680f0c45dc07b6b09afe1b0a9eb163993f2bc41a +ninjabuild-checksum = 381f63e6b9d2b6863a35ceedc3d07b81a6b41d7539af9b4c3fae859c34a0f4b68340e5579193abfc221b0e4f413dc20c7caaf70480147d93ca7ade497bd01571 openblas-checksum = c87485bb2b74cee175a6e631c63cb52fab174a2275d375c00680c4cbcd8d12a116223fda0349fc677cc9fe49794ce7b0342740855840df491b6141c8f2f4784a -openmpi-checksum = 88c73fd708dd5fe3a893d5517425a8a4c3e252fae4715df9a9b0f2311abe94cf5f71be92f153820fbaeb32c7d0ac0175d5165bdadcadc731427309102182c1bd +openmpi-checksum = 8480402a75c27e73ef51ca8410c6b527fccd62029aa45f36468e0c4dc0e5281f161a8a56237a1cc7e107ba7561c47acd63630e7bf8560e87bfd357c83c614607 openssh-checksum = e280fa2d56f550efd37c5d2477670326261aa8b94d991f9eb17aad90e0c6c9c939efa90fe87d33260d0f709485cb05c379f0fd1bd44fc0d5190298b6398c9982 +parallel-checksum = 612adb4da6304eebf303d40f620c5ffcdea57c7e67ef9616efecdc4fce28041257c955a5d3a35135eeebfa0446514731716e729be865038d4eee7262179d56e8 patch-checksum = 75d4e1544484da12185418cd4a1571994398140a91ac606fa08dd067004187dad77d1413f0eb3319b3fe4df076714615c98b29df06af052bb65960fa8b0c86bf pcre-checksum = abac4c4f9df9e61d7d7761a9c50843882611752e1df0842a54318f358c28f5953025eba2d78997d21ee690756b56cc9f1c04a5ed591dd60654cc78ba16d9ecfb +pcre2-checksum = f6c7b59fe0195411eb25c63ed67b8921845ba5d9ce56fe538f654d8030132073f2ad8b166d3014d1c78951339c7553203e22c6aa1c92a0751a88c9757a208f52 pixman-checksum = 333732b99994f7ea636d647e0b6123075351b27601b5b6370c9bc821a1ab3921386ddb92b51e015f3fc35104ba09be1e0d7bef47f0b4f73036b01d1d70396dd9 plplot-checksum = 2c5a36d84ebd948402c924d304427b5702bc75bdd22753c9b3b910b382dfdcc26910382aae452f02d86b25c1c813bbe1f40c9305de1d2809d575975f610fdf10 python-checksum = de940d7eceee69b8a04f12c613753f61db53ddac1c5cde5388dd342e88a09bae8da1f56b71b96f20997f6203d169e92e9e2558c43b38dbc323221d7bb362abfc @@ -135,14 +139,19 @@ rpcsvc-proto-checksum = c3011d7d7ef97a4a751f6921df1a23e1dd8ac50fb0690c759d37010e scamp-checksum = c7a192f357c6808acd16d1c5d2657c5c8c2f61e4fecb8a4b18d39b07f4c444f85d5029c21571d41ec3ba9c8d075de8bd660b0e029bf5dd63e3819e963430a938 scons-checksum = 0477038b014674049f12899b64584d44a85283d521b2422561e42020a5ae296a5af005684087c3ff410ed3fcbdcc5ff61998bc429eb29513f2a864138ffc4945 sextractor-checksum = 4035710f9b8a20a0bb1a3913dab2dadd8444c179bf6dee425e0e8bb66a772944ea189bfce89fd791d316a790fc4b2cb15a62633b19d1d5331b1803dec2e70af7 +sqlite-checksum = f27d1fc39d3505ce85892bd9d5ef284c7f6235a1ad473a2fb0ab8c221a3a450aa572d1a9e37eeea66ad615a7957e3326400eff5f51854a389e0f008d33879276 swarp-checksum = 810af6ea0a2dfc7b78b0f6aaa486b80c29f517b90c29f5037d05f31f761e96a55c8b5dcc259e1b435b25f45f01dac45ddba61da84000a69676cce119880bb05d -swig-checksum = ecb1d940f584c23df036a8f77288916003f861851dfdf836ddc5bc45b95312f9259150084191c1d9f086c006e64e12a4db22f7ea382fbb71667b811b8f99995d +swig-checksum = 7db62945be0514e1196750f6e00f48edfbad1524501f72d646706a6b3149bceeb904ba28dce73f8a98a8d9b9a91814492482aa88b996366b411e8723a9f4bf4d +tcl-checksum = e7e40a07d6c7d0a4c6a0055c36c12c1303167680768706f53f981aace063c802486d39a715dd393cdca0371757c34df2600642472ae36e7991d9bb70674da8d2 tides-checksum = c3360ff0d023b43749ba09a33302ca059f017a157b3ce7cdcf4f1a1578e90d3e7fa420077043adbee6b1ebf94bd698c8d6b279012f36d2a05b4de5351e30e108 -util-linux-checksum = 9d421976a344f982f2bd910f0c888a7df4370a1e367c7d2582d03a58cb8500f1fe65cf6a7164c492a9ebd76bff04560c344b3a0f63ecdb372bfea2379b383030 +tk-checksum = 87d431e4d995e78839fc5cf23ab8bebdd783adcf0c8ee1169cd7166aaf4884185ca3e72dd7c7ba6c55a3018d3529b036fde4dbde1c076a767fa5eb2668db5283 +tzdb-checksum = 6c0f7f24f08dae63900499ff55dca5775fb687d76451bfdb82ec24aa4bd29263fed5e0cc3efc32ea54eb2dde80cc17e4becdeb3d70b1f63646dd52a5538dc632 +util-linux-checksum = 3053377ca3e399a12ea86ba96ac38749ff15a42e76742ab4346eb2f96c8b24564077db44c5b173e8474c8770846dc355b1281a697ef5a168e239b1b6d693469c vim-checksum = 85d083fcf3638f2c3a049b88da46d569d6a250ae132f3821d440c07c4792befffc09e4235241ca96d7b0307a1bb96cd2222ac8a63fb41c6add8c2bdc6c17aabb -wcslib-checksum = cc72804611df1b0087faed2ec19edbc17352cc631559edc8ceb03792162837cdaa6bbf144897c69a58c9fab7e569a1958149be7eaea6552d72190406e4755d12 +wcslib-checksum = 68f40f268d4a4d6d052e444859070138f90f17fb1723cba2da88293b5f7a372885461a09da1ca0ba1686d08e7d731cae52e305b65a6f5edf13586f48a61dca55 xlsxio-checksum = 22870fda7bd4eefd5fea2a9ad7530c9049135129d9b69805091777e6b54b2fc6c3f0e69c6954f36bce54eebbfeccaf637cce9e271a593221a4296d6632470a6c yaml-checksum = dadd7d8e0d88b5ebab005e5d521d56d541580198aa497370966b98c904586e642a1cd4f3881094eb57624f218d50db77417bbfd0ffdce50340f011e35e8c4c02 +zip-checksum = 74966fda50ea18b28a3bd03e91be3a75a9034cd923149a321e4cff3f233d696c4079181a996f81fa62ba121c4beb6588c31dea682e174b18b858121b72409ea4 # Xorg related packages fontconfig-checksum = e71c684ffcc75015b1aa13918d6598a213b926630317e4babf876fc503e0444ed3211277080a08505cc26da033ee8e673bf5e246cb9d75d2fbe18a43160a0dad @@ -172,7 +181,8 @@ xtrans-checksum = 84f1142994f93882c9c68a4a85a218290c1c979162dbd4c5ee6b6becf0f96b # hash here. asn1crypto-checksum = 44d442a6ddfa971e31e24712fe084368356deb5e1c4c3b3e813e0910931860215bc1c4f9eb2c4bd4fdef607c324086c096e9357068646efd28c97f2d4f85c62f asteval-checksum = 4d64900b2f7dfdd098d6c8c102f9d9fd46f9ec265a54330e7d94479ba41f0ee0698855658e18b8b32b9c255159eb9a085af5f0306eb6508663d3fea7d2e00b4a -astropy-checksum = a91e327784a6ce0bb55119639e52037db0d802305dafedd46f27d9de96db819b43e336a4d43558a57e57459c4d1b7d6d3fe290771006e061d3779034f089aa9b +astropy-checksum = 9c32139b880934f6fd82c47cbdd201b2c934bcb93216136bf24d7d676f2acfbc81e08f35e7290fc1fb9c49ed2c0f4632f0b5182b86bffe6d26abe16b52f2783c +astropy-healpix-checksum = 685ec136eff28fd25ce67403f932297a8297e2de84d527890a81ddaa9a6bbbc70922a204c08225840b03d3c6be8af7476e01d4f87285ffa71b6d7d19493fb332 astropy-iers-data-checksum = b8bb1aacf67fdab241e95952919a8691db09488493d0e4ea6eb7a65822c6e4bfb7fc55d4ee2f6a0e9e563fb671d16b06474d38f0ad4d3b0f41d6dc341dc7ec7a astroquery-checksum = 43846791d8469a26cf6bb8819db58b830cfe50a34bc0091c2e843dd7dc78b1317530855d432a3a567a9f6a6f4d2682382a32edc91ea01716246b99b3625ec521 beautifulsoup4-checksum = bf8fd3e54da63a506f294f0e5f1201fd46bf2edcc2db23e99eda995313b8a8d24db3cd2d1903853c539a9320bfb9eb4b29e311772bfddc0d125f0b4aec71e384 @@ -180,7 +190,9 @@ beniget-checksum = 32a19d77323a0a21544ce7fbbb71cc5f4c66949dba280d81deb36f3836454 certifi-checksum = 6a6bf1ff98caefcdbf78a8c83e11e155368bacdd806f0ae0c6afa8f513667df6598e594b3584de61acdca3d6049f4a776937f2aa8672b602bd6db7b737f6074e cffi-checksum = 50e2b8215ddedfa8f1d569680cc0023cf061c2c52b7b86b6b26ced56e5107b362aaf6dc10a77b9dbbbfca2e5611f7cfb42c59501f9ab7da29fb3ad3ccf418cb4 chardet-checksum = 61a03b23447a2bfe52ceed4dd1b9afdb5784da1933a623776883ee9f297e341f633e27f0ce0230bd5fdc5fdb5382105ab42736a74a417ddeb9f83af57455dba5 +colorama-checksum = 0cc6b4d622a2a7bae2903bd8773182674bb8d9a654f9105231418b28f585aab30e0ed8c35aa0494ee1a81c024381bf90654febeb3b16eb78b5e1fdc421d9773c contourpy-checksum = dab72dc7b9c64b1c06674ac4c7ba4d1058bfa5922763087dbe9d30237fe38045201e3801423541cc65ef0cd6c92eb5f4ffb4b6ff8da21d79f2f23a36ddc4bcf3 +cosmodesi-mpytools-checksum = 0d25d990d405c7ab829ce27e274a158defa33d7c3fcabf6f15098903e0a51f6029271e167047427a529b01fd23ccdcffb060af53d6254f184fd2d0de365100af corner-checksum = ebd625ab1e4591b4c21d25ec706c35d37f560b727e1e0d6a79948c4a112ee6f21d3ca30162901a27715074e1345f3bdee1a0345c63e5fec24113e495fb094127 cppy-checksum = 562b41f22d2819fa7be65ad7d05b18afa9646999277df8a5e6399c69d66320ed0119a5d2653de5711ccaf65360d55fadd85611bf2f7c42348e2f74fbf4eed45e cryptography-checksum = 051b5007512521638981a8e975c7bbcb688b1a1c384babc81ba105c7bc2a7e776a8bf46af5939627528b2b57d99aaacf215a041018c47f6d3ae6d7a0e8eafccf @@ -190,62 +202,87 @@ eigency-checksum = 1e7cdfc43071da5edba30a0d32cd655442b516f15c166b049a195d151dec8 emcee-checksum = dd60aace8879525fd3fe42b747d82170b24b2ed21f538f9186ba96b9d04c084812e3303f5d2e04119dabd2f9d3286d510b4d4a5324c71dd24b1c7e5f0a9a0ac6 entrypoints-checksum = aa1274362d3a4b00266103319ca51aa266605b4999c89a9d0673eb61bfae9e646cb0ec6b86c95544493f6fe048385a2c7641d64adca8f45815546fb1e663c858 esutil-checksum = 7f1f22b0a05855819e3bd9bc2eee14942c2f536fb99f4af03575eb41d3d3fd1d2e6091ad076bfab4d51a8f30e213b607af35ed00d063fb2895b58f7a57cde5ae +exceptiongroup-checksum = 4ee3a8196deb7a3ef437a7d0e837313b89226ebadfd9fd294371f32b1bb35c8070236b30967b33fcfe0291728f4b9dc4b662c7eb2ac916253a2314dcf292d188 extension-helpers-checksum = 3a8d971e33542931754baf1626141b6407c11e98d58088f9898aaffcb2dcd9592adc094b236a7fdd74837b6aa4d38e7191763d7a414fb34d6e923985c7db3534 features-checksum = 91b88e02ebcccfaac9a9cf4b8f00e177880e119287514c781fd591c06231428f4f7aa479bb9ba74f59b2544f12ec08061486f75bf91aac8480d23bdc91e2c511 flake8-checksum = a25076bb7f978ca23af0f2016b30bf5fe4680371b4c0939361a6f8c496f3c846cf98ecadee35fc2dedd588116c8cd982529972dd2411ab139621912b69132a1e flit-core-checksum = 59912228fbab70435a438c978c2cfdf7c1fd4e14f45ea223c9b1cb58a56b86ef932b42e1fcc69cadf3fd5c8ca52d407a99428aa874d58029fcaeb2b6acc6bbd5 +flit-scm-checksum = 5f5e9ee88b62bab18bafa842d91181d46addd8dc874e64e0dae7d9a99ac2c92ce00bf92ba9be1aedb0464272a682b9dc58ee7322118710259b3a88861112f57a fonttools-checksum = 3bec5dbc51d93d439a9cfffdb45e1de4d1b79d6bd91c5510eaf4b21fed077648deb6c4858c95222f93bfe5da1f9c82cc57e8b3df67f214ee29425037f1261e22 future-checksum = c70565a660cf87c5e7e994bae0d4eb0b2b8b607ecb5ce65521c027bf0a39ce5699f0578413bd3c7edd5d01aeb1617de48dcea098a9e9021d8487a73007573030 galsim-checksum = 65393c4bcf52a3e723604a956aac5f58f490e41c7d5ddbd5d61660658738e2496aa30f632d4d83b765aaf1e46ac691fcd02fc827380e07c8f921c160a9f6a6fd gast-checksum = 8311f48788daf060bee54ad8986eae054c2a426adcf97f1503cde37c4da2827cd0f664188e51f5c2f932a0a4ded78166a8466465595a640106060c8b6c7072a5 gpep517-checksum = e937f18c9593c76c15f8b2e77143aa11a0a822b1ff686860d5539f4418b02ca7764d71dbef817e8fe8453f9bf3c45fc2f97af1244cc30068701347f584ace91f +hatchling-checksum = 863562fc9f74c7c3c48dd51c0d5e834c19bb9c65490affe0b9da057c8383fb9ec77f9db4e76034b828ae6405a80a15aac05ab545b8019be802e9ccfb7aee8410 +hatch-vcs-checksum = a9b1a1b445d6d997524023e987220be6dca2ef696f017987664a1900b67643d9685e32d9756a5a8efcad03fe98f7493cbd05bbf3f3527f46eeaf2855f047f2bd h5py-checksum = c49b04f7dcddf03f36d4f063d79ecbe544e0b1daee432d4a76cfa83dac3a1f2cb144f40a74fb85ea17cb5b778f57f709969ea5d1a2afc5bdd5aecbc9d732898d -healpy-checksum = installed-with-healpix +healpy-checksum = 46b78a6aee45e726b6ecd636ed9283612990e8057bfc08d37c77cb460f7869551b11ad7268f0ca420352b650c9008478e17c7bcd1039da2825c1ff1609868733 html5lib-checksum = 35939b4450893864da04e735ee5e0addacf1dd34bae6a6909c76572abf6bfded446a78a713dfde91c1485ba45867d7abeb6a45cf0545c16ea968707be7de5dd2 idna-checksum = 8ca5cfe6350c51250bafdac7c6e4ddd54c4a5d6bf7acbcef896760a759868c8e9df1fdf550121d8512fa3eb316dcf031ec6058e03b4f66eadee21b63e2187d33 jeepney-checksum = 43083994a7c6af84a5a68d3ff8f6dc4d9129ce9fa55517838fb62d9f62bb78bdf52067649d0b95d08d689b7d7475cb9b2a956662e265a776ad42dcf4ccc0ab63 jinja2-checksum = db19498dbc9bd6419b474fbb465ce8c689b96f72d33ed1a31cee3c42e2c545dfd0fd5bfca610075e2e126fbdd27300a73efd76fb7ea449727ed61a6127806dcd +joblib-checksum = 62f4dec7760395a18d52f7daccf99cd5cdc2545f8e7698c6224b35d13b24f7cd0e9505d80eb1a51362ae9eb8c0b31a81f9f3a1038164bbe2ff4fd99aea50f108 keyring-checksum = 3863f2cd89131d7e927e55691b4eb4b7c2599af189525293d0948aef5215efbbbd74d45db3cd4885ba5b18a79f0fa0b0edc3783a020a6702f6a6fb7ed86b2cf2 kiwisolver-checksum = baa1e6b8b31b8000e51cc9bdacd57be20c3030551782fd7989fd473b15d225f1cad6309fc846d2ba07d7a7abb729151260a873d6beafa46d881f4fdddcde0f3f +igwn-segments-checksum = 4da828b5feb93f42f79d1f39faccf5fc70fa70890bebfae077e2c8b62af78f70776f0687bd17377b080f19139bcb4bcdc18c3c902ba9e1fb0cc2e135f3134bee +iniconfig-checksum = 384fdffd648b07fe3623a26065d1dfaa763e121995dd14490acf845853b57b341cabb861355aff7d8c7c8ab5b21b69c2eb52ec203ea644564d4601f63d85326b +ligo-skymap-checksum = 9fb6204b3eb2bb44d26b476d166770c86b15b4d62a2f024f7b8c13188d0a4b5a682426ad4838a8f90e2467a535b3a7f398d3da6039425fff7d8a1242f76c46a9 lmfit-checksum = 8435328fd8baf92b424299c341aafb995dc45245e17b1774e9eaabdd994dd6ec3b2e70f7506d67b587631cebd2dc584fcd1a9902286925a61e2135b7e11ece21 +lscsoft-glue-checksum = 6082f26f12c16188cd216abfea580e4b23cc93a31e50cb3e0ea4c3875c544658498e4adf4f9d6b7da66f19685a88d2533dacbc3fd5aa1fe6984a31b37dd18cc9 lsstdesccoord-checksum = a66b8b702cd8122f0c8aee05abe0fc0c6299f8bdb99b151f54bd7ad7430d6bc9f843f7294a7f08f3ce9f8606ea0ced0796a63b79cc8a53ae73068b691e2e6871 markupsafe-checksum = 4bc1171f1dc86516d6002d6dc3df2ccf78481480812025114b3a1511779b152a605904f0dc3aa0a6c8c282111a74564891a2b02650f90c44b667e5e0622c62c0 matplotlib-checksum = 2911fa5bc27d082aac00131f012503572f21228e0da49622b57d1d1a53eed942fa73a3457ae09efd03d09cb80172e2c344eacbb85234287be085a749622f9505 meson-checksum = 6ba8af3a835f24b12e105b90948177964930274fed7d134405b3852cf333a4cbfc11d955778b87994cd9ece57a0de424483b40cf6e43ffe3fd6d9d249257f04c meson-python-checksum = 08ce6b89f4dd898ef4a92f9b358080aa8984047c3c8b1991347496a2e1283f357424d2d19a1cf5e03054e9dc0750773f2936f5eee2838523db2cedb783546477 -mpi4py-checksum = 68b1d4ffaab8425ed3eb6c4adf2c008a489d4086eba1dd042c29cba29c95f790d49db9c4bf2870c677f5d27e6e5103781a8113ade6e28fe6cea655487e5e07b5 +mpi4py-checksum = 14d8ebe9a97841d7af493c5d9ee8b5dfeef4639887904ff730d27c9f2650a9bf0bb182847ec3f089a20bf9ed32041dfa826780f3e21aaf24714138c505d13731 mpmath-checksum = 58c69a801f65d73cc0eeb0d2c79277ed638568c656e7213d06ab4709c218aac908b2752377139010bed0e91bbfff01d129c60835ff0a928ba1185aded6de7c0a +networkx-checksum = cce23a1c8ebb00dc1ff260e62f3706ff74a471ff7524520cc50519ba45e08ce4793569328e02e2ad5439fcd73dd8709e8a37118bdad11c084b794204bc10c22e numpy-checksum = 139ee0c38aa9acf4ddd6d5fd14b3c6c6ff3275b89cd3925ce5c528f58d9dd0e367edf77827685026fe10f686ff662bac95b367249891d4f2360fae8a60a8dec8 packaging-checksum = 0f0be603c444b20917c7a36e1356bf161a958d7326bc47198bb06d23283017e04f4b3be60eaec66a865bb9704c717342a2ef55c9069bd8ea48bd1810d1d424bb +pandas-checksum = 6d3fd74c240cdfff4ec25f5f5309e91120fc6eac70bf16fd994ad3d6702f055a5fd2acf53d03784267bf843faea97f660a378760e88bd3b319585aa74ddcc85e +pathspec-checksum = b9bb195a277a8e1138fb780f5b2c11c7355674a166d0074384dae0341c80547be4dc6e4db9a66e044c9fb1f4d6de878727283d768316957fbd4e6f82f6e8f3ee pexpect-checksum = 4cea4229332c1f3de26dfcad596877665b3c02e91d51ca3c45c1f9b44462adb7c82abc7b76eb09a73822c2d1ccc9d812574cf79bf6bc8fb0b7d2f1093962cc3f pillow-checksum = 5134db8c35fa66c559a16e7aa3f341e00c7d21e90d4906f5d95f3c912d804aa10fc8d8ca889677299d1f5431b421c4963d3420a614c7587bb43e2d7079278913 pip-checksum = c6e13da3a57462371d32982c80575c5181592f5c6a8e70d60ec879e689442f4ad468e7aef97eb58c9da50a5a770385aa35e701eefd713a8e9fafeb12e11d956b +pluggy-checksum = 3b08f695c2a8197c07352c3398bab80e38eecb7a38f5a1d87fa96e48924137cf5412ca49b505cd259788a502cd89b90a436586e7cffd10f3ea61dcc4680274b4 ply-checksum = 29d3c4de42a74497083dc6780fbfccd319bd3be8098a2a59ea733988a159e760bdd0eead3482fda33a4e36d2b9b88916f9f9408db2fd3a6b7a0c51269b024485 pybind11-checksum = b877d8d66b2c47b076224bf8e8414fe889f4fc3281c440d6d097edfad0b7a8edd9fb15e04cc60932358232ebe6c14b89863f4736aa43edfd739f4605b6fa6f44 pycodestyle-checksum = 84e751a7d00048393b02ca743de5d71d1641e948ee1b4daebbdf2d07e0cd8f087ca4e81f826061114b40ef41920bbcd680c9f479e7cc1a159a70188425717208 pycparser-checksum = 7f830e1c9066ee2d297a55e2bf6db4bf6447b6d9da0145d11a88c3bb98505755fb7986eafa6e06ae0b7680838f5e5d6a6d188245ca5ad45c2a727587bac93ab5 pyerfa-checksum = 44a1d3d714a72b9aa66c9f72f692ff46b273b25d3af2b05ddced62305727f23891b6c63a2ff85fd99f2e2d1663da6da995fef0091c541523867100c72c6a18ca +pyfftw-checksum = f6d23145e3f4a9a18b2f7a6a32480dadb53c2d5686fab9dfe4d7c44d6cd3f849ada6c8302593bdbee8cdce819577810d1fa6c79f8872314b0b4e3c94bf9b5d22 pyflakes-checksum = 7ebf5843b38146305c1063e070480fea8ec3b47fa1be546b1fafaeb242a688a5a001f978e7257fd71d5905b9a338b466ef17c7330725191587e9c40ba632c3f8 +pygments-checksum = 6092d2922c44889df158e08c2c7e9b755ae689689b45661bcc544e60265e93803b63cfc55cc348d7a826ad3535212f60fac1e6f000b276c114a25bca489d6923 pyparsing-checksum = 9d10454faaa0608eeb739a36a001f76d433c8f2ac9c7f40e9d1de5a9431b99e66a00fbd1618901d2e938ff36f752b77fc6782328ec08413f6d018bf6e5a420e7 pypkgconfig-checksum = 1fd9aa973bd20a8fab864722598f1d19b94c23c7f2b522556b3182b19fe016bda7aa2be5e48a1b8fefa70a069611007d6d790e24defcb462e4594a382de85b00 pyproject-metadata-checksum = ff36bf73b6d10331d0b58af473a6af4fb4c67b7d2f4d567c36dc75030b6765e4640261041497b7660ce538a5a5baf11cd24acad6752ff45179b50162987a4005 +pytest-checksum = bccaa8b8081cc4a1630c5683a6aed7b33a19184982e64dadf7a4fa4c626f56f78c5d4e71479087f3259e8fae8c44f0e705f8703cc948cd54069825a0c6b7d4cf python-dateutil-checksum = a329abd60db1b3f2ab4cd516c1728c3596831ba7f7ba21003da7a39f7fe571ef38338125a210dc2931a0f1aae34b15f942be148095e0ec0d9a4c10389673c388 python-installer-checksum = df34826ffc192547e5c87ff03e33e6a431d98eb6b9af6d7783963795204318fb69dce8ff49b757379e88ae6163c4802e43f58213f33b19d1d257bcdebe1d5e27 +igwn-ligolw-checksum = 0f76be08be9e85588705d31d96418d74e581f2ff45dbcd57dddbb3326e6d33237596fa4aba4ff4e5c5e6371c3ca800e543e1de50947286ca6d82b9ffd1aff55b +python3-fitsio-checksum = 36089e9cefca372f5e26172a598259591bece7a4a70e46c55902432e6ea2d425e0e383f637fcc2b5f58acff5a5b0153a322951b645db9f375749740fabca010b pythran-checksum = 5d2d14480149b044ffc159990d2a9b02a236c54c348a899929f11637d0545fee7bb16806660aab6e096a22e1f2e220e344084e5d9c96b2318e51c16767b26c59 pyyaml-checksum = 3f19057713a73586fdc73d54a6bd96400bce4edb4939381cbbe4be9bb3294dbc124403f16d880a4d21442df14ea4354231489cc2e97103360407f3936d41781b +pytz-checksum = 4b00a7651cbe21f7ae62b188cff0d401d045aa8f293783279aa16916f855327256b5594a5027ed74be9a6b1ad0369b805ccda0948b5a2bfd8cb4a4b5b37a8ca1 requests-checksum = 385e1d80993a21c09e7c4682500ca8c24155962ba41ecd8e73612722b2ff6618b736e827fc48ad1683b0d2bc7a420cfe680f5107860aca52656ef777f1d60104 scipy-checksum = fe7c3ac114190782ebf2b155182597a78f36fc24a5de841380957d8f78faf865f3cb6b4feff2f601547240a7d09cf7756b7f9f0eb87d40e3a805e4dbc0714bb4 secretstorage-checksum = 295e0f75c772edf153480730dc20051f06e60e040a18f169620cb4aaa37b8c1a254a496464d3794a38cf49fa884a7b561fe364816f0d12a5b2622c77259f03cf semantic-version-checksum = c0a354ade9761168a7e3a22b5edb10fab21abe527996266b02e1dc4d631b7f80440fc58a04c71708c1dc24317cb880f9629bff926b55517f798b680c8eb207e1 -setuptools-checksum = 9c1f68e63fda8e65654f9b04423f35e20c1a854cc12d012a3cc9d48bf14aff0bfff37e038dc98675cc246997cfc33af6ee80bde36fd96b58c7cdf917c68a5c67 +setuptools-checksum = fa112ba12b2323c909fd381b3bc9580f71284c9cbeefe194d288b51b39f327dac464c09925ec1354af41146ee08f7942fb18a2b4adf5a658637aa8a4c9924843 setuptools-rust-checksum = fb440e1c619c1b50b67686cf8fc214c88beb01b189f06d43848a31dac910c9c6a8bf915a621299371d8a5148469665f6c2704c4853d218f5dc52fff2b014b6f2 setuptools-scm-checksum = c00990fd7ea58fc246874c8216e746c2914a6adac2780e7596382a771cfb48a65a0114e04eae0ec12f44dbfb79ccb29a35ebc3c46a03caaa7399b57cd79c5e64 +scikit-base-checksum = add6d52875c8faa3409f234b80a07fcc31366968c5b66e8b90e24df3041605958e1e61bca30cd5d7409b6ae03593b202127ba10a4aa08bff9d674b9ec2788252 +scikit-learn-checksum = c68f1680884920b71c5fe2129b8e044e7999ed01cf2f24d1f5b844f2226c4eb56f49945559e27840f0ed7000243ca086ff9a401b1fca192bde680c884edc2ef0 sip_tpv-checksum = 5e03279cb3650dd506332dfcb31aa4a20f23f55b8a29fd18da5c6d422d1b7dc49e12362ceae2ff7417c874401b5e87a73ca1ac0f3c8747c8984e4269cad56c3f six-checksum = abc5da98a94fc20b8f4edbd881497ebdd5d86f30867d7838bfe388fc35ff415b8973884e24509ddda7de1b4eb8af6ac5fe22e6408826b9297b217d3c53a5a6bb soupsieve-checksum = abdcbb6a13563e7afadd3056141587fdc3d7d644e346f789bca0a16242d860219e462491b0c624b287300af960fb8e3f85c79f5137580939a9fc8c3d6961478c sympy-checksum = 6ae09be7260b1624b4f92d39c68d5cdf54e6e33010d9215f46d62d989c04cdbee6f9f9c8b11ebeda53257d154954fb926b3ab7335b738e33ad248764875b6ddb uncertainties-checksum = 5f09e75db2e4b2838858f2640a8427cd01cf691a6bec66e58f75125cb80515e2a7c164ef6d414dab2cb175a2cc696f42259729d0ebe4b4603f98e3f3ef9e262d urllib3-checksum = 4c12d08076b0f260727d5aac780f5e9a24e0164755ff05b02a1f5a697876741ff13ba278fdd6e46ef678e8e1146bc39de1fc49ee10ee839229a70540a9424a99 +threadpoolctl-checksum = f9c979bde7eb5a70e81f7e97797e2b2260e3b7fd72cf5e1f2de9eddbcd70b96a533404fdc9688011988d78c7066d2388a0367d2383c5c5a608902ff0d84add8a +tqdm-checksum = 6a94ded7ea339e0308816626aaab80337718e7176d0e402aa91fbdd2e61f4310dbdf0bb79013b76c7681ff38bfce78746b8d812866176e2f56f56ea9027d9b60 +trove-classifiers-checksum = fa178e060a14e1df95400eecb1f430354b4809b96c214eb104de33c77d95ebabbe70ab30546e331a7fbc33a3db0931c3dc6cb4283ad33249ea2bd549cdd0a5a9 +versioneer-checksum = 17bcb8d9029dbdff18b07d10d14ce10ebfb220fb04fa7ea057f0ba1e3923b756f3752ceba3952673dabbc9d563e99cbd3455cf32a4db3f3240ba2a9a34259d7d virtualenv-checksum = 3306f59bbcb48ceec225c07083e0b6831379b3e632e4a23c376849559449de1a04db66e0e7ceabb40ddcd3ae984a2a18dfdc4f1c38777d4bc04537f85a0137ac webencodings-checksum = b727b01bac6ec79bca517960d27b4c0668b295f25559471b9641c2c33dab55db6dac9c990952177964c6418382c22831b14d57df5e632d51d7abf97b61f24326 wheel-checksum = df45f00e9eaeae2f27f813f31591590c961da2f6bff15bba6fb2a14d529c221f39b29894b8da408fe49cd4b760840a0e05c4baef377ccfacd9983c0bba83d6d8 diff --git a/reproduce/software/config/numpy-scipy.cfg b/reproduce/software/config/numpy-scipy.cfg index c3cea11..40f672c 100644 --- a/reproduce/software/config/numpy-scipy.cfg +++ b/reproduce/software/config/numpy-scipy.cfg @@ -7,8 +7,8 @@ # appropriate sections. Not all packages will use all sections so you # should leave out sections that your package does not use. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/software/config/servers-backup.conf b/reproduce/software/config/servers-backup.conf index f0e040e..bb3f8db 100644 --- a/reproduce/software/config/servers-backup.conf +++ b/reproduce/software/config/servers-backup.conf @@ -3,12 +3,13 @@ # is irrelevant). Note that this is not a to be read as a variable but will # be parsed as a list. # -# Copyright (C) 2020-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2020-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and # this notice are preserved. This file is offered as-is, without any # warranty. -http://gitlab.cefca.es/maneage/tarballs-software/-/raw/master/ +http://gitlab.cefca.es/maneage/tarballs-software/-/raw/master +http://archive.org/download/maneage-tarballs http://gitlab.com/maneage/tarballs-software/-/raw/master http://git.maneage.org/tarballs-software.git/plain diff --git a/reproduce/software/config/software_acknowledge_context.sh b/reproduce/software/config/software_acknowledge_context.sh index deae01b..e2c7bb7 100755 --- a/reproduce/software/config/software_acknowledge_context.sh +++ b/reproduce/software/config/software_acknowledge_context.sh @@ -10,8 +10,8 @@ # your project to make a smoothly readable English text. Afterwards, please # feel free to modify them as you wish. # -# Copyright (C) 2021-2025 Boud Roukema <boud@cosmo.torun.pl> -# Copyright (C) 2021-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2021-2026 Boud Roukema <boud@cosmo.torun.pl> +# Copyright (C) 2021-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This script is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the diff --git a/reproduce/software/config/texlive-packages.conf b/reproduce/software/config/texlive-packages.conf index e68a771..7210fd5 100644 --- a/reproduce/software/config/texlive-packages.conf +++ b/reproduce/software/config/texlive-packages.conf @@ -4,8 +4,8 @@ # name to this variable (you can check in 'ctan.org' to find the official # name). # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2022-2025 Boud Roukema <boud@astro.uni.torun.pl> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2022-2026 Boud Roukema <boud@astro.uni.torun.pl> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/software/config/texlive.conf b/reproduce/software/config/texlive.conf index 0babd69..56f1129 100644 --- a/reproduce/software/config/texlive.conf +++ b/reproduce/software/config/texlive.conf @@ -2,7 +2,7 @@ # # installdir: Install directory # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and diff --git a/reproduce/software/config/urls.conf b/reproduce/software/config/urls.conf index 8bf7b23..8010c9d 100644 --- a/reproduce/software/config/urls.conf +++ b/reproduce/software/config/urls.conf @@ -3,8 +3,8 @@ # If un-commented the URLs of this file will be used. Otherwise default # servers (primarily on 'zenodo.org') will be checked for the tarball. # -# Copyright (C) 2020-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2020-2025 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2020-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2020-2026 Raul Infante-Sainz <infantesainz@gmail.com> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and @@ -25,7 +25,7 @@ #curl-url = https://curl.haxx.se/download #dash-url = http://gondor.apana.org.au/~herbert/dash/files #diffutils-url = http://ftp.gnu.org/gnu/diffutils -#file-url = ftp://ftp.astron.com/pub/file +#file-url = http://ftp.astron.com/pub/file #findutils-url = http://ftp.gnu.org/gnu/findutils #flock-url = https://github.com/discoteq/flock/releases #gawk-url = http://ftp.gnu.org/gnu/gawk @@ -35,12 +35,12 @@ #gmp-url = https://gmplib.org/download/gmp #grep-url = http://ftp.gnu.org/gnu/grep #gzip-url = http://akhlaghi.org/src -#isl-url = ftp://gcc.gnu.org/pub/gcc/infrastructure +#isl-url = http://gcc.gnu.org/pub/gcc/infrastructure #less-url = http://www.greenwoodsoftware.com/less/ #libiconv-url = https://ftp.gnu.org/pub/gnu/libiconv #libtool-url = http://ftp.gnu.org/gnu/libtool #libunistring-url = http://ftp.gnu.org/gnu/libunistring -#libxml2-url = ftp://xmlsoft.org/libxml2 +#libxml2-url = https://download.gnome.org/sources/libxml2 #lzip-url = https://download.savannah.gnu.org/releases/lzip/ #m4-url = http://akhlaghi.org/maneage-software #make-url = http://akhlaghi.org/src @@ -52,7 +52,6 @@ #patchelf-url = https://github.com/NixOS/patchelf/releases/download/$(patchelf-version) #perl-url = $(shell echo https://www.cpan.org/src/$$(echo $(perl-version) | sed -e's/\./ /g' | awk '{printf("%d.0", $$1)}')) #pkgconfig-url = http://pkg-config.freedesktop.org/releases -#podlators-url = https://www.eyrie.org/~eagle/software/podlators/ #readline-url = http://ftp.gnu.org/gnu/readline #sed-url = http://ftp.gnu.org/gnu/sed #tar-url = http://ftp.gnu.org/gnu/tar @@ -94,6 +93,9 @@ #expat-url = https://github.com/libexpat/libexpat/releases #fftw-url = https://fftw.org #flex-url = https://github.com/westes/flex/files/981163 +#framel: 2025-05-23 from https://software.igwn.org/lscsoft/source/framel-8.47.3.tar.xz; +# see also https://git.ligo.org/virgo/virgoapp/Fr +#framel-url = https://software.igwn.org/lscsoft/source #freetype-url = https://download.savannah.gnu.org/releases/freetype #gdb-url = http://ftp.gnu.org/gnu/gdb #ghostscript-url = https://www.ghostscript.com/releases/gsdnld.html @@ -109,6 +111,7 @@ #icu-url = https://github.com/unicode-org/icu/releases #imagemagick-url = https://download.imagemagick.org/ImageMagick/download #imfit-url = http://www.mpe.mpg.de/~erwin/resources/imfit +#jq-url = https://jqlang.org/download #lapack-url = http://www.netlib.org/lapack #libbsd-url = http://libbsd.freedesktop.org/releases #libffi-url = https://github.com/libffi/libffi @@ -140,13 +143,16 @@ #scamp-url = http://akhlaghi.org/maneage-software #scons-url = https://sourceforge.net/projects/scons/files/scons/$(scons-version)/scons-$(scons-version).tar.gz/download #sextractor-url = http://akhlaghi.org/maneage-software +#sqlite-url - https://sqlite.org #swarp-url = https://www.astromatic.net/download/swarp #swig-url = https://sourceforge.net/projects/swig/files/swig/swig-$(swig-version) +#tcltk-defaults - 2025-05-24 - repacked from Debian/testing https://packages.debian.org/source/testing/tcltk-defaults #tides-url = http://akhlaghi.org/maneage-software +#tzdb-url = https://www.iana.org/time-zones #util-linux-url = https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/v2.40 #valgrind-url = https://sourceware.org/pub/valgrind #vim-url = ftp://ftp.vim.org/pub/vim/unix -#wcslib-url = ftp://ftp.atnf.csiro.au/pub/software/wcslib +#wcslib-url = https://www.atnf.csiro.au/computing/software/wcs/wcslib-releases #xlsxio-url = https://github.com/brechtsanders/xlsxio/archive #yaml-url = pyyaml.org/download/libyaml @@ -195,8 +201,10 @@ # But be sure to send us a link to the tarball so we upload it to Maneage's # backup servers. #astropy-url = https://files.pythonhosted.org/packages/45/12/a1c582b3f9df5e2680eae0ed82c8057ae06d6130a1f7a5c0770fa899737e +#astropy-healpix-url = https://files.pythonhosted.org/packages/5c/8a/dfd71e4db13706383ca2cea0b2831eb10da2c286d4494f183f80fc03cbc2 #astropy-iers-data-url = https://files.pythonhosted.org/packages/a1/4d/b9511aba29d4330437497166a7049ab9bac53e344c54e44a35390724ca37 #contourpy-url = https://files.pythonhosted.org/packages/25/c2/fc7193cc5383637ff390a712e88e4ded0452c9fbcf84abe3de5ea3df1866 +#cosmodesi-mpytools = https://github.com/cosmodesi/mpytools/archive #cppy-url = https://files.pythonhosted.org/packages/45/ed/b35645a1b285bce356f30cc0fe77a042375c385660ccd61e0cdc4c1f7c44 #cycler-url = https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8 #cython-url = https://files.pythonhosted.org/packages/5a/25/886e197c97a4b8e254173002cdc141441e878ff29aaa7d9ba560cd6e4866 @@ -205,7 +213,9 @@ #flit-core-url = https://files.pythonhosted.org/packages/bc/18/b9b81cab2b8f63e6e7f72e1ba2766a0454fcd563e7a77b8299cb917ba805 #fonttools-url = https://files.pythonhosted.org/packages/1c/8c/9ffa2a555af0e5e5d0e2ed7fdd8c9bef474ed676995bb4c57c9cd0014248 #gpep517-url = https://files.pythonhosted.org/packages/1c/53/e6ad6aa7036d62ac009986e1e5180465353344afd958ddc8ac15e3b632bc +#healpy-url = https://files.pythonhosted.org/packages/a4/db/4d1e7fcd429151edb1e2c6d10365f86de217097e276c0b44dbc197028d21 #kiwisolver-url = https://files.pythonhosted.org/packages/82/59/7c91426a8ac292e1cdd53a63b6d9439abd573c875c3f92c146767dd33faf +#ligo-skymap-url = https://files.pythonhosted.org/packages/f5/f5/fa82a88c104a9d1fcf863e05a56b213b512db05bc72b5c4f1c7c7d3bc118 #matplotlib-url = https://files.pythonhosted.org/packages/68/dd/fa2e1a45fce2d09f4aea3cee169760e672c8262325aa5796c49d543dc7e6 #meson-url = https://files.pythonhosted.org/packages/02/98/bbcaf6caaaa0510a68834f119ac793a8abade6ff827fc2791eeb6f8b4a66 #meson-python-url = https://files.pythonhosted.org/packages/67/66/91d242ea8dd1729addd36069318ba2cd03874872764f316c3bb51b633ed2 diff --git a/reproduce/software/config/versions.conf b/reproduce/software/config/versions.conf index 9c82c8d..552043f 100644 --- a/reproduce/software/config/versions.conf +++ b/reproduce/software/config/versions.conf @@ -1,8 +1,8 @@ # Versions of the various dependencies # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> -# Copyright (C) 2022-2025 Pedram Ashofteh Ardakani <pedramardakani@pm.me> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2022-2026 Pedram Ashofteh Ardakani <pedramardakani@pm.me> # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice and @@ -17,42 +17,36 @@ # -------------------------------------------------------------- # # CLASS:BASIC (important identifier for 'awk'; don't modify this line) -bash-version = 5.2.37 -binutils-version = 2.43.1 +bash-version = 5.3.9 +binutils-version = 2.45.1 bison-version = 3.8.2 -coreutils-version = 9.6 -curl-version = 8.11.1 -dash-version = 0.5.12 -diffutils-version = 3.10 -file-version = 5.46 +coreutils-version = 9.11 +curl-version = 8.17.0 +diffutils-version = 3.12 findutils-version = 4.10.0 flock-version = 0.4.0 -gawk-version = 5.3.1 -gcc-version = 14.2.0 -gettext-version = 0.23.1 -git-version = 2.48.1 -gmp-version = 6.3.0 -grep-version = 3.11 -gzip-version = 1.13 +gawk-version = 5.3.2 +gcc-version = 16.1.0 +gettext-version = 1.0 +git-version = 2.52.0 +grep-version = 3.12 +gzip-version = 1.14 help2man-version = 1.49.3 isl-version = 0.27 -less-version = 668 +less-version = 685 libiconv-version = 1.18 libtool-version = 2.5.4 -libunistring-version = 1.3 -libxml2-version = 2.13.5 +libunistring-version = 1.4.2 +libxml2-version = 2.15.1 lzip-version = 1.25 -m4-version = 1.4.19 +m4-version = 1.4.21 make-version = 4.4.1 mpc-version = 1.3.1 -mpfr-version = 4.2.1 -nano-version = 8.3 -ncurses-version = 6.5 -openssl-version = 3.4.0 -perl-version = 5.40.1 -pkgconfig-version = 0.29.2 -podlators-version = 6.0.2 -readline-version = 8.2.13 +mpfr-version = 4.2.2 +nano-version = 9.0 +openssl-version = 3.6.0 +perl-version = 5.42.0 +readline-version = 8.3.3 sed-version = 4.9 tar-version = 1.35 texinfo-version = 7.2 @@ -60,7 +54,6 @@ unzip-version = 6.0 wget-version = 1.25.0 which-version = 2.23 xz-version = 5.6.3 -zip-version = 3.0 zlib-version = 1.3.1 # Version-dependent build @@ -90,6 +83,32 @@ certpem-version = 2025-02-10 # supported. patchelf-version = 0.13 +# Dash +# ---- +# +# Dash 0.5.13.1 causes the build of pkg-config to put the building of +# Maneage within apptainer to go into the background! Forcing the user to +# run 'fg' to continue the build. When updating to a later version, be +# careful about this: that it does not cause this issue. +dash-version = 0.5.12 + +# Not working with C23 +# -------------------- +# +# As of GCC 15.1, the default C standard has been changed from C17 to C23 +# and the following software cannot be built with C23. So we have added +# '-std=$(GNU_STD_OLD)' to the CFLAGS environment variable in their build +# rules. A test is done in 'configure.sh' to see if the host compiler has +# 'gnu17' and if not, to use 'gnu99'. After updating a package's version +# (and if you have GCC 15.1 or later) first remove '-std=$(GNU_STD_OLD)' +# and then try the build. If it works, move the software package back up to +# the main list before making a commit. +gmp-version = 6.3.0 +file-version = 5.46 +ncurses-version = 6.5 +pkgconfig-version = 0.29.2 + + @@ -112,24 +131,27 @@ automake-version = 1.17 boost-version = 1.87.0 cairo-version = 1.16.0 cdsclient-version = 3.84 -cfitsio-version = 4.5.0 +cfitsio-version = 4.6.3 cmake-version = 3.31.5 eigen-version = 3.4.0 emacs-version = 28.1 expat-version = 2.6.4 fftw-version = 3.3.10 flex-version = 2.6.4-410-74a89fd +framel-version = 8.47.3 freetype-version = 2.13.3 gdb-version = 11.1 ghostscript-fonts-gnu-version = 6.0 ghostscript-fonts-std-version = 8.11 -ghostscript-version = 10.04.0 -gnuastro-version = 0.23 +ghostscript-version = 10.06.0 +gnuastro-version = 0.24 gperf-version = 3.1 gsl-version = 2.8 +healpix-version = 3.83-2024Nov13 icu-version = 70.1 imagemagick-version = 7.1.0-13 imfit-version = 1.6.1 +jq-version = 1.8.2 libbsd-version = 0.11.3 libffi-version = 3.4.7 libidn-version = 1.42 @@ -140,14 +162,17 @@ libpaper-version = 1.1.29 libpng-version = 1.6.46 libtiff-version = 4.7.0 libtirpc-version = 1.2.6 +metaio-version = 8.5.0-a5da19d metastore-version = 1.1.2-23-fa9170b missfits-version = 2.8.0 -ninjabuild-version = 1.12.1 +ninjabuild-version = 1.13.2 openblas-version = 0.3.29 -openmpi-version = 4.1.1 +openmpi-version = 5.0.10 openssh-version = 8.0p1 +parallel-version = 20260322 patch-version = 2.7.6 pcre-version = 8.44 +pcre2-version = 10.45 pixman-version = 0.40.0 plplot-version = 5.15.0 r-cran-version = 4.1.2 @@ -155,15 +180,19 @@ rpcsvc-proto-version = 1.4 scamp-version = 2.10.0 scons-version = 3.0.5 sextractor-version = 2.25.0 +sqlite-version = 3.53.0 swarp-version = 2.41.5 -swig-version = 4.0.2 +swig-version = 4.3.1 +tcl-version = 9.1a1 tides-version = 2.0 -util-linux-version = 2.40.4 +tk-version = 9.1a1 +tzdb-version = 2026a +util-linux-version = 2.41.3 valgrind-version = 3.18.1 -vim-version = 9.0 -wcslib-version = 8.4 +wcslib-version = 8.5 xlsxio-version = 0.2.21 yaml-version = 0.2.5 +zip-version = 3.0-15 # Xorg packages fontconfig-version = 2.16.0 @@ -181,8 +210,12 @@ xcb-proto-version = 1.17.0 xorgproto-version = 2024.1 xtrans-version = 1.5.2 -# Version-dependent build -# ----------------------- +# Not working with C23 (similar to same topic in basic): future versions +# may have fixed this problem, so when updating, first remove '-std=gnu17' +# and if it builds without crashing, put this back in the normal list. +vim-version = 9.0 + +# Version is in the build recipe lapack-version = 3.8.0 libgit2-version = 1.9.0 @@ -202,6 +235,17 @@ minizip-version = $(zlib-version) # version 1.12.0. hdf5-version = 1.13.1 +# The lalsuite tarball within Maneage is based on the upstream commit +# 678514ed of 21 May 2025 of https://git.ligo.org/lscsoft/lalsuite. But +# instead of being a 950 Mb '.tar.gz' file including the '.git/' directory +# (or a 320 Mb '.tar.gz' file without the '.git/' directory), our tarball +# is a 25 Mb '.tar.lz' file (a factor of 30 to 10 times smaller). The main +# difference (apart from the extra compression provided by lzip) is that +# this light tarball does not include the big test data files. A +# description of how this tarball was made and what files have been removed +# is in the 'README-Maneage-LightWeight.md' file that is within the +# tarball. +lalsuite-version = v7.26-15-g1e4c5e5080 @@ -216,74 +260,77 @@ hdf5-version = 1.13.1 # URL in 'reproduce/software/make/python.mk'. # # CLASS:PYTHON-START (important identifier for 'awk'; don't modify this line) -asn1crypto-version = 0.24.0 -asteval-version = 0.9.16 -astropy-version = 7.0.1 +astropy-healpix-version = 1.1.2 astropy-iers-data-version = 0.2025.2.24.0.34.4 +astropy-version = 7.1.0 beautifulsoup4-version = 4.10.0 -beniget-version = 0.4.1 -certifi-version = 2018.11.29 cffi-version = 1.15.0 -chardet-version = 3.0.4 +colorama-version = 0.4.6 contourpy-version = 1.3.1 -corner-version = 2.0.1 +cosmodesi-mpytools-version = 1.0.0-04369fc9 cppy-version = 1.3.1 cycler-version = 0.12.1 cython-version = 3.0.12 -eigency-version = 1.77 -emcee-version = 3.0.1 -entrypoints-version = 0.3 -esutil-version = 0.6.9 +exceptiongroup-version = 1.3.0 extension-helpers-version = 1.2.0 features-version = 0.5.12 flake8-version = 3.7.8 flit-core-version = 3.11.0 +flit-scm-version = 1.7.0 fonttools-version = 4.56.0 -future-version = 0.18.1 galsim-version = 2.3.5 -gast-version = 0.5.3 gpep517-version = 16 h5py-version = 2.10.0 +hatch-vcs-version = 0.5.0 +hatchling-version = 1.27.0 +healpy-version = 1.18.1 html5lib-version = 1.0.1 -idna-version = 2.8 -jeepney-version = 0.4 +igwn-ligolw-version = 2.1.0 +igwn-segments-version = 2.1.0 +iniconfig-version = 2.1.0 jinja2-version = 3.0.3 +joblib-version = 1.5.0 kiwisolver-version = 1.4.8 +ligo-skymap-version = 2.4.0 lmfit-version = 0.9.14 +lscsoft-glue-version = 4.1.0 lsstdesccoord-version = 1.2.0 -markupsafe-version = 2.0.1 -meson-version = 1.7.0 meson-python-version = 0.17.1 -mpi4py-version = 3.1.3 -mpmath-version = 1.1.0 +meson-version = 1.7.0 +mpi4py-version = 4.0.3 +networkx-version = 3.5 numpy-version = 2.2.3 packaging-version = 24.2 -pexpect-version = 4.7.0 +pandas-version = 2.2.3 +pathspec-version = 0.12.1 pillow-version = 11.1.0 pip-version = 19.0.2 -ply-version = 3.11 +pluggy-version = 1.6.0 pybind11-version = 2.13.6 -pycodestyle-version = 2.5.0 -pycparser-version = 2.19 pyerfa-version = 2.0.1.5 -pyflakes-version = 2.1.1 -pyproject-metadata-version = 0.9.0 +pyfftw-version = 0.15.1 +pygments-version = 2.19.2 pyparsing-version = 3.2.1 -pypkgconfig-version = 1.5.1 +pyproject-metadata-version = 0.9.0 +pytest-version = 8.4.1 python-dateutil-version = 2.9.0.post0 python-installer-version = 0.7.0 +python3-fitsio-version = 1.3.0 pythran-version = 0.11.0 +pytz-version = 2025.2 pyyaml-version = 6.0.2 requests-version = 2.21.0 -setuptools-version = 75.8.0 +scikit-base-version = 0.12.2 +scikit-learn-version = 1.6.1 setuptools-scm-version = 8.2.0 +setuptools-version = 80.8.0 sip_tpv-version = 1.1 six-version = 1.17.0 -soupsieve-version = 1.8 sympy-version = 1.4 -uncertainties-version = 3.1.6 -urllib3-version = 1.24.1 -webencodings-version = 0.5.1 +threadpoolctl-version = 3.6.0 +tqdm-version = 4.67.1 +trove-classifiers-version = 2025.5.9.12 +versioneer-version = 0.29 wheel-version = 0.45.1 # Matplotlib and Scipy are big packages that may have difficulty resolving @@ -293,26 +340,11 @@ scipy-version = 1.15.2 # CLASS:PYTHON-END (important identifier for 'awk'; don't modify this line) -# Special Python modules: -# -# Healpy: When any Python module is requested, healpix will also build its -# Python module Healpy. -# -# Bug: Healpix 3.50 doesn't yet support Python 3.10, so we are commenting -# it here. When future versions fix the problem, we'll un-comment it again. -#healpy-version = xxxxx - -# Problematic software that are currently ignored: - -# Healpix 3.80 has different installation scheme that is not yet -# implemented here, and unfortunately healpix 3.50 has conflicts with -# CFITSIO's new version macro and Python 3.10. So for now, we are ignoring -# it. -#healpix-version = 3.50 +# Problematic software that is currently ignored: # Setuptools-rust crash (https://savannah.nongnu.org/bugs/index.php?61731), # so it and its dependencies are being ignored: 'cryptography', and thus @@ -324,6 +356,34 @@ semantic-version-version = 2.10.0 #keyring-version = 18.0.0 #astroquery-version = 0.4 +# Not yet adated to the GPEP517 build method: +#asn1crypto-version = 0.24.0 +#asteval-version = 0.9.16 +#beniget-version = 0.4.1 +#certifi-version = 2018.11.29 +#chardet-version = 3.0.4 +#corner-version = 2.0.1 +#eigency-version = 1.77 +#emcee-version = 3.0.1 +#entrypoints-version = 0.3 +#esutil-version = 0.6.9 +#future-version = 0.18.1 +#gast-version = 0.5.3 +#idna-version = 2.8 +#jeepney-version = 0.4 +#markupsafe-version = 2.0.1 +#mpmath-version = 1.1.0 +#pexpect-version = 4.7.0 +#ply-version = 3.11 +#pycodestyle-version = 2.5.0 +#pycparser-version = 2.19 +#pyflakes-version = 2.1.1 +#pypkgconfig-version = 1.5.1 +#soupsieve-version = 1.8 +#uncertainties-version = 3.1.6 +#urllib3-version = 1.24.1 +#webencodings-version = 0.5.1 + diff --git a/reproduce/software/make/README.md b/reproduce/software/make/README.md index 00afef4..0d1464d 100644 --- a/reproduce/software/make/README.md +++ b/reproduce/software/make/README.md @@ -1,7 +1,7 @@ Software building instructions ------------------------------ -Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ +Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ See the end of the file for license conditions. This directory contains Makefiles that are called by the high-level diff --git a/reproduce/software/make/atlas-multiple.mk b/reproduce/software/make/atlas-multiple.mk index 6e7d415..a20f558 100644 --- a/reproduce/software/make/atlas-multiple.mk +++ b/reproduce/software/make/atlas-multiple.mk @@ -8,7 +8,7 @@ # # ------------------------------------------------------------------------ # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/reproduce/software/make/atlas-single.mk b/reproduce/software/make/atlas-single.mk index 2b68677..79f5b42 100644 --- a/reproduce/software/make/atlas-single.mk +++ b/reproduce/software/make/atlas-single.mk @@ -8,7 +8,7 @@ # # ------------------------------------------------------------------------ # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/reproduce/software/make/basic.mk b/reproduce/software/make/basic.mk index 745aeca..b045bb1 100644 --- a/reproduce/software/make/basic.mk +++ b/reproduce/software/make/basic.mk @@ -21,9 +21,9 @@ # # ------------------------------------------------------------------------ # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> -# Copyright (C) 2022-2025 Pedram Ashofteh Ardakani <pedramardakani@pm.me> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2022-2026 Pedram Ashofteh Ardakani <pedramardakani@pm.me> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -39,7 +39,7 @@ # along with this Makefile. If not, see <http://www.gnu.org/licenses/>. # Top level environment -include reproduce/software/config/LOCAL.conf +include .build/software/config/LOCAL.conf include reproduce/software/make/build-rules.mk include reproduce/software/config/versions.conf include reproduce/software/config/checksums.conf @@ -63,10 +63,13 @@ ibidir = $(BDIR)/software/installed/version-info/proglib # editor) is installed by default, it is recommended to have it in the # 'basic.mk', so Maneaged projects can be edited on any system (even when # there is no command-line text editor available). +# +# The recipe is '@echo > /dev/null' so Make does not print "make: Nothing +# to be done for 'all'." targets-proglib = low-level-links \ gcc-$(gcc-version) \ nano-$(nano-version) -all: $(foreach p, $(targets-proglib), $(ibidir)/$(p)) +all: $(foreach p, $(targets-proglib), $(ibidir)/$(p)); @echo > /dev/null # Define the shell environment # ---------------------------- @@ -84,6 +87,7 @@ all: $(foreach p, $(targets-proglib), $(ibidir)/$(p)) # own build-directory comes first. .ONESHELL: .SHELLFLAGS := -e -c +export LC_ALL := C export CCACHE_DISABLE := 1 export SHELL := $(ibdir)/dash export PATH := $(ibdir):$(PATH) @@ -102,12 +106,13 @@ endif export CPPFLAGS := -I$(idir)/include $(CPPFLAGS) $(noccwarnings) # This is the "basic" tools where we are relying on the host operating -# system, but are slowly populating our basic software envirnoment. To run -# (system or template) programs, 'LD_LIBRARY_PATH' is necessary, so here, -# we'll first tell the programs to look into any possible pre-defined -# 'LD_LIBRARY_PATH', then we'll add our own newly installed libraries. We -# will also make sure that there is no "current directory" in it (by -# removing a starting or trailing ':' and any occurance of '::'. +# system, but are slowly populating our basic software envirnoment. So +# here, we are also appending the host's 'LD_LIBRARY_PATH' to Maneage's +# installed library path. It is just important to keep Maneage's library +# directories first so when something has been built, the newly built +# software use (link-with) Maneage's libraries, not the host's. We will +# also make sure that there is no "current directory" in it (by removing a +# starting or trailing ':' and any occurance of '::'). # # But first: in case LD_LIBRARY_PATH is empty, give it the default value of # $(sys_library_sh_path) (which was the location of the libraries needed by @@ -117,8 +122,9 @@ export CPPFLAGS := -I$(idir)/include $(CPPFLAGS) $(noccwarnings) ifeq ($(strip $(LD_LIBRARY_PATH)),) export LD_LIBRARY_PATH=$(sys_library_sh_path) endif -export LD_LIBRARY_PATH := $(shell echo $(LD_LIBRARY_PATH):$(ildir) \ +export LD_LIBRARY_PATH := $(shell echo $(ildir):$(LD_LIBRARY_PATH) \ | sed -e's/::/:/g' -e's/^://' -e's/:$$//') +ldpathorig := $(subst $(ildir):,,$(LD_LIBRARY_PATH)) # RPATH is automatically written in macOS, so 'DYLD_LIBRARY_PATH' is # ultimately redundant. But on some systems, even having a single value @@ -144,10 +150,17 @@ backupservers_all = $(user_backup_urls) $(maneage_backup_urls) topbackupserver = $(word 1, $(backupservers_all)) backupservers = $(filter-out $(topbackupserver),$(backupservers_all)) - - - - +# When building in Apptainer containers, as of 2025-04-18, we need to +# configure Maneage as root (within the container). In such cases, we need +# to activate the 'FORCE_UNSAFE_CONFIGURE' environment variable to build +# some of the software. The 'if' statement is here to make sure we are in +# Apptainer: in other situations, the "unsafe" configure script shouldn't +# be activated. Note that this doesn't happen in Docker (where the Maneage +# source is in the same directory) because we build a non-root ('maneager' +# user there who executes the configure command. +unsafe-config = if [ $$(pwd) = "/home/maneager/source" ] \ + && [ $$(whoami) = root ]; then \ + export FORCE_UNSAFE_CONFIGURE=1; fi @@ -243,6 +256,23 @@ $(ibidir)/low-level-links: $(ibidir)/grep-$(grep-version) \ # systems) $(call makelink,ldd) +# zic: On some systems like Debian GNU/Linux, 'zic' may be in +# '/usr/sbin' and is available to users. But it is not normally in +# the 'PATH' of a non-root user. This is because the executables +# there are normally system-level commands, and most are executable +# only by the root user. + $(call makelink,zic) + if [ "x$$(command -v zic)" = x ]; then + zicfullpath=$$(PATH=$${PATH}:/usr/sbin command -v zic) + if [ "x$${zicfullpath}" = x ]; then + printf "Warning: 'zic' not found. You may need it for" + printf " higher-level packages such as 'tzdb'. Install" + printf " 'zic' if needed.\n" + else + ln -sf $$(realpath $${zicfullpath}) $(ibdir)/zic + fi + fi + # We want this to be empty (so it doesn't interefere with the other # files in 'ibidir'. touch $@ @@ -261,11 +291,8 @@ $(ibidir)/low-level-links: $(ibidir)/grep-$(grep-version) \ # # The first set of programs to be built are those that we need to unpack # the source code tarballs of each program. We have already installed Lzip -# before calling 'basic.mk', so it is present and working. Hence we first -# build the Lzipped tarball of Gzip, then use our own Gzip to unpack the -# tarballs of the other compression programs. Once all the compression -# programs/libraries are complete, we build our own GNU Tar and continue -# with other software. +# before calling 'basic.mk', so it is present and working. So the only +# prerequisites of these (until reaching Tar) is the necessary directories. $(lockdir): | $(BDIR); mkdir $@ $(ibidir)/gzip-$(gzip-version): | $(ibdir) $(ildir) $(lockdir) tarball=gzip-$(gzip-version).tar.lz @@ -273,13 +300,13 @@ $(ibidir)/gzip-$(gzip-version): | $(ibdir) $(ildir) $(lockdir) $(call gbuild, gzip-$(gzip-version), static, , V=1) echo "GNU Gzip $(gzip-version)" > $@ -$(ibidir)/xz-$(xz-version): $(ibidir)/gzip-$(gzip-version) +$(ibidir)/xz-$(xz-version): | $(ibdir) $(ildir) $(lockdir) tarball=xz-$(xz-version).tar.lz $(call import-source, $(xz-url), $(xz-checksum)) $(call gbuild, xz-$(xz-version), static) echo "XZ Utils $(xz-version)" > $@ -$(ibidir)/bzip2-$(bzip2-version): $(ibidir)/gzip-$(gzip-version) +$(ibidir)/bzip2-$(bzip2-version): | $(ibdir) $(ildir) $(lockdir) # Download the tarball. tarball=bzip2-$(bzip2-version).tar.lz @@ -308,7 +335,7 @@ $(ibidir)/bzip2-$(bzip2-version): $(ibidir)/gzip-$(gzip-version) fi cd $(ddir) rm -rf $$tdir - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$tdir $(shsrcdir)/prep-source.sh $(ibdir) sed -e 's@\(ln -s -f \)$$(PREFIX)/bin/@\1@' Makefile \ @@ -330,30 +357,32 @@ $(ibidir)/bzip2-$(bzip2-version): $(ibidir)/gzip-$(gzip-version) # # Note for a static-only build: Zlib's './configure' doesn't use Autoconf's # configure script, it just accepts a direct '--static' option. -$(ibidir)/zlib-$(zlib-version): $(ibidir)/gzip-$(gzip-version) +$(ibidir)/zlib-$(zlib-version): | $(ibdir) $(ildir) $(lockdir) tarball=zlib-$(zlib-version).tar.lz $(call import-source, $(zlib-url), $(zlib-checksum)) $(call gbuild, zlib-$(zlib-version)) echo "Zlib $(zlib-version)" > $@ # GNU Tar: When built statically, tar gives a segmentation fault on -# unpacking Bash. So we'll build it dynamically. Note that technically, zip -# and unzip aren't dependencies of Tar, but for a clean build, we'll set -# Tar to be the last compression-related software (the first-set of -# software to be built). +# unpacking Bash. So we'll build it dynamically. $(ibidir)/tar-$(tar-version): \ $(ibidir)/xz-$(xz-version) \ $(ibidir)/gzip-$(gzip-version) \ $(ibidir)/zlib-$(zlib-version) \ $(ibidir)/bzip2-$(bzip2-version) -# Since all later programs depend on Tar, the configuration will hit -# a bottleneck here: only making Tar. So its more efficient to built -# it on multiple threads (even when the user's Make doesn't pass down -# the number of threads). +# About the configurations: disabled because they are not relevant in +# the context of Maneage usage. +# - nls: conflicts with iconv on macOS. +# - acl: causing crashes when Tar was compiled with GCC 16.1.1. + $(call unsafe-config) tarball=tar-$(tar-version).tar.lz $(call import-source, $(tar-url), $(tar-checksum)) - $(call gbuild, tar-$(tar-version), , , -j$(numthreads) V=1) + $(call gbuild, tar-$(tar-version), , \ + --disable-nls \ + --disable-acl \ + am_cv_func_iconv=no, \ + -j$(numthreads) V=1) echo "GNU Tar $(tar-version)" > $@ @@ -370,8 +399,8 @@ $(ibidir)/tar-$(tar-version): \ # Patchelf is necessary for some software on GNU/Linux systems, its job is # to manually insert RPATH into the dynamically-linked executable. Since # all the other software depend on Pathelf, to avoid manually repeating as -# a prerequisite (and forgetting in others causing bugs), we'll put it as a -# dependancy of 'tar'. +# a prerequisite (and forgetting in others causing bugs), it is installed +# in this phase (right after 'tar'). $(ibidir)/patchelf-$(patchelf-version): $(ibidir)/tar-$(tar-version) tarball=patchelf-$(patchelf-version).tar.lz $(call import-source, $(patchelf-url), $(patchelf-checksum)) @@ -382,6 +411,84 @@ $(ibidir)/patchelf-$(patchelf-version): $(ibidir)/tar-$(tar-version) echo "PatchELF $(patchelf-version)" > $@ fi +$(ibidir)/pkg-config-$(pkgconfig-version): $(ibidir)/tar-$(tar-version) + +# Download the tarball. + tarball=pkg-config-$(pkgconfig-version).tar.lz + $(call import-source, $(pkgconfig-url), $(pkgconfig-checksum)) + +# An existing 'libiconv' can cause a conflict with 'pkg-config' (this +# is why 'libiconv' depends on 'pkg-config'). On a clean build, +# 'pkg-config' is built first. But when we don't have a clean build +# (and 'libiconv' exists) there will be a problem. So before +# re-building 'pkg-config', we'll remove any installation of +# 'libiconv'. This can create a known bug: +# https://savannah.nongnu.org/bugs/?67001 + rm -f $(ildir)/libiconv* $(idir)/include/iconv.h + +# On GNU/Linux no changes necessary, so we'll start with those. If +# the runner is on a macOS, some changes are necessary. + export compiler="" + confsh=./configure + if [ x$(on_mac_os) = xyes ]; then + +# On macOS we force Clang, because some GCC versions shipped with +# Xcode lack features needed to build Glib. + export compiler="CC=clang" + +# The bundled glib's gatomic.c uses integer-to-pointer conversions +# that newer Clang (Xcode 26+) treats as hard errors on all macOS +# architectures. + CFLAGS="-Wno-int-conversion $$CFLAGS" + +# On macOS ARM64 the bundled glib sub-configure checks for Carbon +# support with the C preprocessor only. Since 'Carbon/Carbon.h' +# exists in the Xcode SDK on Apple Silicon, the check passes and +# '-framework Carbon' is appended to LDFLAGS. But Carbon cannot +# actually be linked on ARM64, so every subsequent linker test in +# glib's configure fails silently and libglib-2.0.la is never +# created. We work around this by passing gbuild a tiny wrapper +# script (instead of ./configure) that patches glib/configure to +# force glib_have_carbon=no before the real configure runs. +# +# NOTE: the initial two SPACES in the SED command are very +# important (see 'glib/configure'): without them, the wrong +# variable will be set. + confsh=$(ddir)/pkg-config-configure + printf '#!/bin/sh\n' > $$confsh + printf 'sed -i.bak "s/^ glib_have_carbon=yes$$/' >> $$confsh + printf ' glib_have_carbon=no/"' >> $$confsh + printf ' glib/configure\n' >> $$confsh + printf 'exec ./configure "$$@"\n' >> $$confsh + chmod +x $$confsh + +# For its 'glib' installation within macOS, pkg-config needs a +# working Python for its build (not affecting its final code), but +# only searches for '/usr/bin/python'. However, some users may have +# Python available in another location, so we'll set it externally +# here to be safe. Note that 'which' is not present on some +# systems, 'command -v' is more portable). + if type python > /dev/null 2> /dev/null; then + pypath=$$(command -v python) + elif type python3 > /dev/null 2> /dev/null; then + pypath=$$(command -v python3) + else + printf "pkg-config: ERROR: a local Python is necessary"; exit 1 + fi + export PYTHON=$$pypath + fi + +# Configuration options: +# --with-internal-glib: the internal glib is necessary because +# without it, we have many problems in macOS. + export CFLAGS="-std=$(std_c_old) $$CFLAGS" + $(call gbuild, pkg-config-$(pkgconfig-version), static, \ + $$compiler \ + --with-internal-glib \ + --with-pc-path=$(ildir)/pkgconfig, V=1, , $$confsh) + if [ -f $$confsh ]; then rm $$confsh; fi # Only on macOS. + echo "pkg-config $(pkgconfig-version)" > $@ + @@ -402,7 +509,10 @@ $(ibidir)/patchelf-$(patchelf-version): $(ibidir)/tar-$(tar-version) # function (for tilde expansion). The first can be disabled with # '--disable-load', but unfortunately I don't know any way to fix the # second. So, we'll have to build it dynamically for now. -$(ibidir)/ncurses-$(ncurses-version): $(ibidir)/patchelf-$(patchelf-version) +$(ibidir)/ncurses-$(ncurses-version): $(ibidir)/patchelf-$(patchelf-version) \ + $(ibidir)/pkg-config-$(pkgconfig-version) + +# Prepare the input. tarball=ncurses-$(ncurses-version).tar.lz $(call import-source, $(ncurses-url), $(ncurses-checksum)) @@ -418,11 +528,18 @@ $(ibidir)/ncurses-$(ncurses-version): $(ibidir)/patchelf-$(patchelf-version) rm -f $(ibdir)/bash* $(ibdir)/awk* $(ibdir)/gawk* # Standard build process. + export CFLAGS="-std=$(std_c_old) $$CFLAGS" $(call gbuild, ncurses-$(ncurses-version), static, \ - --with-shared --enable-rpath --without-normal \ - --without-debug --with-cxx-binding \ - --with-cxx-shared --enable-widec --enable-pc-files \ - --with-pkg-config=$(ildir)/pkgconfig, -j$(numthreads)) + --with-shared \ + --enable-rpath \ + --enable-widec \ + --without-debug \ + --without-normal \ + --enable-pc-files \ + --with-cxx-shared \ + --with-cxx-binding \ + --with-pkg-config=$(ildir)/pkgconfig, \ + -j$(numthreads)) # Unfortunately there are many problems with 'ncurses' using "normal" # (or 8-bit) characters. The standard way that will work is to build @@ -531,6 +648,7 @@ $(ibidir)/readline-$(readline-version): \ # pname=bash50-$(printf "%03d" $i); \ # wget http://ftp.gnu.org/gnu/bash/bash-5.0-patches/$pname -O ../$pname;\ # patch -p0 -i ../$pname; \ +# rm ../$pname; \ # done # $ cd .. # $ mv bash-5.0 bash-5.0.$number @@ -559,7 +677,7 @@ $(ibidir)/bash-$(bash-version): \ export CFLAGS="$$CFLAGS \ -DDEFAULT_PATH_VALUE='\"$(ibdir)\"' \ -DSTANDARD_UTILS_PATH='\"$(ibdir)\"' \ - -DSYS_BASHRC='\"$(BASH_ENV)\"' " + -DSYS_BASHRC='\"$(BASH_ENV)\"'" $(call gbuild, bash-$(bash-version),, $$stopt \ --with-installed-readline=$(ildir) \ --with-curses=yes, \ @@ -576,9 +694,6 @@ $(ibidir)/bash-$(bash-version): \ # shell. By convention, 'sh' is just a symbolic link to the preferred # shell executable. So we'll define '$(ibdir)/sh' as a symbolic link # to the Bash that we just built and installed. -# -# Just to be sure that the installation step above went well, before -# making the link, we'll see if the file actually exists there. ln -fs $(ibdir)/bash $(ibdir)/sh echo "GNU Bash $(bash-version)" > $@ @@ -604,18 +719,24 @@ perl-conflddlflags = else perl-conflddlflags = -Dlddlflags="-shared $$LDFLAGS" endif -$(ibidir)/perl-$(perl-version): $(ibidir)/patchelf-$(patchelf-version) +$(ibidir)/perl-$(perl-version): $(ibidir)/tar-$(tar-version) + +# Import tarball. tarball=perl-$(perl-version).tar.lz $(call import-source, $(perl-url), $(perl-checksum)) + +# Extract the version strings. major_version=$$(echo $(perl-version) \ | sed -e's/\./ /g' \ | awk '{printf("%d", $$1)}') base_version=$$(echo $(perl-version) \ | sed -e's/\./ /g' \ | awk '{printf("%d.%d", $$1, $$2)}') + +# Unpack, build and install. cd $(ddir) rm -rf perl-$(perl-version) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd perl-$(perl-version) $(shsrcdir)/prep-source.sh $(ibdir) ./Configure -des \ @@ -623,15 +744,15 @@ $(ibidir)/perl-$(perl-version): $(ibidir)/patchelf-$(patchelf-version) -Duseshrplib \ -Dprefix=$(idir) \ -Dvendorprefix=$(idir) \ - -Dprivlib=$(idir)/share/perl$$major_version/core_perl \ - -Darchlib=$(idir)/lib/perl$$major_version/$$base_version/core_perl \ - -Dsitelib=$(idir)/share/perl$$major_version/site_perl \ - -Dsitearch=$(idir)/lib/perl$$major_version/$$base_version/site_perl \ - -Dvendorlib=$(idir)/share/perl$$major_version/vendor_perl \ - -Dvendorarch=$(idir)/lib/perl$$major_version/$$base_version/vendor_perl \ - -Dscriptdir=$(idir)/bin/core_perl \ - -Dsitescript=$(idir)/bin/site_perl \ - -Dvendorscript=$(idir)/bin/vendor_perl \ + -Dprivlib=$(idir)/share/perl$$major_version/perl-core \ + -Darchlib=$(idir)/lib/perl$$major_version/$$base_version/perl-core \ + -Dsitelib=$(idir)/share/perl$$major_version/perl-site \ + -Dsitearch=$(idir)/lib/perl$$major_version/$$base_version/perl-site \ + -Dvendorlib=$(idir)/share/perl$$major_version/perl-vendor \ + -Dvendorarch=$(idir)/lib/perl$$major_version/$$base_version/perl-vendor \ + -Dscriptdir=$(idir)/bin/perl-core \ + -Dsitescript=$(idir)/bin/perl-site \ + -Dvendorscript=$(idir)/bin/perl-vendor \ -Dinc_version_list=none \ -Dman1ext=1perl \ -Dman3ext=3perl \ @@ -642,7 +763,14 @@ $(ibidir)/perl-$(perl-version): $(ibidir)/patchelf-$(patchelf-version) make install cd .. rm -rf perl-$(perl-version) - cd $$topdir + +# Symbolic links to necessary Perl programs (that Perl installs in +# the 'perl-core' directory. + ln -sf $(ibdir)/perl-core/xsubpp $(ibdir)/xsubpp + ln -sf $(ibdir)/perl-core/pod2man $(ibdir)/pod2man + ln -sf $(ibdir)/perl-core/pod2text $(ibdir)/pod2text + +# Final target. echo "Perl $(perl-version)" > $@ @@ -675,14 +803,15 @@ $(ibidir)/coreutils-$(coreutils-version): \ $(ibidir)/perl-$(perl-version) \ $(ibidir)/openssl-$(openssl-version) -# Import the source tarball. +# Import, unpack and enter the source directory. + $(call unsafe-config) tarball=coreutils-$(coreutils-version).tar.lz $(call import-source, $(coreutils-url), $(coreutils-checksum)) # Unpack and enter the source. cd $(ddir) rm -rf coreutils-$(coreutils-version) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd coreutils-$(coreutils-version) $(shsrcdir)/prep-source.sh $(ibdir) @@ -696,7 +825,7 @@ $(ibidir)/coreutils-$(coreutils-version): \ # Fix RPATH if necessary. if [ -f $(ibdir)/patchelf ]; then make SHELL=$(ibdir)/bash install DESTDIR=junkinst - unalias ls || true # avoid decorated 'ls' commands with extra characters + unalias ls || true # Not decorated 'ls' (with extra characters). instprogs=$$(ls junkinst/$(ibdir)) for f in $$instprogs; do $(ibdir)/patchelf --set-rpath $(ildir) $(ibdir)/$$f @@ -710,29 +839,6 @@ $(ibidir)/coreutils-$(coreutils-version): \ rm -rf coreutils-$(coreutils-version) echo "GNU Coreutils $(coreutils-version)" > $@ -# Podlators -# -# POD is short for "Plain Old Documentation", that is the format used in -# Perl's documentation. Podlators provies two executables pod2man and -# pod2text convert this into the roff format (used in man pages) or pod2 It -# is used by some software like OpenSSL to create their man pages. -$(ibidir)/podlators-$(podlators-version): $(ibidir)/perl-$(perl-version) - tarball=podlators-$(podlators-version).tar.lz - $(call import-source, $(podlators-url), $(podlators-checksum)) - cd $(ddir) - rm -rf podlators-$(podlators-version) - tar -xf $(tdir)/$$tarball - cd podlators-$(podlators-version) - $(shsrcdir)/prep-source.sh $(ibdir) - perl Makefile.PL - make - make install - ln -sf $(ibdir)/site_perl/pod2man $(ibdir)/pod2man - ln -sf $(ibdir)/site_perl/pod2text $(ibdir)/pod2text - cd .. - rm -rf podlators-$(podlators-version) - echo "podlators $(podlators-version)" > $@ - # OpenSSL # # Until we find a nice and generic way to create an updated CA file in the @@ -740,8 +846,9 @@ $(ibidir)/podlators-$(podlators-version): $(ibidir)/perl-$(perl-version) # along with the other tarballs. $(idir)/etc:; mkdir $@ $(idir)/etc/ssl: | $(idir)/etc; mkdir $@ -$(ibidir)/openssl-$(openssl-version): $(ibidir)/podlators-$(podlators-version) \ - | $(idir)/etc/ssl +$(ibidir)/openssl-$(openssl-version): $(ibidir)/perl-$(perl-version) \ + $(ibidir)/patchelf-$(patchelf-version) \ + | $(idir)/etc/ssl # First download the certificates and copy them into the # installation directory. @@ -851,6 +958,23 @@ $(ibidir)/wget-$(wget-version): \ tarball=wget-$(wget-version).tar.lz $(call import-source, $(wget-url), $(wget-checksum)) +# Wget needs a working Python for its tests (VERY BAD DESIGN!), this +# does not affect its final operation. But only searches for +# '/usr/bin/python'. However, some users may have Python available in +# another location, so we'll set it externally here to be safe. Note +# that 'which' is not present on some systems, 'command -v' is more +# portable). + if [ x$(on_mac_os) = xyes ]; then + if type python > /dev/null 2> /dev/null; then + pypath=$$(command -v python) + elif type python3 > /dev/null 2> /dev/null; then + pypath=$$(command -v python3) + else + printf "Wget: ERROR: a local Python is necessary"; exit 1 + fi + export PYTHON=$$pypath + fi + # We need to explicitly disable 'libiconv', because of the # 'pkg-config' and 'libiconv' problem. libs="-pthread" @@ -897,7 +1021,7 @@ $(ibidir)/diffutils-$(diffutils-version): \ echo "GNU Diffutils $(diffutils-version)" > $@ $(ibidir)/file-$(file-version): $(ibidir)/coreutils-$(coreutils-version) - export CFLAGS="-std=c99 $$CFLAGS" + export CFLAGS="-std=$(std_c_old) $$CFLAGS" tarball=file-$(file-version).tar.lz $(call import-source, $(file-url), $(file-checksum)) $(call gbuild, file-$(file-version), static, \ @@ -949,6 +1073,8 @@ $(ibidir)/help2man-$(help2man-version): \ $(call gbuild, help2man-$(help2man-version), static, ,V=1) echo "Help2man $(Help2man-version)" > $@ +# Even though libiconv does not use pkg-config, pkg-config conflicts with +# it, see the comments in pkg-config for more. $(ibidir)/libiconv-$(libiconv-version): \ $(ibidir)/pkg-config-$(pkgconfig-version) tarball=libiconv-$(libiconv-version).tar.lz @@ -964,12 +1090,14 @@ $(ibidir)/libunistring-$(libunistring-version): \ -j$(numthreads)) echo "GNU libunistring $(libunistring-version)" > $@ -$(ibidir)/libxml2-$(libxml2-version): $(ibidir)/patchelf-$(patchelf-version) +$(ibidir)/libxml2-$(libxml2-version): $(ibidir)/tar-$(tar-version) # The libxml2 tarball also contains Python bindings which are built # and installed to a system directory by default. If you don't need # the Python bindings, the easiest solution is to compile without # Python support: './configure --without-python'. If you really need -# the Python bindings, use '--with-python-install-dir=DIR' instead. +# its Python bindings: +# 1. Add the Python dependency. +# 2. Add the '--with-python-install-dir=DIR' configure option. tarball=libxml2-$(libxml2-version).tar.lz $(call import-source, $(libxml2-url), $(libxml2-checksum)) $(call gbuild, libxml2-$(libxml2-version), static, \ @@ -1008,6 +1136,7 @@ $(ibidir)/gmp-$(gmp-version): \ $(ibidir)/coreutils-$(coreutils-version) tarball=gmp-$(gmp-version).tar.lz $(call import-source, $(gmp-url), $(gmp-checksum)) + export CFLAGS="-std=$(std_c_old) $$CFLAGS" $(call gbuild, gmp-$(gmp-version), static, \ --enable-cxx --enable-fat, \ -j$(numthreads)) @@ -1054,10 +1183,6 @@ $(ibidir)/grep-$(grep-version): $(ibidir)/coreutils-$(coreutils-version) -j$(numthreads) V=1) echo "GNU Grep $(grep-version)" > $@ -# M4 doesn't depend on PatchELF, but just to be consistent with the -# levels/phases introduced here (where the compressors are level 1, -# PatchELF is level 2, and ...), we'll set it as a dependency. -# # The '--with-syscmd-shell' is used as the default shell and if not given, # 'm4' will use '/bin/sh' (which is not under Maneage control and can cause # problems in 'high-level.mk' because it closes off the system's @@ -1065,7 +1190,7 @@ $(ibidir)/grep-$(grep-version): $(ibidir)/coreutils-$(coreutils-version) # library, the high-level programs will not be built). We are setting this # default shell to Dash because M4 is built before our own Bash. Recall # that Dash is built before we enter this Makefile. -$(ibidir)/m4-$(m4-version): $(ibidir)/patchelf-$(patchelf-version) +$(ibidir)/m4-$(m4-version): $(ibidir)/libiconv-$(libiconv-version) tarball=m4-$(m4-version).tar.lz $(call import-source, $(m4-url), $(m4-checksum)) $(call gbuild, m4-$(m4-version), static, \ @@ -1079,32 +1204,6 @@ $(ibidir)/mpfr-$(mpfr-version): $(ibidir)/gmp-$(gmp-version) $(call gbuild, mpfr-$(mpfr-version), static) echo "GNU Multiple Precision Floating-Point Reliably $(mpfr-version)" > $@ -$(ibidir)/pkg-config-$(pkgconfig-version): $(ibidir)/patchelf-$(patchelf-version) - -# Download the tarball. - tarball=pkg-config-$(pkgconfig-version).tar.lz - $(call import-source, $(pkgconfig-url), $(pkgconfig-checksum)) - -# An existing 'libiconv' can cause a conflict with 'pkg-config', this -# is why 'libiconv' depends on 'pkg-config'. On a clean build, -# 'pkg-config' is built first. But when we don't have a clean build -# (and 'libiconv' exists) there will be a problem. So before -# re-building 'pkg-config', we'll remove any installation of -# 'libiconv'. - rm -f $(ildir)/libiconv* $(idir)/include/iconv.h - -# Some Mac OS systems may have a version of the GNU C Compiler (GCC) -# installed that doesn't support some necessary features of building -# Glib (as part of pkg-config). So to be safe, for Mac systems, we'll -# make sure it will use LLVM's Clang. - if [ x$(on_mac_os) = xyes ]; then export compiler="CC=clang" - else export compiler="" - fi - $(call gbuild, pkg-config-$(pkgconfig-version), static, \ - $$compiler --with-internal-glib \ - --with-pc-path=$(ildir)/pkgconfig, V=1) - echo "pkg-config $(pkgconfig-version)" > $@ - $(ibidir)/sed-$(sed-version): $(ibidir)/coreutils-$(coreutils-version) tarball=sed-$(sed-version).tar.lz $(call import-source, $(sed-url), $(sed-checksum)) @@ -1212,6 +1311,7 @@ $(ibidir)/binutils-$(binutils-version): \ $(call makelink,ld) $(call makelink,nm) $(call makelink,ps) + $(call makelink,lipo) $(call makelink,strip) $(call makelink,ranlib) echo "" > $@ @@ -1400,7 +1500,7 @@ $(ibidir)/gcc-$(gcc-version): $(ibidir)/binutils-$(binutils-version) # Unpack GCC and prepare the 'build' directory inside it for all # the built files. rm -rf gcc-$(gcc-version) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions if [ $$odir != $(ddir) ]; then ln -s $$odir/gcc-$(gcc-version) $(ddir)/gcc-$(gcc-version) fi @@ -1417,24 +1517,6 @@ $(ibidir)/gcc-$(gcc-version): $(ibidir)/binutils-$(binutils-version) # first look into its own 'include' directory before anything else. export CPPFLAGS="-I$$(pwd)/include $(CPPFLAGS)" -# In the GNU C Library 2.36 (which is more recent than GCC 12.1.0), -# the 'linux/mount.h' (loaded by 'linux/fs.h', which is loaded by -# 'libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cpp' -# in GCC) conflicts with 'sys/mount.h' which is directly loaded by -# the same file! This is a known conflict in glibc 2.36 (see -# [1]). As described in [1], one solution is the final job done in -# [2]. We therefore do this process here: 1) Not loading -# 'linux/fs.h', and adding the necessary macros directly. -# -# [1] https://sourceware.org/glibc/wiki/Release/2.36#Usage_of_.3Clinux.2Fmount.h.3E_and_.3Csys.2Fmount.h.3E -# [2] https://reviews.llvm.org/D129471 - sed -e's|\#include <linux/fs.h>||' \ - -e"s|FS_IOC_GETFLAGS;|_IOR('f', 1, long);|" \ - -e"s|FS_IOC_GETVERSION;|_IOR('v', 1, long);|" \ - -e"s|FS_IOC_SETFLAGS;|_IOW('f', 2, long);|" \ - -e"s|FS_IOC_SETVERSION;|_IOW('v', 2, long);|" \ - -i libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cpp - # Set the build directory for the processing. mkdir build cd build @@ -1530,7 +1612,7 @@ $(ibidir)/lzip-$(lzip-version): $(ibidir)/gcc-$(gcc-version) unpackdir=lzip-$(lzip-version) cd $(ddir) rm -rf $$unpackdir - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) ./configure --build --check --installdir="$(ibdir)" diff --git a/reproduce/software/make/build-rules.mk b/reproduce/software/make/build-rules.mk index 62cb6d5..454a255 100644 --- a/reproduce/software/make/build-rules.mk +++ b/reproduce/software/make/build-rules.mk @@ -3,7 +3,7 @@ # imported into 'basic.mk' and 'high-level.mk'. They should be activated # with Make's 'Call' function. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -40,6 +40,14 @@ # Necessary shell variables # 'tarball': This is the name of the actual tarball file without a # directory. +# +# Notes: +# - The LD_LIBRARY_PATH correction is because we build the dependencies of +# the downloaders (Wget or cURL) before they are built them selves. In +# the time between the installation of the downloader dependencies and +# the downloader, the Gnuastro libraries may cause crashes in the host's +# downloader since normally, Gnuastro's library has higher precedence in +# 'LD_LIBRARY_PATH'. import-source = final=$(tdir)/$$tarball; \ url=$(strip $(1)); \ exp_checksum="$(strip $(2))"; \ @@ -82,8 +90,14 @@ import-source = final=$(tdir)/$$tarball; \ if [ x"$$exp_checksum" = x"NO-CHECK-SUM" ]; then \ mv "$$unchecked" "$$final"; \ else \ - if type sha512sum > /dev/null 2>/dev/null; then \ + if type sha512sum > /dev/null 2> /dev/null; then \ + if ! sha512sum --version > /dev/null 2> /dev/null; then \ + export LD_LIBRARY_PATH=$(ldpathorig); \ + fi; \ checksum=$$(sha512sum "$$unchecked" | awk '{print $$1}'); \ + if [ LD_LIBRARY_PATH = $(ldpathorig) ]; then \ + export LD_LIBRARY_PATH=$(LD_LIBRARY_PATH); \ + fi; \ if [ x"$$checksum" = x"$$exp_checksum" ]; then \ mv "$$unchecked" "$$final"; \ else \ @@ -160,7 +174,7 @@ uncompress = csuffix=$$(echo $$utarball \ intarrm=0; \ intar=$$utarball; \ fi; \ - if tar -xf $$intar; then \ + if tar -xf $$intar --no-same-owner --no-same-permissions; then \ if [ x$$intarrm = x1 ]; then rm $$intar; fi; \ else \ echo; echo "Tar error"; exit 1; \ diff --git a/reproduce/software/make/high-level.mk b/reproduce/software/make/high-level.mk index c81f0b9..c6dcf90 100644 --- a/reproduce/software/make/high-level.mk +++ b/reproduce/software/make/high-level.mk @@ -12,8 +12,8 @@ # # ------------------------------------------------------------------------ # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -29,7 +29,7 @@ # along with this Makefile. If not, see <http://www.gnu.org/licenses/>. # Top level environment (same as 'basic.mk') -include reproduce/software/config/LOCAL.conf +include .build/software/config/LOCAL.conf include reproduce/software/make/build-rules.mk include reproduce/software/config/versions.conf include reproduce/software/config/checksums.conf @@ -123,11 +123,12 @@ ifneq ($(strip $(offline)),1) target-texlive := $(itidir)/texlive endif -# Ultimate Makefile target. +# Ultimate Makefile target. The recipe is '@echo > /dev/null' so Make does +# not print "make: Nothing to be done for 'all'." all: $(foreach p, $(targets-proglib), $(ibidir)/$(p)) \ $(foreach p, $(targets-python), $(ipydir)/$(p)) \ $(foreach p, $(targets-r-cran), $(ircrandir)/$(p)) \ - $(target-texlive) + $(target-texlive); @echo > /dev/null # Define the shell environment # ---------------------------- @@ -165,6 +166,7 @@ export PKG_CONFIG_PATH := $(ildir)/pkgconfig:$(idir)/share/pkgconfig .SUFFIXES: # Settings specific to this Makefile. +export LC_ALL := C export CC := $(ibdir)/gcc export CXX := $(ibdir)/g++ export F77 := $(ibdir)/gfortran @@ -348,7 +350,8 @@ $(ibidir)/atlas-$(atlas-version): # 'rpath_command'. export LDFLAGS=-L$(ildir) cd $(ddir) - tar -xf $(tdir)/atlas-$(atlas-version).tar.lz + tar -xf $(tdir)/atlas-$(atlas-version).tar.lz \ + --no-same-owner --no-same-permissions cd ATLAS $(shsrcdir)/prep-source.sh $(ibdir) rm -rf build @@ -405,7 +408,7 @@ $(ibidir)/boost-$(boost-version): \ rm -rf $(ddir)/$$unpackdir topdir=$(pwd) cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) ./bootstrap.sh --prefix=$(idir) --with-libraries=all \ @@ -427,7 +430,8 @@ $(ibidir)/cfitsio-$(cfitsio-version): # systems. So we need to change it to our library installation # path. It doesn't affect GNU/Linux, so we'll just do it in any case # to keep things clean. - topdir=$(pwd); cd $(ddir); tar -xf $(tdir)/$$tarball + topdir=$(pwd); cd $(ddir) + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions customtar=cfitsio-$(cfitsio-version)-custom.tar.gz cd cfitsio-$(cfitsio-version) sed -i -e's|@rpath|$(ildir)|g' configure @@ -435,6 +439,23 @@ $(ibidir)/cfitsio-$(cfitsio-version): tar cf $$customtar cfitsio-$(cfitsio-version) cd $$topdir +# Add SSE2 optimization only when we are on a Linux kernel or using +# macOS on an Intel CPU. + arch=$$(uname -m) + unames=$$(uname -s) + if [ "$$unames" = "Linux" ]; then + confopts="$$confopts --enable-sse2" + elif [ "$$unames" = "Darwin" ] && [ "$$arch" = "x86_64" ]; then + confopts="$$confopts --enable-sse2" + fi + +# On macOS, system libraries (e.g., libSystem) are inside the SDK, +# not standard paths. 'xcrun' provides the correct SDK path; use $$ +# so it is evaluated by the shell (not Make). + if [ $(on_mac_os) = yes ]; then + export SDKROOT=$$(xcrun --show-sdk-path) + fi + # Continue the standard build on the customized tarball. Note that # with the installation of CFITSIO, 'fpack' and 'funpack' are not # installed by default. Because of that, they are added explicity. @@ -443,7 +464,10 @@ $(ibidir)/cfitsio-$(cfitsio-version): # specific 'shared' target for the building of the shared libraries. export gbuild_tar=$(ddir)/$$customtar $(call gbuild, cfitsio-$(cfitsio-version), , \ - --enable-sse2 --enable-reentrant \ + $$confopts \ + --disable-curl \ + --without-fortran \ + --enable-reentrant \ --with-bzip2=$(idir), , \ make fpack funpack) rm $$customtar @@ -467,7 +491,8 @@ $(ibidir)/eigen-$(eigen-version): tarball=eigen-$(eigen-version).tar.lz $(call import-source, $(eigen-url), $(eigen-checksum)) rm -rf $(ddir)/eigen-eigen-* - topdir=$(pwd); cd $(ddir); tar -xf $(tdir)/$$tarball + topdir=$(pwd); cd $(ddir) + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd eigen-$(eigen-version) if ! [ -d $(iidir)/eigen3 ]; then mkdir $(iidir)/eigen3; fi cp -r Eigen/* $(iidir)/eigen3/ # Some expect 'eigen3'. @@ -501,36 +526,62 @@ $(ibidir)/expat-$(expat-version): $(call gbuild, expat-$(expat-version), static) echo "Expat $(expat-version)" > $@ +# FFTW manual customizations: +# +# - To activate the OpenMPI features of FFTW, add OpenMPI as a dependency +# of FFTW: the necessary flags are automatically added if OpenMPI was +# installed before FFTW. +# +# - There are Intel-specific optimizations that can be enabled by adding +# the following two options to 'opts' +# +# --enable-avx --enable-sse2 +# +# However, they cause crashes on non-Intel processors (has been confirmed +# in ARM's aarch64). So in the generic scenario they are +# removed. Checking how these optimizations affect the numerical accuracy +# of the result (and thus optionally adding them for Intel-based +# processors) should be studied before they are optionally added for +# Intel-based CPUs (and ignored for others). $(ibidir)/fftw-$(fftw-version): # Prepare the source tarball. tarball=fftw-$(fftw-version).tar.lz $(call import-source, $(fftw-url), $(fftw-checksum)) -# FFTW's single and double precision libraries must be built -# independently: for the the single-precision library, we need to add -# the '--enable-float' option. We will build this first, then the -# default double-precision library. -# -# There are Intel-specific optimizations that can be enabled by -# adding the following two options to 'confop' -# -# --enable-avx --enable-sse2 -# -# However, they cause crashs on non-Intel processors (has been -# confirmed in ARM's aarch64). So in the generic scenario they are -# removed. Checking how these optimizations affect the numeric -# accuracy of the result (and thus optionally adding them for -# Intel-based processors) should be studied before they are -# optionally added for Intel-based CPUs (and ignored for others). - confop="--enable-shared --enable-threads" - $(call gbuild, fftw-$(fftw-version), static, \ - $$confop --enable-float) - $(call gbuild, fftw-$(fftw-version), static, \ - $$confop) - cp $(dtexdir)/fftw.tex $(ictdir)/ +# Specific customizations: you can add the options mentioned above +# after the 'if' statements, for example: "opts=$$opts EXTRA". + opts="--enable-shared --enable-threads" + if [ x$(on_mac_os) = xyes ]; then opts="$$opts --disable-fortran" + else + if [ -f $(ibidir)/openmpi-$(openmpi-version) ]; then + opts="$$opts --enable-openmp --enable-mpi" + fi + fi + +# FFTW's single, double and long-double precision libraries must be +# built separately (fully independent builds and installs). + $(call gbuild, fftw-$(fftw-version), static, $$opts) + $(call gbuild, fftw-$(fftw-version), static, $$opts \ + --enable-float) + $(call gbuild, fftw-$(fftw-version), static, $$opts \ + --enable-long-double) + cp -pv $(dtexdir)/fftw.tex $(ictdir)/ echo "FFTW $(fftw-version) \citep{fftw}" > $@ +# 2026-02-20 'framel' is needed by the python package lalsuite, which is +# needed by ligo-skymap; currently the compile of 'framel' 8.49.0 fails with +# +# error: initialization of 'void (*)(void)' from incompatible +# pointer type 'void (*)(int, char *)' [-Wincompatible-pointer-types] +# +# Hints: try using gcc -std=gnu99 or -std=gnu11 (default is -std=gnu17). +$(ibidir)/framel-$(framel-version): $(ibidir)/cmake-$(cmake-version) + tarball=framel-$(framel-version).tar.lz + $(call import-source, $(framel-url), $(framel-checksum)) + $(call cbuild, framel-$(framel-version), static) + echo "Framel $(framel-version)" > $@ + $(ibidir)/freetype-$(freetype-version): $(ibidir)/libpng-$(libpng-version) # As of version 2.13.2, FreeType doesn't account for the 'SHELL' # environment variable. The issue has been reported to the @@ -567,64 +618,114 @@ $(ibidir)/hdf5-$(hdf5-version): $(ibidir)/openmpi-$(openmpi-version) echo "HDF5 library $(hdf5-version)" > $@ # HEALPix includes the source of its C, C++, Python (and several other -# languages) libraries within one tarball. We will include the Python -# installation only when any other Python module is requested (in -# 'TARGETS.conf'). +# languages) libraries within one tarball. However, we do not build the +# python version here; instead we provide 'healpy' as a separate package; +# see 'python.mk' for the build rule. # # Note that the default './configure' script is an interactive script which # is hard to automate. So we need to go into the 'autotools' directory of # the 'C' and 'cxx' directories and configure the GNU Build System (with # 'autoreconf', which uses 'autoconf' and 'automake') to easily build the # HEALPix C/C++ libraries in batch mode. -ifeq ($(strip $(targets-python)),) -healpix-python-dep = -else -healpix-python-dep = $(ipydir)/matplotlib-$(matplotlib-version) \ - $(ipydir)/astropy-$(astropy-version) -endif -$(ibidir)/healpix-$(healpix-version): $(healpix-python-dep) \ +$(ibidir)/healpix-$(healpix-version): \ $(ibidir)/cfitsio-$(cfitsio-version) \ $(ibidir)/autoconf-$(autoconf-version) \ $(ibidir)/automake-$(automake-version) + +# Import the tarball. tarball=healpix-$(healpix-version).tar.lz $(call import-source, $(healpix-url), $(healpix-checksum)) - if [ x"$(healpix-python-dep)" = x ]; then - pycommand1="echo no-healpy-because-no-other-python" - pycommand2="echo no-healpy-because-no-other-python" - else - pycommand1="python setup.py build" - pycommand2="python setup.py install" - fi - rm -rf $(ddir)/Healpix_$(healpix-version) + +# Setup the build directory + rm -rf $(ddir)/healpix-$(healpix-version) topdir=$(pwd); cd $(ddir); - tar -xf $(tdir)/$$tarball - cd Healpix_$(healpix-version) + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions + cd healpix-$(healpix-version) $(shsrcdir)/prep-source.sh $(ibdir) - cd src/C/autotools + +# Compile and install libsharp: + cd src/common_libraries/libsharp + autoreconf --install + ./configure --prefix=$(idir) + make V=1 -j$(numthreads) SHELL=$(ibdir)/bash + make install + +# Compile and install the C version: + cd ../../C/autotools + printf "\n\nDEBUG ONLY\n\n"; /bin/pwd; printf "\nGUBED\n\n\n" autoreconf --install ./configure --prefix=$(idir) make V=1 -j$(numthreads) SHELL=$(ibdir)/bash make install - cd ../../cxx/autotools/ + +# Compile and install the C++ version: + cd ../../cxx/ + printf "\n\nDEBUG.BBBB ONLY\n\n"; /bin/pwd; printf "\nBBB.GUBED\n\n\n" autoreconf --install ./configure --prefix=$(idir) # With CFITSIO 4.0, the 'CFITSIO_VERSION' macro has three -# components. But this version of Healpix doesn't yet account for -# this. +# components. But this (which?) version of Healpix doesn't yet +# account for this. sed -i -e's/CFITSIO_VERSION/fitsversion/' cxxsupport/fitshandle.cc # Continue with the building. make V=1 -j$(numthreads) SHELL=$(ibdir)/bash make install - cd ../../healpy - $$pycommand1 - $$pycommand2 + +# Finalize the build cd $$topdir - rm -rf $(ddir)/Healpix_$(healpix-version) + rm -rf $(ddir)/healpix-$(healpix-version) cp $(dtexdir)/healpix.tex $(ictdir)/ echo "HEALPix $(healpix-version) \citep{healpix}" > $@ +# 2025-05-24 The build rules here for 'lalsuite' include git initialisation +# with a dummy commit of all the source files since this is required for an +# install unless the .git/ directory (currently 650 Mb) is included in the +# tarball. Our lalsuite tarball also has many big data test files removed, +# since these are unlikely to be needed in typically astronomy usage; this +# brings the .tar.lz file down to 25 Mb from the default of about 950 Mb +# (tar.gz) or 740 Mb (tar.lz). +# +# If you need to do full gravitational wave work with 'lalsuite', then you +# should provide an alternative tarball and edit the three files +# reproduce/software/config/{versions,checksums,urls}.conf in order to use +# your preferred tarball. +$(ipydir)/lalsuite-$(lalsuite-version): \ + $(ibidir)/automake-$(automake-version) \ + $(ibidir)/swig-$(swig-version) \ + $(ibidir)/metaio-$(metaio-version) \ + $(ibidir)/framel-$(framel-version) \ + $(ipydir)/astropy-$(astropy-version) \ + $(ipydir)/igwn-segments-$(igwn-segments-version) \ + $(ipydir)/lscsoft-glue-$(lscsoft-glue-version) \ + $(ipydir)/python-dateutil-$(python-dateutil-version) + tarball=lalsuite-$(lalsuite-version).tar.lz + $(call import-source, $(lalsuite-url), \ + $(lalsuite-checksum)) + +# First: build from source as if it's a C source only. + cd $(ddir) + unpackdir=lalsuite-$(lalsuite-version) + rm -rf $$unpackdir + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions + cd $$unpackdir + $(shsrcdir)/prep-source.sh $(ibdir) + +# A git commit is needed to generate version-control files; the +# strings in these will be rather arbitrary and should be ignored. + git init && git add . + git commit -a --no-edit -m "Dummy commit" \ + --author "Tarball <tar@ball.only>" + autoreconf + ./configure --prefix="$(idir)" --enable-swig-python + make + make install + +# Clean up and build final tarball + cd $(ddir); rm -fr $$unpackdir + echo "LALSuite $(lalsuite-version)" > $@ + $(ibidir)/libbsd-$(libbsd-version): $(ibidir)/libmd-$(libmd-version) tarball=libbsd-$(libbsd-version).tar.lz $(call import-source, $(libbsd-url), $(libbsd-checksum)) @@ -689,7 +790,7 @@ $(ibidir)/libpaper-$(libpaper-version): \ # Unpack, build the configure system, build and install. cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions unpackdir=libpaper-$(libpaper-version) cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) @@ -727,7 +828,8 @@ $(ibidir)/libtiff-$(libtiff-version): $(ibidir)/libjpeg-$(libjpeg-version) $(call gbuild, libtiff-$(libtiff-version), static, \ --disable-jbig \ --disable-webp \ - --disable-zstd) + --disable-zstd \ + --disable-libdeflate ) echo "Libtiff $(libtiff-version)" > $@ $(ibidir)/libtirpc-$(libtirpc-version): @@ -824,7 +926,7 @@ $(ibidir)/ninjabuild-$(ninjabuild-version): $(ibidir)/cmake-$(cmake-version) tarball=ninjabuild-$(ninjabuild-version).tar.lz $(call import-source, $(ninjabuild-url), $(ninjabuild-checksum)) cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd ninjabuild-$(ninjabuild-version) cmake -Bbuild-cmake cmake --build build-cmake -j$(numthreads) @@ -835,11 +937,54 @@ $(ibidir)/ninjabuild-$(ninjabuild-version): $(ibidir)/cmake-$(cmake-version) echo "Ninja build system $(ninjabuild-version)" > $@ $(ibidir)/openblas-$(openblas-version): + +# Import the tarball. tarball=openblas-$(openblas-version).tar.lz $(call import-source, $(openblas-url), $(openblas-checksum)) - if [ x$(on_mac_os) = xyes ]; then export CC=clang; fi + +# If 'gcc' is actually a symbolic link to 'clang', then print the +# following message to guide the user on how to prepare their +# environment. + if $$CC --version | grep clang &> /dev/null; then + +# Print message. + echo; echo + printf "*********************************************\n" + printf "OpenBlas could not be installed with Clang!\n\n" + printf "In order to install OpenBLAS, GCC and Gfortran are " + printf "needed. You can take the following steps to fix " + printf "this:\n\n" + printf "1. Install GCC with Homebrew.\n" + printf "2. Get the full path of 'gcc', 'g++', and 'gfortran' " + printf "(for example with 'which gcc').\n" + printf "3. Comment the 'exit 1' and un-comment (and edit) the " + printf "'export's that are below this message (in the " + printf "'openblas' rule of " + printf "'reproduce/software/make/high-level.mk', within the " + printf "'openblas' rule):\n\n" + printf " exit 1\n" + printf " #export CC=/opt/homebrew/bin/gcc-XX\n" + printf " #export CXX=/opt/homebrew/bin/g++-XX\n" + printf " #export FC=/opt/homebrew/bin/gfortran-XX\n\n" + printf " with the lines:\n\n" + printf " #exit 1\n" + printf " export CC=/opt/homebrew/bin/gcc-15\n" + printf " export CXX=/opt/homebrew/bin/g++-15\n" + printf " export FC=/opt/homebrew/bin/gfortran-15\n\n" + printf "5. Re-configure Maneage.\n" + printf "*********************************************\n" + echo; echo + +# Parts to modify when the message above is printed. + exit 1 + #export CC=/opt/homebrew/bin/gcc-XX + #export CXX=/opt/homebrew/bin/g++-XX + #export FC=/opt/homebrew/bin/gfortran-XX + fi + +# Do the build. cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd openblas-$(openblas-version) $(shsrcdir)/prep-source.sh $(ibdir) make -j$(numthreads) @@ -904,6 +1049,209 @@ $(ibidir)/tides-$(tides-version): cp $(dtexdir)/tides.tex $(ictdir)/ echo "TIDES $(tides-version) \citep{tides}" > $@ +$(ibidir)/sqlite-$(sqlite-version): $(ibidir)/tcl-$(tcl-version) + tarball=sqlite-$(sqlite-version).tar.lz + $(call import-source, $(sqlite-url), $(sqlite-checksum)) + $(call gbuild, sqlite-$(sqlite-version),, \ + --enable-all --enable-static --enable-shared) + echo "Sqlite $(sqlite-version)" > $@ + +$(ibidir)/tcl-$(tcl-version): $(ibidir)/zlib-$(zlib-version) \ + $(ibidir)/tzdb-$(tzdb-version) + +# If both the tcl and tk packages are needed, then they must be built +# together, with identical version numbers. Thus, for simplicity in +# the build system, whether or not we build only tcl, or both tcl and +# tk, we must check that the versions are identical. + if [ "x$(tcl-version)" != "x$(tk-version)" ]; then + printf "Error: This build system requires identical tcl and " + printf "tk versions to be selected **even if** tk is not " + printf "going to be built. However, tcl-version=$(tcl-version) " + printf "while tk-version=$(tk-version)\n" + exit 1 + fi + +# Prepare the input tarball. + tarball=tcl-$(tcl-version).tar.lz + $(call import-source, $(tcl-url), $(tcl-checksum)) + +# Setup the build directory + cd $(ddir) + unpackdir_tcl=tcl-$(tcl-version) + rm -rf $${unpackdir_tcl} + +# Install TCL + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions + cd - # return to the top maneage directory + cd $(ddir)/$${unpackdir_tcl}/unix + $(shsrcdir)/prep-source.sh $(ibdir) + ./configure --prefix="$(idir)" + make -j$(numthreads) + make install + cd - # return to the top maneage directory + +# Setup the 'tclsh' symbolic link. + maj_min_version=$$(printf $(tcl-version)| \ + sed -e "s;^\([0-9]*\.[0-9]*\).*\';\1;") + printf "Will symlink tclsh$${maj_min_version} as tclsh ...\n" + cd $(ibdir) + rm -fv tclsh && ln -sv tclsh$${maj_min_version} tclsh + cd - + cd $(ddir) + +# Finalize. + if (printf "$$unpackdir_tcl" | grep "[a-z][a-z]"); then \ + rm -fr $$unpackdir_tcl + fi + echo "Tcl $(tcl-version)" > $(ibidir)/tcl-$(tcl-version) + + +# The tk library needs the X11 library for building GUIs, and should not +# normally be needed for a non-interactive project. However, GUIs built +# with tk might be convenient during development of a project, since visual +# checks within the maneage shell may be useful. +# +# We need to rebuild tcl temporarily in the temporary build area in order +# to build tk. However, we want the tk target to be dependent on tcl as a +# prerequisite to avoid the possibility of parallel builds that risk +# conflicting within the build area because of duplicate use of the same +# tcl package directory after untarring the archive. +$(ibidir)/tk-$(tk-version): $(ibidir)/zip-$(zip-version) \ + $(ibidir)/tcl-$(tcl-version) \ + $(ibidir)/libx11-$(libx11-version) \ + +# Prepare the tarball. + tarball=tk-$(tk-version).tar.lz + $(call import-source, $(tk-url), $(tk-checksum)) + +# If both the tcl and tk packages are needed, then they must be built +# together, with identical version numbers. Thus, for simplicity in +# the build system, whether or not we build only tcl, or both tcl and +# tk, we must check that the versions are identical. + if [ "x$(tcl-version)" != "x$(tk-version)" ]; then + printf "Error: This build system requires identical tcl and " + printf "tk versions to be selected **even if** tk is not " + printf "going to be built. However, tcl-version=$(tcl-version) " + printf "while tk-version=$(tk-version)\n" + exit 1 + fi + +# Make sure no temporary directory exists. + cd $(ddir) + unpackdir=tk-$(tk-version) + if (printf "$$unpackdir" | grep "[a-z][a-z]"); then + rm -rf $$unpackdir + fi + +# Install TK: zipfs is being disabled because of the problem with +# installing zip (see the comments above zip). + cd $(ddir) + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions + cd $$unpackdir/unix + $(shsrcdir)/prep-source.sh $(ibdir) + ./configure --prefix="$(idir)" --with-tcl=$(ildir) + make -j$(numthreads) + make install + +# Finalize. + cd $(ddir) + if (printf "$$unpackdir" | grep "[a-z][a-z]"); then \ + rm -fr $$unpackdir; fi + echo "Tk $(tk-version)" > $(ibidir)/tk-$(tk-version) + + + +# WARNING: Dependence on glibc: tzdb (time zone data base) uses the 'zic' +# program from the GNU C library. Until Maneage builds its own glibc, it is +# imported from the host operating system in the 'low-level-links' step of +# 'basic.mk'. +$(ibidir)/tzdb-$(tzdb-version): + +# Import the tarball + tarball=tzdb-$(tzdb-version).tar.lz + $(call import-source, $(tzdb-url), $(tzdb-checksum)) + +# Unpack the tarball. + cd $(ddir) + unpackdir=tzdb-$(tzdb-version) + rm -rf $$unpackdir + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions + cd $$unpackdir + +# Make necessary corrections. + TZGEN=$(strip $(ddir))/$$unpackdir/tzgen + printf "\n\n\n_____TZGEN=$${TZGEN}_____\n\n\n" + make AWK=gawk BACKWARD="backward" PACKRATDATA=backzone \ + PACKRATLIST=zone.tab VERSION_DEPS= tzdata.zi leapseconds + +# Build the timezone data +# +# The '-b fat' option was created in glibc by around 2020, when +# '-b slim' became the default. Allowing the default of '-b slim' +# could potentially lead to a #Y2038 bug [3]. +# +# Some other hacks are also needed for older glibc. +# +# [1] https://mm.icann.org/pipermail/tz-announce/2020-October/000059.html +# [2] https://github.com/stub42/pytz/issues/48 +# [3] https://en.wikipedia.org/wiki/Year_2038_problem + BFAT_OPTION=$$(if (zic -b fat 2>/dev/null); then \ + printf "%sb fat" "-"; else printf ""; fi) + if [ "x$${BFAT_OPTION}" = "x" ]; then \ + sed -e 's/Sa\([^a-zA-Z]\)/Sat\1/g' \ + -e 's/Su\([^a-zA-Z]\)/Sun\1/g' \ + -e 's/^\(.\|..\) Sat/\1 Sa/' \ + -e "s/lastSu\'/lastSun/" tzdata.zi \ + | grep -v "^L" \ + > tzdata.zi.3letter.days + zic $${BFAT_OPTION} -d $${TZGEN} -L /dev/null \ + tzdata.zi.3letter.days + zic $${BFAT_OPTION} -d $${TZGEN}/right -L leapseconds \ + tzdata.zi.3letter.days + else + zic $${BFAT_OPTION} -d $${TZGEN} -L /dev/null tzdata.zi + zic $${BFAT_OPTION} -d $${TZGEN}/right -L leapseconds \ + tzdata.zi + fi + +# Replace hardlinks by symlinks + if [ "x$${BFAT_OPTION}" = "x" ]; then + +# Older method of creating symbolic links: + grep '^L ' $(ddir)/$$unpackdir/tzdata.zi \ + | while read L target name ; do + absolute_name="$${TZGEN}/$$name" + relative_target=$$(realpath -m -s \ + --relative-to="$${absolute_name%/*}" \ + "$${TZGEN}/$$target") + maybe_newdir=$$(printf $$name | sed -e "s;/[^/]*\';;") + mkdir -p $${TZGEN}/$${maybe_newdir} + mkdir -p $${TZGEN}/right/$${maybe_newdir} + ln -sf "$$relative_target" "$${TZGEN}/$$name" + ln -sf "$$relative_target" "$${TZGEN}/right/$$name" + done + + else + +# More recent (2025-05-25) method of creating symbolic links: + grep '^L ' $(ddir)/$$unpackdir/tzdata.zi \ + | while read L target name ; do + absolute_name="$${TZGEN}/$$name" + relative_target=$$(realpath -m -s \ + --relative-to="$${absolute_name%/*}" \ + "$${TZGEN}/$$target") + ln -sf "$$relative_target" "$${TZGEN}/$$name" + ln -sf "$$relative_target" "$${TZGEN}/right/$$name" + done + fi + +# Generate a posixrules file (with a UTC timezone) and install. + ln -s Etc/UTC $${TZGEN}/posixrules + mkdir -p $(idir)/share/zoneinfo + cp -rv $${TZGEN}/* $(idir)/share/zoneinfo/ + cd $(ddir); rm -rf $$unpackdir + echo "IANA Time Zone Database (tzdb) $(tzdb-version)" > $@ + $(ibidir)/valgrind-$(valgrind-version): tarball=valgrind-$(valgrind-version).tar.lz $(call import-source, $(valgrind-url), $(valgrind-checksum)) @@ -959,7 +1307,19 @@ $(ibidir)/wcslib-$(wcslib-version): $(ibidir)/cfitsio-$(cfitsio-version) tarball=wcslib-$(wcslib-version).tar.lz $(call import-source, $(wcslib-url), $(wcslib-checksum)) +# macOS arm64 fix: reserve extra space in binary headers so +# install_name_tool (called bellow, after installation) can safely +# update library paths. Without this, we get the following error: +# "larger updated load commands do not fit". SDKROOT is also +# necessary for system libraries. + if [ x$(on_mac_os) = xyes ]; then + sdk=$$(xcrun --show-sdk-path) + LDFLAGS="$$LDFLAGS -isysroot $$sdk" + export LDFLAGS="$$LDFLAGS -Wl,-headerpad_max_install_names" + fi + # Build WCSLIB while disabling some features: +# # - Fortran is disabled because as of May 2023, on macOS systems # where we do not install GCC (and thus a standard 'gfortran'), the # LLVM Fortran compiler is not complete and will cause a crash. If @@ -968,11 +1328,18 @@ $(ibidir)/wcslib-$(wcslib-version): $(ibidir)/cfitsio-$(cfitsio-version) # systems. Hopefully some time in the future, GCC will also be # install-able on macOS within Maneage or LLVM's 'gfortran' will # also be complete and will not cause a crash. +# # - PGPLOT is disabled because it has many manual steps in its # installation. Therefore, we currently do not build PGPlots in # Maneage. Once (if) it is added, we can remove '--without-pgplot'. +# +# - Flex is disabled because we have not made it a prerequisite for +# the 'wcslib' target, and we don't want the behaviour to differ +# between host OSes that may or may not have 'flex', or may have +# different versions of 'flex'. $(call gbuild, wcslib-$(wcslib-version), , \ LIBS="-pthread -lcurl -lm" \ + --disable-flex \ --without-pgplot \ --disable-fortran \ --with-cfitsiolib=$(ildir) \ @@ -1048,7 +1415,7 @@ $(ibidir)/astrometrynet-$(astrometrynet-version): \ # 'astrometrynet' cd $(ddir) rm -rf astrometry.net-$(astrometrynet-version) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd astrometry.net-$(astrometrynet-version) $(shsrcdir)/prep-source.sh $(ibdir) sed -e 's|cat /proc/cpuinfo|echo "Ignoring CPU info"|' \ @@ -1087,7 +1454,7 @@ $(ibidir)/cdsclient-$(cdsclient-version): tarball=cdsclient-$(cdsclient-version).tar.lz $(call import-source, $(cdsclient-url), $(cdsclient-checksum)) cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd cdsclient-$(cdsclient-version) $(shsrcdir)/prep-source.sh $(ibdir) touch * @@ -1120,7 +1487,7 @@ $(ibidir)/cmake-$(cmake-version): # Go into the unpacked directory and prepare CMake. cd $(ddir) rm -rf cmake-$(cmake-version) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd cmake-$(cmake-version) $(shsrcdir)/prep-source.sh $(ibdir) @@ -1182,20 +1549,26 @@ $(ibidir)/ghostscript-$(ghostscript-version): \ tarball=ghostscript-$(ghostscript-version).tar.lz $(call import-source, $(ghostscript-url), $(ghostscript-checksum)) +# On macOS we use the system compiler and linker. The system linker +# there doesn't support '--copy-dt-needed-entries', while the one on +# Linux crashes if we remove it. So we only activate it on macOS. + if [ x$(on_mac_os) = xno ]; then + export LDFLAGS="$$LDFLAGS -Wl,--copy-dt-needed-entries" + fi + # Unpack it and configure Ghostscript. The option # '-DPNG_ARM_NEON_OPT=0' prevents an arm64 'neon' library from being # required at compile time. cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd ghostscript-$(ghostscript-version) $(shsrcdir)/prep-source.sh $(ibdir) - ./configure --prefix=$(idir) \ - --disable-cups \ + ./configure --disable-cups \ + --prefix=$(idir) \ --enable-dynamic \ --disable-compile-inits \ - --disable-hidden-visibility \ - CFLAGS="-DPNG_ARM_NEON_OPT=0" \ - LDFLAGS=-Wl,--copy-dt-needed-entries + --disable-hidden-visibility \ + CFLAGS="-DPNG_ARM_NEON_OPT=0" # Build and install the program and the shared libraries. make V=1 -j$(numthreads) @@ -1209,9 +1582,9 @@ $(ibidir)/ghostscript-$(ghostscript-version): \ # Install the fonts. tar -xvf $(tdir)/ghostscript-fonts-std-$(ghostscript-fonts-std-version).tar.lz \ - -C $(idir)/share/ghostscript + -C $(idir)/share/ghostscript --no-same-owner --no-same-permissions tar -xvf $(tdir)/ghostscript-fonts-gnu-$(ghostscript-fonts-gnu-version).tar.lz \ - -C $(idir)/share/ghostscript + -C $(idir)/share/ghostscript --no-same-owner --no-same-permissions fc-cache -v $(idir)/share/ghostscript/fonts/ echo; echo "Ghostscript fonts added to Fontconfig."; echo; @@ -1220,18 +1593,61 @@ $(ibidir)/ghostscript-$(ghostscript-version): \ rm -rf ghostscript-$(ghostscript-version) echo "GPL Ghostscript $(ghostscript-version)" > $@ +# Gnuastro can optionally depend on libgit2, but it is not included as a +# dependency here for the two reasons below. If you would like to have it, +# add it as a dependency (its build instruction and dependencies are here +# already), then remove the '--without-libgit2' configure option) and also +# comment the following two generic configuration lines: 'outfitsnocommit' +# and 'outfitsnoconfig'. +# - Within Maneage, we have everything under Git already and users are +# expected to include the version in all their products. +# - libgit2 can only be built with CMake (which takes extremely long to +# compile: possibly even longer than GCC!). +# - Maneage is often run on HPCs that can have network connections +# between the storage and processer and the necessary libgit2 +# operations on every created file can have a performance impact. $(ibidir)/gnuastro-$(gnuastro-version): \ $(ibidir)/gsl-$(gsl-version) \ $(ibidir)/wcslib-$(wcslib-version) \ $(ibidir)/libjpeg-$(libjpeg-version) \ $(ibidir)/libtiff-$(libtiff-version) \ - $(ibidir)/libgit2-$(libgit2-version) \ $(ibidir)/ghostscript-$(ghostscript-version) + +# Generic installation. tarball=gnuastro-$(gnuastro-version).tar.lz $(call import-source, $(gnuastro-url), $(gnuastro-checksum)) - $(call gbuild, gnuastro-$(gnuastro-version), static, , \ - -j$(numthreads)) + $(call gbuild, gnuastro-$(gnuastro-version), static, \ + --without-libgit2, -j$(numthreads)) cp $(dtexdir)/gnuastro.tex $(ictdir)/ + +# Generally, besides the Git commit, we are also disabling the +# default mode of printing any type of metadata and versions of +# dependencies in output headers (through the installation-wide +# configuration file). This is done because within a large pipeline, +# Gnuastro is used to create many intermediate files (that are +# deleted shortly after being created) and it is not worth the +# overhread to keep this information in those intermediate products: +# it is the pipeline's responsibility to put them in the final +# outputs. We recommend to use the Gnuastro Fits program's keyword +# writing options to add as much contextual metadata on your final +# products as possible. Some tips: +# - The creation date is not good (because it is not reproducible +# and will make simple validation hard). The project's Commit +# should be used instead. +# - Define a keyword to keep the public URL of the repository of +# the Maneage'd project. In this way, a person who gets your +# final product can easily check that for all the information +# (including software versions and configuration options). + gconf=$(idir)/etc/gnuastro/gnuastro.conf + echo "" >> $$gconf + echo "# Maneage specific (see Gnuastro build rule for details)." \ + >> $$gconf + echo " outfitsnocommit = 1" >> $$gconf + echo " outfitsnoconfig = 1" >> $$gconf + echo " outfitsnodate = 1" >> $$gconf + echo " outfitsnoversions = 1" >> $$gconf + +# Final target. echo "GNU Astronomy Utilities $(gnuastro-version) \citep{gnuastro}" > $@ $(ibidir)/icu-$(icu-version): $(ibidir)/python-$(python-version) @@ -1248,7 +1664,7 @@ $(ibidir)/icu-$(icu-version): $(ibidir)/python-$(python-version) tarball=icu-$(icu-version).tar.lz $(call import-source, $(icu-url), $(icu-checksum)) cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions unpackdir=icu-$(icu-version) cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) @@ -1317,7 +1733,7 @@ $(ibidir)/imfit-$(imfit-version): \ cd $(ddir) unpackdir=imfit-$(imfit-version) rm -rf $$unpackdir - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) sed -i 's|/usr/local|$(idir)|g' SConstruct @@ -1349,6 +1765,30 @@ $(ibidir)/imfit-$(imfit-version): \ rm -rf $$unpackdir echo "Imfit $(imfit-version) \citep{imfit2015}" > $@ +# jq (to parse JSON files on the command-line) uses the external Oniguruma +# regular expression library that has ended development in 2025-04-24, see +# https://github.com/kkos/oniguruma. If this library ('libonig') is present +# on the host, jq will build with that: causing a leaking of os environment +# into the Maneage'd environment. Fortunately, jq maintains a copy of this +# library in its own source (within its tarball) and will build it with the +# '--with-oniguruma=builtin' configuration option. +$(ibidir)/jq-$(jq-version): + tarball=jq-$(jq-version).tar.lz + $(call import-source, $(jq-url), $(jq-checksum)) + $(call gbuild, jq-$(jq-version), static, \ + --with-oniguruma=builtin, V=1) + echo "jq $(jq-version)" > $@ + +# This is the LIGO lscsoft package 'metaio' [1] (to be distinguished from +# some other packages with the same name). +# +# [1] https://git.ligo.org/lscsoft/metaio +$(ibidir)/metaio-$(metaio-version): + tarball=metaio-$(metaio-version).tar.lz + $(call import-source, $(metaio-url), $(metaio-checksum)) + $(call gbuild, metaio-$(metaio-version), static,, V=1) + echo "Lscsoft Metaio $(metaio-version)" > $@ + # Minizip 1.x is actually distributed within zlib. It doesn't have its own # independent tarball. So we need a custom build, which include the GNU # Autotools (Autoconf and Automake). Note that Minizip 2.x isn't like this @@ -1365,7 +1805,8 @@ $(ibidir)/minizip-$(minizip-version): $(ibidir)/automake-$(automake-version) unpackdir=minizip-$(minizip-version) rm -rf $$unpackdir mkdir $$unpackdir - tar -xf $(tdir)/$$tarball -C$$unpackdir --strip-components=1 + tar -xf $(tdir)/$$tarball -C$$unpackdir --strip-components=1 \ + --no-same-owner --no-same-permissions cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) ./configure --prefix=$(idir) @@ -1427,7 +1868,7 @@ $(ibidir)/netpbm-$(netpbm-version): \ cd $(ddir) unpackdir=netpbm-$(netpbm-version) rm -rf $$unpackdir - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) @@ -1454,6 +1895,25 @@ $(ibidir)/netpbm-$(netpbm-version): \ rm -rf $$unpackdir echo "Netpbm $(netpbm-version)" > $@ +$(ibidir)/parallel-$(parallel-version): + +# Prepare the tarball and build it. + tarball=parallel-$(parallel-version).tar.lz + $(call import-source, $(parallel-url), $(parallel-checksum)) + $(call gbuild, parallel-$(parallel-version), static) + +# GNU Parallel requires the users to acknowledge that they will cite +# it. Otherwise, it prints a warning on every run: reminding the user +# to cite it. In Maneage, we make sure that all software that request +# citation are cited automatically, so there is no need for a +# Maneager to get this warning and we can automatically ensure the +# developer that it will be cited. + echo "will cite" | $(ibdir)/parallel --citation + +# Copy the BibTeX source of the citation and finish. + cp -pv $(dtexdir)/parallel.tex $(ictdir)/ + echo "GNU Parallel $(parallel-version) \citep{parallel}" > $@ + $(ibidir)/patch-$(patch-version): tarball=patch-$(patch-version).tar.gz $(call import-source, $(patch-url), $(patch-checksum)) @@ -1472,6 +1932,13 @@ $(ibidir)/pcre-$(pcre-version): , V=1 -j$(numthreads)) echo "Perl Compatible Regular Expressions $(pcre-version)" > $@ +$(ibidir)/pcre2-$(pcre2-version): + tarball=pcre2-$(pcre2-version).tar.lz + $(call import-source, $(pcre2-url), $(pcre2-checksum)) + $(call gbuild, pcre2-$(pcre2-version), static) + echo "Perl Compatible Regular Expressions 2 $(pcre2-version)" > $@ + + # On macOS 12.3 Monterey with AppleClang 13.1.6.13160021, Plplot 5.15.0 # needs the 'finite' function of 'math.h' which has been deprecated in # macOS. By manually adding "#define finite isfinite" in 'math.h' like @@ -1547,7 +2014,7 @@ $(ibidir)/scons-$(scons-version): $(ibidir)/python-$(python-version) cd $(ddir) unpackdir=scons-$(scons-version) rm -rf $$unpackdir - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) @@ -1593,7 +2060,7 @@ $(ibidir)/sextractor-$(sextractor-version): \ unpackdir=sextractor-$(sextractor-version) cd $(ddir) rm -rf $$unpackdir - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$unpackdir # See comment above 'missfits' for '-fcommon'. @@ -1638,15 +2105,13 @@ $(ibidir)/swarp-$(swarp-version): $(ibidir)/fftw-$(fftw-version) cp $(dtexdir)/swarp.tex $(ictdir)/ echo "SWarp $(swarp-version) \citep{swarp}" > $@ -$(ibidir)/swig-$(swig-version): - -# Option --without-pcre was a suggestion once the configure step was -# tried and it failed. It was not recommended but it works! pcr is a -# dependency of swig +$(ibidir)/swig-$(swig-version): \ + $(ibidir)/pcre2-$(pcre2-version) +# Not disabling pcre, since PCRE is needed for lalsuite (python). tarball=swig-$(swig-version).tar.lz $(call import-source, $(swig-url), $(swig-checksum)) $(call gbuild, swig-$(swig-version), static, \ - --without-pcre --without-tcl) + --without-tcl) echo "Swig $(swig-version)" > $@ # The disables: @@ -1694,7 +2159,7 @@ $(ibidir)/util-linux-$(util-linux-version): \ # explained above). As shown below, later, we'll put a symbolic link # of all the necessary binaries in the main '$(idir)/bin'. cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd util-linux-$(util-linux-version) $(shsrcdir)/prep-source.sh $(ibdir) @@ -1713,8 +2178,19 @@ $(ibidir)/util-linux-$(util-linux-version): \ # manually by commenting the relevant files in the # 'configure.ac'. sed -e's|UL_BUILD_INIT(\[mkswap\], \[yes\])|UL_BUILD_INIT(\[mkswap\], \[no\])|' \ + -e's|UL_BUILD_INIT(\[bits\], \[yes\])|UL_BUILD_INIT(\[bits\], \[no\])|' \ -i configure.ac +# 'autogen.sh' checks for dependencies, but 'flex' (that is not built +# in Maneage) is not needed when compiling from tarball (see: +# https://github.com/util-linux/util-linux/pull/2531#issuecomment-1798020594) + sed -i '/flex/d' autogen.sh + ./autogen.sh + +# The 'autogen.sh' call resets the 'SHELL' (that was corrected in +# 'prep-source.sh') to '/bin/sh'. Therefore, we need to correct it. + sed -i -e's|SHELL = /bin/sh|SHELL = $(SHELL)|' po/Makefile.in.in + # Having updated 'configure.ac', we need to re-generate the # './configure' script with 'autoreconf' (which is part of Autoconf # and needs Automake; hence why they are dependencies. @@ -1724,7 +2200,6 @@ $(ibidir)/util-linux-$(util-linux-version): \ export CONFIG_SHELL=$(ibdir)/bash ./configure --prefix=$(idircustom)/util-linux \ --disable-dependency-tracking \ - --enable-libmount-support-mtab \ --disable-silent-rules \ --disable-liblastlog2 \ --disable-mountpoint \ @@ -1761,6 +2236,7 @@ $(ibidir)/util-linux-$(util-linux-version): \ $(ibidir)/xlsxio-$(xlsxio-version): \ $(ibidir)/cmake-$(cmake-version) \ $(ibidir)/expat-$(expat-version) \ + $(ibidir)/libbsd-$(libbsd-version) \ $(ibidir)/minizip-$(minizip-version) tarball=xlsxio-$(xlsxio-version).tar.gz $(call import-source, $(xlsxio-url), $(xlsxio-checksum)) @@ -1799,10 +2275,11 @@ $(ibidir)/vim-$(vim-version): tarball=vim-$(vim-version).tar.lz $(call import-source, $(vim-url), $(vim-checksum)) cd $(ddir) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions unpackdir=vim-$(vim-version) cd $(ddir)/$$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) + export CFLAGS="-std=gnu17 $$CFLAGS" ./configure --prefix=$(idir) \ --disable-canberra \ --enable-multibyte \ @@ -1818,26 +2295,69 @@ $(ibidir)/vim-$(vim-version): rm -rf $$unpackdir echo "VIM $(vim-version)" > $@ -$(ibidir)/unzip-$(unzip-version): $(ibidir)/gzip-$(gzip-version) +$(ibidir)/unzip-$(unzip-version): + +# Prepare the tarball and build directory. tarball=unzip-$(unzip-version).tar.lz $(call import-source, $(unzip-url), $(unzip-checksum)) - $(call gbuild, unzip-$(unzip-version), static,, \ - -f unix/Makefile generic \ - CFLAGS="-DBIG_MEM -DMMAP",,pwd, \ - -f unix/Makefile generic \ - BINDIR=$(ibdir) MANDIR=$(idir)/man/man1 ) + +# Unpack and go into the directory. + export LDFLAGS="$$LDFLAGS -static" + cd $(ddir) + rm -fr unzip-$(unzip-version)/ + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions + cd unzip-$(unzip-version)/ + +# Remove reference to the old 'gmtime' function and include the +# system's 'time.h instead. + sed -i -e's|struct tm *gmtime(), *localtime();|#include <time.h>|' \ + -e'/gmtime/d; /localtime/d' unix/unxcfg.h + +# The 'NO_DIR' variable causes conflicts: the unzip code has its own +# internal fallback implementation called 'NO_DIR'. The modern Linux +# system already provides a standard implementation using +# 'dirent.h'. These two systems conflict with each other. This causes +# compilation errors such as conflicting types for DIR and invalid +# use of DIR. + sed -i -e's/-DNO_DIR//g' unix/configure + sed -i -e's/#ifdef NO_DIR/#if 0/g' unix/unix.c + sed -i -e's/-DNO_DIR//g' -e's/-U_NO_DIR//g' unix/Makefile + +# Since unzip is no longer maintained, it uses an old C standard. + export CFLAGS="-std=gnu89 -fcommon" + +# Build and install + make SHELL=$(ibdir)/bash -f unix/Makefile generic + make SHELL=$(ibdir)/bash install -f unix/Makefile \ + generic BINDIR=$(ibdir) MANDIR=$(idir)/man/man1 + +# Go back to the top build directory, clean up and finalize. + cd $(ddir) + rm -rf unzip-$(unzip-version)/ echo "Unzip $(unzip-version)" > $@ -$(ibidir)/zip-$(zip-version): $(ibidir)/gzip-$(gzip-version) +$(ibidir)/zip-$(zip-version): + +# Prepare the tarball and build directory. tarball=zip-$(zip-version).tar.lz $(call import-source, $(zip-url), $(zip-checksum)) - $(call gbuild, zip-$(zip-version), static,, \ - -f unix/Makefile generic \ - CFLAGS="-DBIG_MEM -DMMAP",,pwd, \ - -f unix/Makefile generic \ - BINDIR=$(ibdir) MANDIR=$(idir)/man/man1 ) - echo "Zip $(zip-version)" > $@ +# Unpack and go into the directory. + export LDFLAGS="$$LDFLAGS -static" + cd $(ddir) + rm -fr zip-$(zip-version)/ + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions + cd zip-$(zip-version)/ + +# Build and install + make SHELL=$(ibdir)/bash -f unix/Makefile generic + make SHELL=$(ibdir)/bash install -f unix/Makefile \ + generic BINDIR=$(ibdir) MANDIR=$(idir)/man/man1 + +# Go back to the top build directory, clean up and finalize. + cd $(ddir) + rm -rf zip-$(zip-version)/ + echo "Zip $(zip-version)" > $@ @@ -1869,7 +2389,8 @@ $(itidir)/texlive-ready-tlmgr: reproduce/software/config/texlive.conf @topdir=$$(pwd) cd $(ddir) rm -rf install-tl-* - tar -xf $(tdir)/install-tl-unx.tar.gz + tar -xf $(tdir)/install-tl-unx.tar.gz \ + --no-same-owner --no-same-permissions cd install-tl-* $(shsrcdir)/prep-source.sh $(ibdir) sed -e's|@installdir[@]|$(idir)|g' \ @@ -1956,7 +2477,8 @@ $(itidir)/texlive-ready-tlmgr: reproduce/software/config/texlive.conf "$(backupservers)"; then cd $(ddir) rm -rf install-tl-* - tar -xf $(tdir)/install-tl-unx.tar.gz + tar -xf $(tdir)/install-tl-unx.tar.gz \ + --no-same-owner --no-same-permissions cd install-tl-* $(shsrcdir)/prep-source.sh $(ibdir) sed -e's|@installdir[@]|$(idir)|g' \ @@ -2014,7 +2536,7 @@ $(itidir)/texlive: reproduce/software/config/texlive-packages.conf \ # We do not build TeXLive from source and for its installation it # downloads components from the web internally; and those # components can use '/bin/sh' (which needs 'sys_library_sh_path'). - export LD_LIBRARY_PATH="$(sys_library_sh_path):$$LD_LIBRARY_PATH" + export LD_LIBRARY_PATH="$(sys_library_sh_path)" # To update itself, tlmgr needs a backup directory. backupdir=$(idir)/texlive/backups @@ -2043,7 +2565,6 @@ $(itidir)/texlive: reproduce/software/config/texlive-packages.conf \ # files (this is because we do no yet install LaTeX from source): cdir=$$(pwd) cd $(idir)/texlive - $(shsrcdir)/prep-source.sh $(ibdir) cd $$cdir # Get all the necessary versions. diff --git a/reproduce/software/make/python.mk b/reproduce/software/make/python.mk index c017fd2..6153f0a 100644 --- a/reproduce/software/make/python.mk +++ b/reproduce/software/make/python.mk @@ -12,8 +12,8 @@ # # ------------------------------------------------------------------------ # -# Copyright (C) 2019-2025 Raul Infante-Sainz <infantesainz@gmail.com> -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -83,7 +83,11 @@ $(ibidir)/libffi-$(libffi-version): fi echo "Libffi $(libffi-version)" > $@ -$(ibidir)/python-$(python-version): $(ibidir)/libffi-$(libffi-version) +# 2025-05-26 '--with-system-ffi' seems to be obsolete +# TODO: For a recommended list of dependencies, see +# https://github.com/pyenv/pyenv/wiki#suggested-build-environment +$(ibidir)/python-$(python-version): $(ibidir)/libffi-$(libffi-version) \ + $(ibidir)/sqlite-$(sqlite-version) # Download the source. tarball=python-$(python-version).tar.lz @@ -99,7 +103,7 @@ $(ibidir)/python-$(python-version): $(ibidir)/libffi-$(libffi-version) # Unpack the tarball (see below for the necessary modification). cd $(ddir) unpackdir=python-$(python-version) - tar -xf $(tdir)/$$tarball + tar -xf $(tdir)/$$tarball --no-same-owner --no-same-permissions cd $$unpackdir $(shsrcdir)/prep-source.sh $(ibdir) @@ -215,7 +219,7 @@ pybuild = cd $(ddir); \ packagedir=$(strip $(2)); \ if (printf "$$packagedir" | grep "[a-z][a-z]"); then rm -rf $$packagedir; fi; \ printf "\nStarting to install python package with maneage pybuild rule: $(4)\n ..."; \ - if ! $(1) $(tdir)/$$tarball; then \ + if ! $(1) $(tdir)/$$tarball --no-same-owner --no-same-permissions; then \ echo; echo "Tar error"; exit 1; \ fi; \ cd $$packagedir; \ @@ -346,12 +350,27 @@ $(ipydir)/astropy-$(astropy-version): \ rm -fv $(idir)/bin/fits{diff,check,header,info,2bitmap} rm -fv $(idir)/bin/{samp_hub,showtable,volint,wcslint} +# Uncomment the 'export' command below only when building from a git +# tarball, e.g. for testing if a bug still occurs: +# export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_ASTROPY=7.2.0.87 + # Do the basic build. $(call pybuild, tar -xf, astropy-$(astropy-version),,, \ GPEP517) cp -pv $(dtexdir)/astropy.tex $(ictdir)/ echo "Astropy $(astropy-version) \citep{astropy2013,astropy2018}" > $@ +$(ipydir)/astropy-healpix-$(astropy-healpix-version): \ + $(ipydir)/extension-helpers-$(extension-helpers-version) \ + $(ipydir)/numpy-$(numpy-version) + tarball=astropy-healpix-$(astropy-healpix-version).tar.lz + $(call import-source, $(astropy-healpix-url), \ + $(astropy-healpix-checksum)) + $(call pybuild, tar -xf, \ + astropy-healpix-$(astropy-healpix-version),,, \ + GPEP517) + echo "Astropy-healpix $(astropy-healpix-version)" > $@ + $(ipydir)/astropy-iers-data-$(astropy-iers-data-version): \ $(ipydir)/setuptools-$(setuptools-version) tarball=astropy-iers-data-$(astropy-iers-data-version).tar.lz @@ -360,7 +379,7 @@ $(ipydir)/astropy-iers-data-$(astropy-iers-data-version): \ $(call pybuild, tar -xf, \ astropy-iers-data-$(astropy-iers-data-version),,, \ GPEP517) - echo "Astropy-Iers-Data $(astropy-iers-data-version)" > $@ + echo "Astropy-IERS-data $(astropy-iers-data-version)" > $@ $(ipydir)/beautifulsoup4-$(beautifulsoup4-version): \ $(ipydir)/soupsieve-$(soupsieve-version) @@ -400,6 +419,13 @@ $(ipydir)/chardet-$(chardet-version): \ $(call pybuild, tar -xf, chardet-$(chardet-version), , \ Chardet $(chardet-version)) +$(ipydir)/colorama-$(colorama-version): \ + $(ipydir)/hatchling-$(hatchling-version) + tarball=colorama-$(colorama-version).tar.lz + $(call import-source, $(colorama-url), $(colorama-checksum)) + $(call pybuild, tar -xf, \ + colorama-$(colorama-version),,, GPEP517) + $(ipydir)/contourpy-$(contourpy-version): \ $(ipydir)/pybind11-$(pybind11-version) \ $(ipydir)/meson-python-$(meson-python-version) @@ -417,6 +443,15 @@ $(ipydir)/corner-$(corner-version): $(ipydir)/matplotlib-$(matplotlib-version) cp $(dtexdir)/corner.tex $(ictdir)/ echo "Corner $(corner-version) \citep{corner}" > $@ +$(ipydir)/cosmodesi-mpytools-$(cosmodesi-mpytools-version): \ + $(ipydir)/numpy-$(numpy-version) \ + $(ipydir)/scipy-$(scipy-version) + tarball=cosmodesi-mpytools-$(cosmodesi-mpytools-version).tar.lz + $(call import-source, $(cosmodesi-mpytools-url), $(cosmodesi-mpytools-checksum)) + $(call pybuild, tar -xf, cosmodesi-mpytools-$(cosmodesi-mpytools-version), , \ + Cosmodesi-mpytools $(cosmodesi-mpytools-version), GPEP517) + echo "Cosmodesi-mpytools $(cosmodesi-mpytools-version)" > $@ + $(ipydir)/cppy-$(cppy-version): \ $(ipydir)/setuptools-scm-$(setuptools-scm-version) tarball=cppy-$(cppy-version).tar.lz @@ -482,6 +517,15 @@ $(ipydir)/entrypoints-$(entrypoints-version): \ $(call pybuild, tar -xf, entrypoints-$(entrypoints-version), , \ EntryPoints $(entrypoints-version)) +$(ipydir)/exceptiongroup-$(exceptiongroup-version): \ + $(ipydir)/setuptools-$(setuptools-version) \ + $(ipydir)/flit-scm-$(flit-scm-version) + tarball=exceptiongroup-$(exceptiongroup-version).tar.lz + $(call import-source, $(exceptiongroup-url), $(exceptiongroup-checksum)) + $(call pybuild, tar -xf, exceptiongroup-$(exceptiongroup-version), , \ + Exceptiongroup $(exceptiongroup-version), GPEP517) + + $(ipydir)/extension-helpers-$(extension-helpers-version): \ $(ipydir)/setuptools-scm-$(setuptools-scm-version) tarball=extension-helpers-$(extension-helpers-version).tar.lz @@ -507,6 +551,16 @@ $(ipydir)/flit-core-$(flit-core-version): \ $(call pybuild, tar -xf, flit-core-$(flit-core-version), , \ Flit-core $(flit-core-version), GPEP517) + +$(ipydir)/flit-scm-$(flit-scm-version): \ + $(ipydir)/flit-core-$(flit-core-version) \ + $(ipydir)/setuptools-scm-$(setuptools-scm-version) + tarball=flit-scm-$(flit-scm-version).tar.lz + $(call import-source, $(flit-scm-url), $(flit-scm-checksum)) + $(call pybuild, tar -xf, flit-scm-$(flit-scm-version), , \ + flit-scm $(flit-scm-version), GPEP517) + + # Although cython is not an obligatory prerequisite of fonttools, we force # it as a prerequisite for reproducibility; otherwise build parallelism may # lead to some builds with and some builds without cython, depending on how @@ -558,6 +612,23 @@ $(ipydir)/gpep517-$(gpep517-version): \ BOOT_GPEP517) echo "gpep517 $(gpep517-version)" > $@ +$(ipydir)/hatchling-$(hatchling-version): \ + $(ipydir)/packaging-$(packaging-version) \ + $(ipydir)/pathspec-$(pathspec-version) \ + $(ipydir)/pluggy-$(pluggy-version) \ + $(ipydir)/trove-classifiers-$(trove-classifiers-version) + tarball=hatchling-$(hatchling-version).tar.lz + $(call import-source, $(hatchling-url), $(hatchling-checksum)) + $(call pybuild, tar -xf, \ + hatchling-$(hatchling-version),,, GPEP517) + +$(ipydir)/hatch-vcs-$(hatch-vcs-version): \ + $(ipydir)/hatchling-$(hatchling-version) + tarball=hatch-vcs-$(hatch-vcs-version).tar.lz + $(call import-source, $(hatch-vcs-url), $(hatch-vcs-checksum)) + $(call pybuild, tar -xf, hatch-vcs-$(hatch-vcs-version), , \ + Hatch-vcs $(hatch-vcs-version), GPEP517) + $(ipydir)/h5py-$(h5py-version): \ $(ipydir)/six-$(six-version) \ $(ibidir)/hdf5-$(hdf5-version) \ @@ -572,15 +643,38 @@ $(ipydir)/h5py-$(h5py-version): \ $(call pybuild, tar -xf, h5py-$(h5py-version), , \ h5py $(h5py-version)) -# 'healpy' is actually installed as part of the HEALPix package. It will be -# installed with its C/C++ libraries if any other Python library is -# requested with HEALPix. So actually calling for 'healpix' (when 'healpix' -# is requested) is not necessary. But some users might not know about this -# and just ask for 'healpy'. To avoid confusion in such cases, we'll just -# set 'healpy' to be dependent on 'healpix' and not download any tarball -# for it, or write anything in the final target. -$(ipydir)/healpy-$(healpy-version): $(ibidir)/healpix-$(healpix-version) - touch $@ +# As of 2025-05-20, there are at least three python healpix packages at +# pypi.org, with versions: +# +# https://pypi.org/project/healpy 1.18.1 +# https://pypi.org/project/healpix 2024.2 +# https://pypi.org/project/astropy-healpix 1.1.2 +# +# Maneage provides both 'healpy' and 'astropy-healpix', independently. We +# ignore pypix 'healpix', whose name risks confusion with the C/C++/fortran +# version of healpix. +# +# For simplicity and error tracing, you should best install either healpy +# or astropy-healpix, not both. For example, astropy_healpix/healpy.py has +# an interface to provide 'healpy' type function names. However, it may be +# possible to use both if your python imports avoid namespace clashes. See +# the documentation and source code of the respective packages to study how +# they relate to one another. +# +# The 'healpy' package requires the C/C++ 'healpix' package to be built +# in order to use the shared 'libsharp' library. +$(ipydir)/healpy-$(healpy-version): \ + $(ibidir)/healpix-$(healpix-version) \ + $(ipydir)/matplotlib-$(matplotlib-version) \ + $(ipydir)/scipy-$(scipy-version) + tarball=healpy-$(healpy-version).tar.lz + $(call import-source, $(healpy-url), \ + $(healpy-checksum)) + $(call pybuild, tar -xf, \ + healpy-$(healpy-version),,, \ + GPEP517) + echo "Healpy $(healpy-version)" > $@ + $(ipydir)/html5lib-$(html5lib-version): \ $(ipydir)/six-$(six-version) \ @@ -612,6 +706,13 @@ $(ipydir)/jinja2-$(jinja2-version): $(ipydir)/markupsafe-$(markupsafe-version) $(call pybuild, tar -xf, jinja2-$(jinja2-version), , \ Jinja2 $(jinja2-version)) +$(ipydir)/joblib-$(joblib-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=joblib-$(joblib-version).tar.lz + $(call import-source, $(joblib-url), $(joblib-checksum)) + $(call pybuild, tar -xf, \ + joblib-$(joblib-version),,, GPEP517) + $(ipydir)/keyring-$(keyring-version): \ $(ipydir)/entrypoints-$(entrypoints-version) \ $(ipydir)/secretstorage-$(secretstorage-version) \ @@ -631,6 +732,48 @@ $(ipydir)/kiwisolver-$(kiwisolver-version): \ cp -pv $(dtexdir)/kiwisolver.tex $(ictdir)/ echo "Kiwisolver $(kiwisolver-version) \citep{cassowary2001}" > $@ +# 2025-05-21: As of Feb 2025, it appears the lscsoft group is not maintaining +# python-ligo-lw and ligo-segments; the LIGO Computing Group forked these +# to igwn-ligolw and igwn-segments, respectively [1]. +# [1] https://github.com/ahnitz/ligo-segments/issues/1 +$(ipydir)/igwn-segments-$(igwn-segments-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=igwn-segments-$(igwn-segments-version).tar.lz + $(call import-source, $(igwn-segments-url), $(igwn-segments-checksum)) + $(call pybuild, tar -xf, \ + igwn-segments-$(igwn-segments-version),, \ + Igwn-segments $(igwn-segments-version), GPEP517) + + +$(ipydir)/iniconfig-$(iniconfig-version): \ + $(ipydir)/hatch-vcs-$(hatch-vcs-version) + tarball=iniconfig-$(iniconfig-version).tar.lz + $(call import-source, $(iniconfig-url), $(iniconfig-checksum)) + $(call pybuild, tar -xf, iniconfig-$(iniconfig-version), , \ + Iniconfig $(iniconfig-version), GPEP517) + +# Upstream this is called ligo.skymap with a full stop '.'; we replace it +# by a hyphen '-' to reduce the chance of regex errors. +# +# 2025-05-19 Warning: this might have some undetected dependencies. +$(ipydir)/ligo-skymap-$(ligo-skymap-version): \ + $(ibidir)/gsl-$(gsl-version) \ + $(ipydir)/tqdm-$(tqdm-version) \ + $(ibidir)/sqlite-$(sqlite3-version) \ + $(ipydir)/astropy-$(astropy-version) \ + $(ipydir)/lalsuite-$(lalsuite-version) \ + $(ipydir)/igwn-ligolw-$(igwn-ligolw-version) \ + $(ipydir)/igwn-segments-$(igwn-segments-version) \ + $(ipydir)/astropy-healpix-$(astropy-healpix-version) + tarball=ligo-skymap-$(ligo-skymap-version).tar.lz + $(call import-source, $(ligo-skymap-url), \ + $(ligo-skymap-checksum)) + $(call pybuild, tar -xf, \ + ligo-skymap-$(ligo-skymap-version),,, \ + GPEP517) + echo "ligo-skymap $(ligo-skymap-version)" > $@ + + $(ipydir)/lmfit-$(lmfit-version): \ $(ipydir)/six-$(six-version) \ $(ipydir)/scipy-$(scipy-version) \ @@ -644,6 +787,14 @@ $(ipydir)/lmfit-$(lmfit-version): \ $(call pybuild, tar -xf, lmfit-$(lmfit-version), , \ LMFIT $(lmfit-version)) +$(ipydir)/lscsoft-glue-$(lscsoft-glue-version): \ + $(ipydir)/gpep517-$(gpep517-version) \ + $(ipydir)/python-installer-$(python-installer-version) + tarball=lscsoft-glue-$(lscsoft-glue-version).tar.lz + $(call import-source, $(lscsoft-glue-url), $(lscsoft-glue-checksum)) + $(call pybuild, tar -xf, \ + lscsoft-glue-$(lscsoft-glue-version),,, GPEP517) + $(ipydir)/lsstdesccoord-$(lsstdesccoord-version): \ $(ipydir)/cffi-$(cffi-version) \ $(ipydir)/numpy-$(numpy-version) \ @@ -661,11 +812,19 @@ $(ipydir)/markupsafe-$(markupsafe-version): \ $(call pybuild, tar -xf, markupsafe-$(markupsafe-version), , \ MarkupSafe $(markupsafe-version)) +# As of 2026-04-19: Matplotlib needs internet to download and build its +# custom versions of 'freetype' and 'qhull'. The proposed option to 'python +# -m pip instsall' in [1] for using system libraries was tested, but did +# not work in this setup (our 'pybuild' uses 'python -m gpep517 +# install-from-source', not 'pip'). +# +# [1] https://matplotlib.org/stable/install/dependencies.html $(ipydir)/matplotlib-$(matplotlib-version): \ $(itidir)/texlive \ $(ipydir)/numpy-$(numpy-version) \ $(ipydir)/cycler-$(cycler-version) \ $(ipydir)/pillow-$(pillow-version) \ + $(ibidir)/freetype-$(freetype-version) \ $(ipydir)/fonttools-$(fonttools-version) \ $(ipydir)/contourpy-$(contourpy-version) \ $(ipydir)/kiwisolver-$(kiwisolver-version) \ @@ -706,12 +865,11 @@ $(ipydir)/meson-python-$(meson-python-version): \ echo "Meson-Python $(meson-python-version)" > $@ $(ipydir)/mpi4py-$(mpi4py-version): \ - $(ipydir)/gpep517-$(gpep517-version) \ $(ibidir)/openmpi-$(openmpi-version) \ - $(ipydir)/python-installer-$(python-installer-version) + $(ipydir)/cython-$(cython-version) tarball=mpi4py-$(mpi4py-version).tar.lz $(call import-source, $(mpi4py-url), $(mpi4py-checksum)) - $(call pybuild, tar -xf, mpi4py-$(mpi4py-version)) + $(call pybuild, tar -xf, mpi4py-$(mpi4py-version),,, GPEP517) cp $(dtexdir)/mpi4py.tex $(ictdir)/ echo "mpi4py $(mpi4py-version) \citep{mpi4py2011}" > $@ @@ -723,8 +881,17 @@ $(ipydir)/mpmath-$(mpmath-version): \ $(call pybuild, tar -xf, mpmath-$(mpmath-version), , \ mpmath $(mpmath-version)) -# Until 2025-02-22: we had 'export CFLAGS="--std=c99 $$CFLAGS"' before -# calling pybuild; but that doesn't seem to be necessary. +$(ipydir)/networkx-$(networkx-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=networkx-$(networkx-version).tar.lz + $(call import-source, $(networkx-url), $(networkx-checksum)) + $(call pybuild, tar -xf, \ + networkx-$(networkx-version),,, \ + GPEP517) + echo "networkx $(networkx-version)" > $@ + +# Numpy's MacOS builds using Xcode prior to 26.3 needed the following flags +# added to 'LDFLAGS': '-undefined dynamic_lookup -bundle'. $(ipydir)/numpy-$(numpy-version): \ $(ipydir)/cython-$(cython-version) \ $(ibidir)/openblas-$(openblas-version) \ @@ -732,9 +899,6 @@ $(ipydir)/numpy-$(numpy-version): \ $(ipydir)/meson-python-$(meson-python-version) tarball=numpy-$(numpy-version).tar.lz $(call import-source, $(numpy-url), $(numpy-checksum)) - if [ x$(on_mac_os) = xyes ]; then - export LDFLAGS="$(LDFLAGS) -undefined dynamic_lookup -bundle" - fi conf="$$(pwd)/reproduce/software/config/numpy-scipy.cfg" $(call pybuild, tar -xf, numpy-$(numpy-version),$$conf, \ Numpy $(numpy-version), GPEP517) @@ -748,6 +912,23 @@ $(ipydir)/packaging-$(packaging-version): \ $(call pybuild, tar -xf, packaging-$(packaging-version), , \ Packaging $(packaging-version), GPEP517) +$(ipydir)/pandas-$(pandas-version): \ + $(ipydir)/numpy-$(numpy-version) \ + $(ipydir)/versioneer-$(versioneer-version) \ + $(ipydir)/pytz-$(pytz-version) + tarball=pandas-$(pandas-version).tar.lz + $(call import-source, $(pandas-url), $(pandas-checksum)) + $(call pybuild, tar -xf, pandas-$(pandas-version), , \ + Pandas $(pandas-version), GPEP517) + echo "pandas $(pandas-version)" > $@ + +$(ipydir)/pathspec-$(pathspec-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=pathspec-$(pathspec-version).tar.lz + $(call import-source, $(pathspec-url), $(pathspec-checksum)) + $(call pybuild, tar -xf, \ + pathspec-$(pathspec-version),,, GPEP517) + $(ipydir)/pexpect-$(pexpect-version): \ $(ipydir)/gpep517-$(gpep517-version) \ $(ipydir)/python-installer-$(python-installer-version) @@ -773,6 +954,13 @@ $(ipydir)/pillow-$(pillow-version): $(ibidir)/libjpeg-$(libjpeg-version) \ # $(call pybuild, tar -xf, pip-$(pip-version), , \ # PiP $(pip-version)) +$(ipydir)/pluggy-$(pluggy-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=pluggy-$(pluggy-version).tar.lz + $(call import-source, $(pluggy-url), $(pluggy-checksum)) + $(call pybuild, tar -xf, \ + pluggy-$(pluggy-version),,, GPEP517) + $(ipydir)/ply-$(ply-version): \ $(ipydir)/gpep517-$(gpep517-version) \ $(ipydir)/python-installer-$(python-installer-version) @@ -828,6 +1016,13 @@ $(ipydir)/pyflakes-$(pyflakes-version): \ $(call pybuild, tar -xf, pyflakes-$(pyflakes-version), , \ pyflakes $(pyflakes-version)) +$(ipydir)/pygments-$(pygments-version): \ + $(ipydir)/hatchling-$(hatchling-version) + tarball=pygments-$(pygments-version).tar.lz + $(call import-source, $(pygments-url), $(pygments-checksum)) + $(call pybuild, tar -xf, pygments-$(pygments-version), , \ + Pygments $(pygments-version), GPEP517) + $(ipydir)/pyparsing-$(pyparsing-version): \ $(ipydir)/gpep517-$(gpep517-version) \ $(ipydir)/flit-core-$(flit-core-version) \ @@ -854,8 +1049,26 @@ $(ipydir)/pyproject-metadata-$(pyproject-metadata-version): \ $(call pybuild, tar -xf, \ pyproject-metadata-$(pyproject-metadata-version),,, GPEP517) + +$(ipydir)/pytest-$(pytest-version): \ + $(ipydir)/setuptools-scm-$(setuptools-scm-version) \ + $(ipydir)/colorama-$(colorama-version) \ + $(ipydir)/exceptiongroup-$(exceptiongroup-version) \ + $(ipydir)/iniconfig-$(iniconfig-version) \ + $(ipydir)/packaging-$(packaging-version) \ + $(ipydir)/pluggy-$(pluggy-version) \ + $(ipydir)/pygments-$(pygments-version) + tarball=pytest-$(pytest-version).tar.lz + $(call import-source, $(pytest-url), $(pytest-checksum)) + $(call pybuild, tar -xf, \ + pytest-$(pytest-version),,, \ + GPEP517) + echo "pytest $(pytest-version)" > $@ + + $(ipydir)/python-dateutil-$(python-dateutil-version): \ - $(ipydir)/setuptools-scm-$(setuptools-scm-version) + $(ipydir)/setuptools-scm-$(setuptools-scm-version) \ + $(ipydir)/six-$(six-version) tarball=python-dateutil-$(python-dateutil-version).tar.lz $(call import-source, $(python-dateutil-url), $(python-dateutil-checksum)) $(call pybuild, tar -xf, python-dateutil-$(python-dateutil-version), , \ @@ -885,6 +1098,37 @@ $(ipydir)/python-installer-$(python-installer-version): \ BOOT_INSTALLER) echo "Python-installer $(python-installer-version)" > $@ +# 2025-05-21: As of Feb 2025, it appears the lscsoft group is not maintaining +# python-ligo-lw and ligo-segments; the LIGO Computing Group forked these +# to igwn-ligolw and igwn-segments, respectively [1]. +# [1] https://github.com/ahnitz/ligo-segments/issues/1 +$(ipydir)/igwn-ligolw-$(igwn-ligolw-version): \ + $(ipydir)/setuptools-scm-$(setuptools-scm-version) + tarball=igwn-ligolw-$(igwn-ligolw-version).tar.lz + $(call import-source, $(igwn-ligolw-url), \ + $(igwn-ligolw-checksum)) + $(call pybuild, tar -xf, \ + igwn-ligolw-$(igwn-ligolw-version),,, \ + GPEP517) + echo "igwn-ligolw $(igwn-ligolw-version)" > $@ + +$(ipydir)/pyfftw-$(pyfftw-version): $(ipydir)/numpy-$(numpy-version) \ + $(ibidir)/fftw-$(fftw-version) + tarball=pyfftw-$(pyfftw-version).tar.lz + $(call import-source, $(pyfftw-url), $(pyfftw-checksum)) + $(call pybuild, tar -xf, pyfftw-$(pyfftw-version), , \ + Pyfftw $(pyfftw-version), GPEP517) + + +# Called 'fitsio' in pypi.org: https://pypi.org/project/fitsio +$(ipydir)/python3-fitsio-$(python3-fitsio-version): \ + $(ibidir)/patch-$(patch-version) \ + $(ipydir)/numpy-$(numpy-version) + tarball=python3-fitsio-$(python3-fitsio-version).tar.lz + $(call import-source, $(python3-fitsio-url), $(python3-fitsio-checksum)) + $(call pybuild, tar -xf, python3-fitsio-$(python3-fitsio-version), , \ + Python3 Fitsio $(python3-fitsio-version), GPEP517) + $(ipydir)/pythran-$(pythran-version): \ $(ipydir)/ply-$(ply-version) \ $(ipydir)/gast-$(gast-version) \ @@ -904,6 +1148,13 @@ $(ipydir)/pyyaml-$(pyyaml-version): \ $(call pybuild, tar -xf, pyyaml-$(pyyaml-version), , \ PyYAML $(pyyaml-version), GPEP517) +$(ipydir)/pytz-$(pytz-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=pytz-$(pytz-version).tar.lz + $(call import-source, $(pytz-url), $(pytz-checksum)) + $(call pybuild, tar -xf, pytz-$(pytz-version), , \ + Pytz $(pytz-version), GPEP517) + $(ipydir)/requests-$(requests-version): $(ipydir)/idna-$(idna-version) \ $(ipydir)/numpy-$(numpy-version) \ $(ipydir)/certifi-$(certifi-version) \ @@ -914,41 +1165,39 @@ $(ipydir)/requests-$(requests-version): $(ipydir)/idna-$(idna-version) \ $(call pybuild, tar -xf, requests-$(requests-version), , \ Requests $(requests-version)) -# 'pythran' is disabled in the build of Scipy because of complications it -# caused on some systems. We explicitly disable it using a preprocessor -# directive. 'Pythran' can in principle speed up compilation of scientific -# software [1][2]. +# Scipy building notes: +# +# 1. 'pythran' is disabled in the build of Scipy because of complications +# it caused on some systems. 'Pythran' can in principle speed up +# compilation of scientific software [1][2]. We explicitly disable it by +# modifying the source. Technically it should also be possible to pass +# the following option to the gpep517 build recipes with correct +# escaping of single and double quotes, but this has not tried as of +# 2025-02-25. +# --config-json='{"setup-args":"-Duse-pythran=false"}' +# +# 2. Scipy's MacOS builds using Xcode prior to 26.3 needed the following +# flags added to 'LDFLAGS': '-undefined dynamic_lookup -bundle'. +# # [1] https://pythran.readthedocs.io/en/latest -# [2] https://docs.scipy.org/doc/scipy-1.15.2/dev/roadmap-detailed.html +# [2] https://docs.scipy.org/doc/scipy-1.15.2/dev/roadmap-detailed.html#use-of-pythran $(ipydir)/scipy-$(scipy-version): \ $(ipydir)/numpy-$(numpy-version) \ $(ipydir)/pybind11-$(pybind11-version) + +# Import the tarball. tarball=scipy-$(scipy-version).tar.lz $(call import-source, $(scipy-url), $(scipy-checksum)) - if [ x$(on_mac_os) = xyes ]; then - export LDFLAGS="$(LDFLAGS) -undefined dynamic_lookup -bundle" - else -# Same question as for 'numpy': why '-shared'? This obstructs -# the meson build. -# export LDFLAGS="$(LDFLAGS) -shared" - : - fi - conf="$$(pwd)/reproduce/software/config/numpy-scipy.cfg" -# Disable pythran: see -# https://docs.scipy.org/doc/scipy-1.15.2/dev/roadmap-detailed.html#use-of-pythran -# export SCIPY_USE_PYTHRAN=0 # deprecated(?) -# Option 1: Hack the source: +# Disable pythran (see above). pyhook_before() { mv -iv meson.options meson.options.orig; \ sed -e 's/\(use-pythran.*value: *\)true/\1false/' \ meson.options.orig > meson.options } -# Option 2: pass the string -# --config-json='{"setup-args": "-Duse-pythran=false"}' -# to gpep517 with correct escaping of single and double quotes. -# Not tried as of 2025-02-25. +# Build Scipy. + conf="$$(pwd)/reproduce/software/config/numpy-scipy.cfg" $(call pybuild, tar -xf, scipy-$(scipy-version),$$conf,, GPEP517) cp $(dtexdir)/scipy.tex $(ictdir)/ echo "Scipy $(scipy-version) \citep{scipy2020}" > $@ @@ -1003,6 +1252,26 @@ $(ipydir)/sip_tpv-$(sip_tpv-version): \ cp $(dtexdir)/sip_tpv.tex $(ictdir)/ echo "sip_tpv $(sip_tpv-version) \citep{sip-tpv}" > $@ +$(ipydir)/scikit-base-$(scikit-base-version): \ + $(ipydir)/gpep517-$(gpep517-version) \ + $(ipydir)/python-installer-$(python-installer-version) + tarball=scikit-base-$(scikit-base-version).tar.lz + $(call import-source, $(scikit-base-url), $(scikit-base-checksum)) + $(call pybuild, tar -xf, scikit-base-$(scikit-base-version), , \ + Scikit-base $(scikit-base-version), GPEP517) + echo "scikit-base $(scikit-base-version)" > $@ + +$(ipydir)/scikit-learn-$(scikit-learn-version): \ + $(ipydir)/joblib-$(joblib-version) \ + $(ipydir)/threadpoolctl-$(threadpoolctl-version) \ + $(ipydir)/numpy-$(numpy-version) \ + $(ipydir)/scipy-$(scipy-version) + tarball=scikit-learn-$(scikit-learn-version).tar.lz + $(call import-source, $(scikit-learn-url), $(scikit-learn-checksum)) + $(call pybuild, tar -xf, scikit-learn-$(scikit-learn-version), , \ + Scikit-learn $(scikit-learn-version), GPEP517) + echo "scikit-learn $(scikit-learn-version)" > $@ + $(ipydir)/six-$(six-version): \ $(ipydir)/setuptools-$(setuptools-version) tarball=six-$(six-version).tar.lz @@ -1026,6 +1295,29 @@ $(ipydir)/sympy-$(sympy-version): $(ipydir)/mpmath-$(mpmath-version) cp $(dtexdir)/sympy.tex $(ictdir)/ echo "SymPy $(sympy-version) \citep{sympy}" > $@ +$(ipydir)/threadpoolctl-$(threadpoolctl-version): \ + $(ipydir)/flit-core-$(flit-core-version) + tarball=threadpoolctl-$(threadpoolctl-version).tar.lz + $(call import-source, $(threadpoolctl-url), $(threadpoolctl-checksum)) + $(call pybuild, tar -xf, threadpoolctl-$(threadpoolctl-version), , \ + threadpoolctl $(threadpoolctl-version), GPEP517) + echo "threadpoolctl $(threadpoolctl-version)" > $@ + +$(ipydir)/tqdm-$(tqdm-version): \ + $(ipydir)/setuptools-scm-$(setuptools-scm-version) + tarball=tqdm-$(tqdm-version).tar.lz + $(call import-source, $(tqdm-url), $(tqdm-checksum)) + $(call pybuild, tar -xf, \ + tqdm-$(tqdm-version),, \ + Tqdm $(tqdm-version), GPEP517) + +$(ipydir)/trove-classifiers-$(trove-classifiers-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=trove-classifiers-$(trove-classifiers-version).tar.lz + $(call import-source, $(trove-classifiers-url), $(trove-classifiers-checksum)) + $(call pybuild, tar -xf, \ + trove-classifiers-$(trove-classifiers-version),,, GPEP517) + $(ipydir)/uncertainties-$(uncertainties-version): $(ipydir)/numpy-$(numpy-version) tarball=uncertainties-$(uncertainties-version).tar.lz $(call import-source, $(uncertainties-url), $(uncertainties-checksum)) @@ -1040,6 +1332,13 @@ $(ipydir)/urllib3-$(urllib3-version): \ $(call pybuild, tar -xf, urllib3-$(urllib3-version), , \ Urllib3 $(urllib3-version)) +$(ipydir)/versioneer-$(versioneer-version): \ + $(ipydir)/setuptools-$(setuptools-version) + tarball=versioneer-$(versioneer-version).tar.lz + $(call import-source, $(versioneer-url), $(versioneer-checksum)) + $(call pybuild, tar -xf, versioneer-$(versioneer-version), , \ + Versioneer $(versioneer-version), GPEP517) + $(ipydir)/webencodings-$(webencodings-version): \ $(ipydir)/setuptools-$(setuptools-version) \ $(ipydir)/setuptools-scm-$(setuptools-scm-version) diff --git a/reproduce/software/make/r-cran.mk b/reproduce/software/make/r-cran.mk index 7c86c23..7a09915 100644 --- a/reproduce/software/make/r-cran.mk +++ b/reproduce/software/make/r-cran.mk @@ -12,8 +12,8 @@ # # ------------------------------------------------------------------------ # -# Copyright (C) 2022-2025 Boud Roukema <boud@cosmo.torun.pl> -# Copyright (C) 2022-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2022-2026 Boud Roukema <boud@cosmo.torun.pl> +# Copyright (C) 2022-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/reproduce/software/make/xorg.mk b/reproduce/software/make/xorg.mk index 864c32a..33847f3 100644 --- a/reproduce/software/make/xorg.mk +++ b/reproduce/software/make/xorg.mk @@ -14,8 +14,8 @@ # # ------------------------------------------------------------------------ # -# Copyright (C) 2021-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2021-2025 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2021-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2021-2026 Raul Infante-Sainz <infantesainz@gmail.com> # # This Makefile is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -120,6 +120,7 @@ $(ibidir)/fontconfig-$(fontconfig-version): \ $(ibidir)/libxml2-$(libxml2-version) \ $(ibidir)/freetype-$(freetype-version) \ $(ibidir)/util-linux-$(util-linux-version) + # Import the source. tarball=fontconfig-$(fontconfig-version).tar.lz $(call import-source, $(fontconfig-url), $(fontconfig-checksum)) diff --git a/reproduce/software/shell/apptainer-README.md b/reproduce/software/shell/apptainer-README.md new file mode 100644 index 0000000..d2a14c8 --- /dev/null +++ b/reproduce/software/shell/apptainer-README.md @@ -0,0 +1,71 @@ +# Maneage'd projects in Apptainer + +Copyright (C) 2025-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ +Copyright (C) 2025-2026 Giacomo Lorenzetti <glorenzetti@cefca.es>\ +See the end of the file for license conditions. + +For an introduction on containers, see the "Building in containers" section +of the `README.md` file within the top-level directory of this +project. Here, we focus on Apptainer with a simple checklist on how to use +the `apptainer-run.sh` script that we have already prepared in this +directory for easy usage in a Maneage'd project. + + + + + +## Building your Maneage'd project in Apptainer + +Through the steps below, you will create an Apptainer image that will only +contain the software environment and keep the project source and built +analysis files (data and PDF) on your host operating system. This enables +you to keep the size of the image to a minimum (only containing the built +software environment) to easily move it from one computer to another. + + 1. Using your favorite text editor, create a `run.sh` in your top Maneage + directory (as described in the comments at the start of the + `apptainer.sh` script in this directory). Just add `--build-only` on + the first run so it doesn't go onto doing the analysis and just sets up + the software environment. Set the respective directory(s) based on your + filesystem (the software directory is optional). The `run.sh` file name + is already in `.gitignore` (because it contains local directories), so + Git will ignore it and it won't be committed by mistake. + + 2. Make the script executable with `chmod +x ./run.sh`, and run it with + `./run.sh`. + + 3. Once the build finishes, the build directory (on your host) will + contain two Singularity Image Format (SIF) files listed below. You can + move them to any other (more permanent) positions in your filesystem or + to other computers as needed. + * `maneage-base.sif`: image containing the base operating system that + was used to build your project. You can safely delete this unless you + need to keep it for future builds without internet (you can give it + to the `--base-name` option of this script). If you want a different + name for this, put the same option in your + * `maneaged.sif`: image with the full software environment of your + project. This file is necessary for future runs of your project + within the container. + + 3. To execute your project remove the `--build-only` and use `./run.sh` to + execute it. If you want to enter your Maneage'd project shell, add the + `--project-shell` option to the call inside `./run.sh`. + + + + + +## Copyright information + +This file is free software: you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) +any later version. + +This file is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along +with this file. If not, see <https://www.gnu.org/licenses/>. diff --git a/reproduce/software/shell/apptainer.sh b/reproduce/software/shell/apptainer.sh new file mode 100755 index 0000000..2aa3535 --- /dev/null +++ b/reproduce/software/shell/apptainer.sh @@ -0,0 +1,475 @@ +#!/bin/sh +# +# Create a Apptainer container from an existing image of the built software +# environment, but with the source, data and build (analysis) directories +# directly within the host file system. This script is assumed to be run in +# the top project source directory (that has 'README.md' and +# 'paper.tex'). If not, use the '--source-dir' option to specify where the +# Maneage'd project source is located. +# +# Usage: +# +# - When you are at the top Maneage'd project directory, run this script +# like the example below. Just set the build directory location on your +# system. See the items below for optional values to optimize the +# process (avoid downloading for exmaple). +# +# ./reproduce/software/shell/apptainer.sh \ +# --build-dir=/PATH/TO/BUILD/DIRECTORY +# +# - Non-mandatory options: +# +# - If you already have the input data that is necessary for your +# project, use the '--input-dir' option to specify its location +# on your host file system. Otherwise the necessary analysis +# files will be downloaded directly into the build +# directory. Note that this is only necessary when '--build-only' +# is not given. +# +# - If you already have the necessary software tarballs that are +# necessary for your project, use the '--software-dir' option to +# specify its location on your host file system only when +# building the container. No problem if you don't have them, they +# will be downloaded during the configuration phase. +# +# - To avoid having to set them every time you want to start the +# apptainer environment, you can put this command (with the proper +# directories) into a 'run.sh' script in the top Maneage'd project +# source directory and simply execute that. The special name 'run.sh' +# is in Maneage's '.gitignore', so it will not be included in your +# git history by mistake. +# +# Known problems: +# +# Copyright (C) 2025-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2025-2026 Giacomo Lorenzetti <glorenzetti@cefca.es> +# +# This script is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# This script is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this script. If not, see <http://www.gnu.org/licenses/>. + + + + + +# Script settings +# --------------- +# Stop the script if there are any errors. +set -e + + + + + +# Default option values +sif="" +jobs=0 +debug=0 +quiet=0 +host_cc=0 +source_dir= +build_only= +base_sif="" +shm_size=20gb +scriptname="$0" +project_shell=0 +container_shell=0 +base_os=debian:stable-slim + +print_help() { + # Print the output. + cat <<EOF +Usage: $scriptname [OPTIONS] + +Top-level script to build and run a Maneage'd project within Apptainer. + + Host OS directories (to be mounted in the container): + -b, --build-dir=STR Dir. to build in (only analysis in host). + -i, --input-dir=STR Dir. of input datasets (optional). + -s, --software-dir=STR Directory of necessary software tarballs. + --source-dir=STR Directory of source code (default: 'pwd -P'). + + Apptainer images + --sif=STR Project's apptainer image (a '.sif' file). + --base-os=STR Base OS name (default: '$base_os'). + --base-sif=STR Base OS apptainer image (a '.sif' file). + + Configuration options + --host-cc Use containers's C compiler, don't build GCC. + -d, --debug[=FLAGS] Use -j1 and stop on first crash. + + Interactive shell + --project-shell Open the project's shell within the container. + --container-shell Open the container shell. + + Operating mode: + -q, --quiet Do not print informative statements. + -?, --help Give this help list. + -j, --jobs=INT Number of threads to use in each phase. + --build-only Just build the container, don't run it. + +Mandatory or optional arguments to long options are also mandatory or +optional for any corresponding short options. + +Maneage URL: https://maneage.org + +Report bugs to mohammad@akhlaghi.org +EOF +} + +on_off_option_error() { + if [ "x$2" = x ]; then + echo "$scriptname: '$1' doesn't take any values" + else + echo "$scriptname: '$1' (or '$2') doesn't take any values" + fi + exit 1 +} + +check_v() { + if [ x"$2" = x ]; then + printf "$scriptname: option '$1' requires an argument. " + printf "Try '$scriptname --help' for more information\n" + exit 1; + fi +} + +while [ $# -gt 0 ] +do + case $1 in + + # OS directories + -b|--build-dir) build_dir="$2"; check_v "$1" "$build_dir"; shift;shift;; + -b=*|--build-dir=*) build_dir="${1#*=}"; check_v "$1" "$build_dir"; shift;; + -b*) build_dir=$(echo "$1" | sed -e's/-b//'); check_v "$1" "$build_dir"; shift;; + -i|--input-dir) input_dir="$2"; check_v "$1" "$input_dir"; shift;shift;; + -i=*|--input-dir=*) input_dir="${1#*=}"; check_v "$1" "$input_dir"; shift;; + -i*) input_dir=$(echo "$1" | sed -e's/-i//'); check_v "$1" "$input_dir"; shift;; + -s|--software-dir) software_dir="$2"; check_v "$1" "$software_dir"; shift;shift;; + -s=*|--software-dir=*) software_dir="${1#*=}"; check_v "$1" "$software_dir"; shift;; + -s*) software_dir=$(echo "$1" | sed -e's/-s//'); check_v "$1" "$software_dir"; shift;; + --source-dir) source_dir="$2"; check_v "$1" "$source_dir"; shift;shift;; + --source-dir=*) source_dir="${1#*=}"; check_v "$1" "$source_dir"; shift;; + + # Container options. + --sif) sif="$2"; check_v "$1" "$sif"; shift;shift;; + --sif=*) sif="${1#*=}"; check_v "$1" "$sif"; shift;; + --base-os) base_os="$2"; check_v "$1" "$base_os"; shift;shift;; + --base-os=*) base_os="${1#*=}"; check_v "$1" "$base_os"; shift;; + --base-sif) base_sif="$2"; check_v "$1" "$base_sif"; shift;shift;; + --base-sif=*) base_sif="${1#*=}"; check_v "$1" "$base_sif"; shift;; + + # Configuration options. + --host-cc) host_cc=1; shift;; + --host-cc=*) on_off_option_error --host-cc;; + -d|--debug) debug=1; shift;; + -d|--debug=*) on_off_option_error --debug;; + + # Interactive shell. + --project-shell) project_shell=1; shift;; + --project_shell=*) on_off_option_error --project-shell;; + --container-shell) container_shell=1; shift;; + --container_shell=*) on_off_option_error --container-shell;; + + # Operating mode + -q|--quiet) quiet=1; shift;; + -q*|--quiet=*) on_off_option_error --quiet;; + -j|--jobs) jobs="$2"; check_v "$1" "$jobs"; shift;shift;; + -j=*|--jobs=*) jobs="${1#*=}"; check_v "$1" "$jobs"; shift;; + -j*) jobs=$(echo "$1" | sed -e's/-j//'); check_v "$1" "$jobs"; shift;; + --build-only) build_only=1; shift;; + --build-only=*) on_off_option_error --build-only;; + --shm-size) shm_size="$2"; check_v "$1" "$shm_size"; shift;shift;; + --shm-size=*) shm_size="${1#*=}"; check_v "$1" "$shm_size"; shift;; + -'?'|--help) print_help; exit 0;; + -'?'*|--help=*) on_off_option_error --help -?;; + + # Unrecognized option: + -*) echo "$scriptname: unknown option '$1'"; exit 1;; + esac +done + + + + + +# Sanity checks +# ------------- +# +# Make sure that the build directory is given and that it exists. +if [ x$build_dir = x ]; then + printf "$scriptname: '--build-dir' not provided, this is the " + printf "location that all built analysis files will be kept on " + printf "the host OS\n" + exit 1; +else + if ! [ -d $build_dir ]; then + printf "$scriptname: '$build_dir' (value to '--build-dir') " + printf "doesn't exist\n" + exit 1; + fi +fi + +# Set the default project and base-OS image names (inside the build +# directory). +if [ x"$base_sif" = x ]; then base_sif=$build_dir/maneage-base.sif; fi +if [ x"$sif" = x ]; then sif=$build_dir/maneaged.sif; fi + + + + + +# Directory preparations +# ---------------------- +# +# If the host operating system has '/dev/shm', then give Apptainer access +# to it also for improved speed in some scenarios (like configuration). +if [ -d /dev/shm ]; then + shm_mnt="--mount type=bind,src=/dev/shm,dst=/dev/shm"; +else shm_mnt=""; +fi + +# If the following directories do not exist within the build directory, +# create them to make sure the '--mount' commands always work and +# that any file. Ideally, the 'input' directory should not be under the 'build' +# directory, but if the user hasn't given it then they don't care about +# potentially deleting it later (Maneage will download the inputs), so put +# it in the build directory. +analysis_dir="$build_dir"/analysis +if ! [ -d $analysis_dir ]; then mkdir $analysis_dir; fi +analysis_dir_mnt="--mount type=bind,src=$analysis_dir,dst=/home/maneager/build/analysis" + +# If no '--source-dir' was given, set it to the output of 'pwd -P' (to get +# the direct path without potential symbolic links) in the running directory. +if [ x"$source_dir" = x ]; then source_dir=$(pwd -P); fi +source_dir_mnt="--mount type=bind,src=$source_dir,dst=/home/maneager/source" + +# Only when an an input directory is given, we need the respective 'mount' +# option for the 'apptainer run' command. +input_dir_mnt="" +if ! [ x"$input_dir" = x ]; then + input_dir_mnt="--mount type=bind,src=$input_dir,dst=/home/maneager/input" +fi + +# If no '--jobs' has been specified, use the maximum available jobs to the +# operating system. Apptainer only works on GNU/Linux operating systems, so +# there is no need to account for reading the number of threads on macOS. +if [ x"$jobs" = x0 ]; then jobs=$(nproc); fi + +# Since the container is read-only and is run with the '--contain' option +# (which makes an empty '/tmp'), we need to make a dedicated directory for +# the container to be able to write to. This is necessary because some +# software (Biber in particular on the default branch) need to write there! +# See https://github.com/plk/biber/issues/494. We'll keep the directory on +# the host OS within the build directory, but as a hidden file (since it is +# not necessary in other types of build and ultimately only contains +# temporary files of programs that need it). +toptmp=$build_dir/.apptainer-tmp-$(whoami) +if ! [ -d $toptmp ]; then mkdir $toptmp; fi +chmod -R +w $toptmp/ # Some software remove writing flags on /tmp files. +if ! [ x"$( ls -A $toptmp )" = x ]; then rm -r "$toptmp"/*; fi + +# [APPTAINER-ONLY] Optional mounting option for the software directory. +software_dir_mnt="" +if ! [ x"$software_dir" = x ]; then + software_dir_mnt="--mount type=bind,src=$software_dir,dst=/home/maneager/tarballs-software" +fi + + + + + +# Maneage'd Apptainer SIF container +# --------------------------------- +# +# Build the base operating system using Maneage's './project configure' +# step. +if [ -f $sif ]; then + if [ $quiet = 0 ]; then + printf "$scriptname: info: project's image ('$sif') " + printf "already exists and will be used. If you want to build a " + printf "new project image, give a new name to '--project-name'. " + printf "To remove this message run with '--quiet'\n" + fi +else + + # Build the basic definition, with just Debian-slim with minimal + # necessary tools. + if [ -f $base_sif ]; then + if [ $quiet = 0 ]; then + printf "$scriptname: info: base OS apptainer image " + printf "('$base_sif') already exists and will be used. " + printf "If you want to build a new base OS image, give " + printf "a new name to '--base-sif'. To remove this " + printf "message run with '--quiet'\n" + fi + else + + base_def=$build_dir/base.def + cat <<EOF > $base_def +Bootstrap: docker +From: $base_os + +%post + apt-get update && apt-get install -y gcc g++ wget +EOF + # Build the base operating system container and delete the + # temporary definition file. + apptainer build $base_sif $base_def + rm $base_def + fi + + # Build the Maneage definition file. + # - About the '$jobs' variable: this definition file is temporarily + # built and deleted immediately after the SIF file is created. So + # instead of using Apptainer's more complex '{{ jobs }}' format to + # pass an argument, we simply write the value of the configure + # script's '--jobs' option as a shell variable here when we are + # building that file. + # - About the removal of Maneage'd tarballs: we are doing this so if + # Maneage has downloaded tarballs during the build they do not + # unecessarily bloat the container. Even when the user has given a + # software tarball directory, they will all be symbolic links that + # aren't valid when the user runs the container (since we only + # mount the software tarballs at build time). + confopts="" + intbuild=/home/maneager/build + maneage_def=$build_dir/maneage.def + if [ $host_cc = 1 ]; then confopts="--host-cc"; fi + if [ $debug = 1 ]; then confopts="$confopts --debug"; fi + cat <<EOF > $maneage_def +Bootstrap: localimage +From: $base_sif + +%setup + mkdir -p \${APPTAINER_ROOTFS}/home/maneager/input + mkdir -p \${APPTAINER_ROOTFS}/home/maneager/source + mkdir -p \${APPTAINER_ROOTFS}/home/maneager/build/analysis + mkdir -p \${APPTAINER_ROOTFS}/home/maneager/tarballs-software + +%post + cd /home/maneager/source + ./project configure --jobs=$jobs --no-pause $confopts \\ + --input-dir=/home/maneager/input \\ + --build-dir=$intbuild \\ + --software-dir=/home/maneager/tarballs-software + rm /home/maneager/build/software/tarballs/* + +%runscript + cd /home/maneager/source + if ./project configure --build-dir=$intbuild $confopts \\ + --existing-conf --no-pause \\ + --offline --quiet; then \\ + if [ x"\$maneage_apptainer_stat" = xshell ]; then \\ + ./project shell --build-dir=$intbuild; \\ + elif [ x"\$maneage_apptainer_stat" = xrun ]; then \\ + if [ x"\$maneage_jobs" = x ]; then \\ + ./project make --build-dir=$intbuild; \\ + else \\ + ./project make --build-dir=$intbuild --jobs=\$maneage_jobs; \\ + fi; \\ + else \\ + printf "$scriptname: '\$maneage_apptainer_stat' (value "; \\ + printf "to 'maneage_apptainer_stat' environment variable) "; \\ + printf "is not recognized: should be either 'shell' or 'run'\n"; \\ + exit 1; \\ + fi; \\ + else \\ + printf "$scriptname: configuration failed! This is probably "; \\ + printf "due to a mismatch between the software versions of "; \\ + printf "the container and the source that it is being "; \\ + printf "executed.\n"; \\ + exit 1; \\ + fi +EOF + + # Build the maneage container. The last two are arguments (where order + # matters). The first few are options where order does not matter (so + # we have sorted them by line length). + apptainer build \ + $shm_mnt \ + $input_dir_mnt \ + $source_dir_mnt \ + $analysis_dir_mnt \ + $software_dir_mnt \ + --ignore-fakeroot-command \ + \ + $sif \ + $maneage_def + + # Clean up. + rm $maneage_def +fi + +# If the user just wanted to build the base operating system, abort the +# script here. +if ! [ x"$build_only" = x ]; then + if [ $quiet = 0 ]; then + printf "$scriptname: info: Maneaged project has been configured " + printf "successfully in the '$sif' image\n" + fi + exit 0 +fi + + + + + +# Run the Maneage'd container +# --------------------------- +# +# Set the high-level Apptainer operational mode. +if [ $container_shell = 1 ]; then + aopt="shell" +elif [ $project_shell = 1 ]; then + aopt="run --env maneage_apptainer_stat=shell" +else + aopt="run --env maneage_apptainer_stat=run --env maneage_jobs=$jobs" +fi + +# Build the hostname from the name of the SIF file of the project name. +hstname=$(echo "$sif" \ + | awk 'BEGIN{FS="/"}{print $NF}' \ + | sed -e's|.sif$||') + +# Execute Apptainer: +# +# - We are not using '--unsquash' (to run within a sandbox) because it +# loads the full multi-gigabyte container into RAM (which we usually +# need for data processing). The container is read-only and we are +# using the following two options instead to ensure that we have no +# influence from outside the container. (description of each is from +# the Apptainer manual) +# --contain: use minimal /dev and empty other directories (e.g. /tmp +# and $HOME) instead of sharing filesystems from your host. +# --cleanenv: clean environment before running container". +# +# - We are not mounting '/dev/shm' since Apptainer prints a warning that +# it is already mounted (apparently does not need it at run time). +# +# --no-home and --home: the first ensures that the 'HOME' variable is +# different from the user's home on the host operating system, the +# second sets it to a directory we specify (to keep things like +# '.bash_history'). +apptainer $aopt \ + --no-home \ + --contain \ + --cleanenv \ + --home $toptmp \ + $input_dir_mnt \ + $source_dir_mnt \ + $analysis_dir_mnt \ + --workdir $toptmp \ + --hostname $hstname \ + --cwd /home/maneager/source \ + \ + $sif diff --git a/reproduce/software/shell/bashrc.sh b/reproduce/software/shell/bashrc.sh index 6bbd774..5534f3d 100755 --- a/reproduce/software/shell/bashrc.sh +++ b/reproduce/software/shell/bashrc.sh @@ -34,7 +34,7 @@ # shell'). # # -# Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -62,13 +62,13 @@ if [ x$PROJECT_STATUS = xshell ]; then # A small introductory message. echo "----------------------------------------------------------------------" echo "Welcome to the Maneage interactive shell for this project, running" - echo " $(sh --version | awk 'NR==1')" + echo " $(sh --version | awk 'NR==1')" echo echo "This shell's home directory is the project's build directory:" - echo " HOME: $HOME" + echo " HOME: $HOME" echo echo "This shell's startup file is in the project's source directory:" - echo " $PROJECT_RCFILE" + echo " $PROJECT_RCFILE" echo echo "To return to your host shell, run the 'exit' command." echo "----------------------------------------------------------------------" diff --git a/reproduce/software/shell/configure.sh b/reproduce/software/shell/configure.sh index 517e1ed..19c14ac 100755 --- a/reproduce/software/shell/configure.sh +++ b/reproduce/software/shell/configure.sh @@ -2,9 +2,9 @@ # # Necessary preparations/configurations for the reproducible project. # -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2021-2025 Raul Infante-Sainz <infantesainz@gmail.com> -# Copyright (C) 2022-2025 Pedram Ashofteh Ardakani <pedramardakani@pm.me> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2021-2026 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2022-2026 Pedram Ashofteh Ardakani <pedramardakani@pm.me> # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -40,6 +40,14 @@ set -e # had the chance to implement it yet (please help if you can!). Until then, # please set them based on your project (if they differ from the core # branch). + +# If equals 1, a message will be printed, showing the nano-seconds since +# previous step: useful with '-e --offline --nopause --quiet' to find +# bottlenecks for speed optimization. Speed is important because this +# script is called automatically every time by the container scripts. +check_elapsed=0 + +# In case a fortran compiler is necessary to check. need_gfortran=0 @@ -52,14 +60,12 @@ need_gfortran=0 # These are defined to help make this script more readable. topdir="$(pwd)" optionaldir="/optional/path" -adir=reproduce/analysis/config cdir=reproduce/software/config -pconf=$cdir/LOCAL.conf -ptconf=$cdir/LOCAL_tmp.conf -poconf=$cdir/LOCAL_old.conf -depverfile=$cdir/versions.conf -depshafile=$cdir/checksums.conf + + + + @@ -73,14 +79,21 @@ depshafile=$cdir/checksums.conf # that their changes are not going to be permenant. create_file_with_notice () { - if echo "# IMPORTANT: file can be RE-WRITTEN after './project configure'" > "$1" + if printf "# IMPORTANT: " > "$1" then - echo "#" >> "$1" - echo "# This file was created during configuration" >> "$1" - echo "# ('./project configure'). Therefore, it is not under" >> "$1" - echo "# version control and any manual changes to it will be" >> "$1" - echo "# over-written if the project re-configured." >> "$1" - echo "#" >> "$1" + # These commands may look messy, but the produced comments in the + # file are the main goal and they are readable. (without having to + # break our source-code line length). + printf "file can be RE-WRITTEN after './project " >> "$1" + printf "configure'.\n" >> "$1" + printf "#\n" >> "$1" + printf "# This file was created during configuration " >> "$1" + printf "('./project configure').\n" >> "$1" + printf "# Therefore, it is not under version control " >> "$1" + printf "and any manual changes\n" >> "$1" + printf "# to it will be over-written when the " >> "$1" + printf "project is re-configured.\n" >> "$1" + printf "#\n" >> "$1" else echo; echo "Can't write to $1"; echo; exit 1 @@ -102,7 +115,7 @@ absolute_dir () if stat "$address" 1> /dev/null; then echo "$(cd "$(dirname "$1")" && pwd )/$(basename "$1")" else - exit 1; + echo "$optionaldir" fi } @@ -200,30 +213,113 @@ free_space_warning() -# See if we are on a Linux-based system -# -------------------------------------- +# Function to empty the temporary software building directory. This can +# either be a symbolic link (to RAM) or an actual directory, so we can't +# simply use 'rm -r' (because a symbolic link is not a directory for 'rm'). +empty_build_tmp() { + + # 'ls -A' does not print the '.' and '..' and the '-z' option of '[' + # checks if the string is empty or not. This allows us to only attempt + # deleting the directory's contents if it actually has anything inside + # of it. Otherwise, '*' will not expand and we'll get an 'rm' error + # complaining that '$tmpblddir/*' doesn't exist. We also don't want to + # use 'rm -rf $tmpblddir/*' because in case of a typo or while + # debugging (if '$tmpblddir' becomes an empty string), this can + # accidentally delete the whole root partition (or a least the '/home' + # partition of the user). + if ! [ x"$( ls -A $tmpblddir )" = x ]; then + rm -rf "$tmpblddir"/* + fi + rm -r "$tmpblddir" +} + + + + + +# Function to report the elapsed time between steps (if it was activated +# above with 'check_elapsed'). +elapsed_time_from_prev_step() { + if [ $check_elapsed = 1 ]; then + chel_now=$(date +"%N"); + chel_delta=$(echo $chel_prev $chel_now \ + | awk '{ delta=($2-$1)/1e6; \ + if(delta>0) d=delta; else d=0; \ + print d}') + chel_dsum=$(echo $chel_dsum $chel_delta | awk '{print $1+$2}') + echo $chel_counter $chel_delta "$1" \ + | awk '{ printf "Step %02d: %-6.2f [millisec]; %s\n", \ + $1, $2, $3}' + chel_counter=$((chel_counter+1)) + chel_prev=$(date +"%N") + fi +} + + + + + + + + + + +# In already-built container +# -------------------------- +# +# We need to run './project configure' at the start of every run of Maneage +# within a container (with 'shell' or 'make'). This is because we need to +# ensure the versions of all software are correct. However, the container +# filesystem (where the build/software directory is located) should be run +# as read-only when doing the analysis. So we will not be able to run some +# of the tests that require writing files or are generally not relevant +# when the container is already built (we want the configure command to be +# as fast as possible). +# +# The project source in Maneage'd containers is '/home/maneager/source'. +built_container=0 +if [ "$topdir" = /home/maneager/source ] \ + && [ -f .build/software/config/hardware-parameters.tex ]; then + built_container=1; +fi + +# Initialize the elapsed time measurement parameters. +if [ $check_elapsed = 1 ]; then + chel_dsum=0.00 + chel_counter=1 + chel_prev=$(date +"%N") + chel_start=$(date +"%N") +fi + + + + +# Identify the running OS +# ----------------------- # # Some features are tailored to GNU/Linux systems, while the BSD-based # behavior is different. Initially we only tested macOS (hence the name of # the variable), but as FreeBSD is also being inlucded in our tests. As # more systems get used, we need to tailor these kinds of things better. -kernelname=$(uname -s) -if [ x$kernelname = xLinux ]; then - on_mac_os=no - - # Don't forget to add the respective C++ compiler below (leave 'cc' in - # the end). - c_compiler_list="gcc clang cc" -elif [ x$kernelname = xDarwin ]; then - host_cc=1 - on_mac_os=yes - - # Don't forget to add the respective C++ compiler below (leave 'cc' in - # the end). - c_compiler_list="clang gcc cc" -else - on_mac_os=no - cat <<EOF +if [ $built_container = 0 ]; then + kernelname=$(uname -s) + if [ $pauseformsg = 1 ]; then pausesec=10; else pausesec=0; fi + if [ x$kernelname = xLinux ]; then + on_mac_os=no + + # Don't forget to add the respective C++ compiler below (leave 'cc' in + # the end). + c_compiler_list="gcc clang cc" + elif [ x$kernelname = xDarwin ]; then + host_cc=1 + on_mac_os=yes + + # Don't forget to add the respective C++ compiler below (leave 'cc' in + # the end). + c_compiler_list="clang gcc cc" + else + on_mac_os=no + cat <<EOF ______________________________________________________ !!!!!!! WARNING !!!!!!! @@ -234,17 +330,20 @@ web-form: https://savannah.nongnu.org/support/?func=additem&group=reproduce -The configuration will continue in 10 seconds... +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - sleep 10 + sleep $pausesec + fi + elapsed_time_from_prev_step os_identify fi - # Collect CPU information # ----------------------- # @@ -255,42 +354,50 @@ fi # later recorded as a LaTeX macro to be put in the final paper, but it # could be used in a more systematic way to optimize/revise project # workflow and build. -hw_class=$(uname -m) -if [ x$kernelname = xLinux ]; then - byte_order=$(lscpu \ - | grep 'Byte Order' \ - | awk '{ \ - for(i=3;i<NF;++i) \ - printf "%s ", $i; \ - printf "%s", $NF}') - address_sizes=$(lscpu \ - | grep 'Address sizes' \ - | awk '{ \ - for(i=3;i<NF;++i) \ - printf "%s ", $i; \ - printf "%s", $NF}') -elif [ x$on_mac_os = xyes ]; then - hw_byteorder=$(sysctl -n hw.byteorder) - if [ x$hw_byteorder = x1234 ]; then byte_order="Little Endian"; - elif [ x$hw_byteorder = x4321 ]; then byte_order="Big Endian"; - fi - # On macOS, the way of obtaining the number of cores is different - # between Intel or Apple M1 CPUs. Here we disinguish between Apple M1 - # or others. - maccputype=$(sysctl -n machdep.cpu.brand_string) - if [ x"$maccputype" = x"Apple M1" ]; then - address_size_physical=$(sysctl -n machdep.cpu.thread_count) - address_size_virtual=$(sysctl -n machdep.cpu.logical_per_package) +if [ $built_container = 0 ]; then + if [ x$kernelname = xLinux ]; then + byte_order=$(lscpu \ + | grep 'Byte Order' \ + | awk '{ \ + for(i=3;i<NF;++i) \ + printf "%s ", $i; \ + printf "%s", $NF}') + address_sizes=$(lscpu \ + | grep 'Address sizes' \ + | awk '{ \ + for(i=3;i<NF;++i) \ + printf "%s ", $i; \ + printf "%s", $NF}') + elif [ x$on_mac_os = xyes ]; then + hw_byteorder=$(sysctl -n hw.byteorder) + if [ x$hw_byteorder = x1234 ]; then byte_order="Little Endian"; + elif [ x$hw_byteorder = x4321 ]; then byte_order="Big Endian"; + fi + + # On macOS, the way of obtaining the number of cores is different + # between Intel or Apple Silicon CPUs. Here we distinguish between + # Apple Silicon (any "Apple M*" brand string, e.g. M1, M1 Pro, M1 + # Max, M2, M2 Pro, ...) and Intel-based Macs. Note that the Apple + # Silicon group can be in the formats of "Apple M1" or "Apple M1 + # Pro", so we use a prefix match via a 'case' pattern to include + # both in one check. + maccputype=$(sysctl -n machdep.cpu.brand_string 2>/dev/null) + case "$maccputype" in + "Apple M"*) + address_size_physical=$(sysctl -n machdep.cpu.thread_count) + address_size_virtual=$(sysctl -n machdep.cpu.logical_per_package) + ;; + *) + address_size_physical=$(sysctl -n machdep.cpu.address_bits.physical) + address_size_virtual=$(sysctl -n machdep.cpu.address_bits.virtual) + ;; + esac + address_sizes="$address_size_physical bits physical, " + address_sizes+="$address_size_virtual bits virtual" else - address_size_physical=$(sysctl -n machdep.cpu.address_bits.physical) - address_size_virtual=$(sysctl -n machdep.cpu.address_bits.virtual) - fi - address_sizes="$address_size_physical bits physical, " - address_sizes+="$address_size_virtual bits virtual" -else - byte_order="unrecognized" - address_sizes="unrecognized" - cat <<EOF + byte_order="unrecognized" + address_sizes="unrecognized" + cat <<EOF ______________________________________________________ !!!!!!! WARNING !!!!!!! @@ -300,10 +407,15 @@ the necessary steps in the 'reproduce/software/shell/configure.sh' script https://savannah.nongnu.org/support/?func=additem&group=reproduce +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - sleep 5 + sleep $pausesec + fi + elapsed_time_from_prev_step cpu-info fi @@ -318,7 +430,7 @@ fi # avoid these error it is highly recommended to install Xcode in the host # system. Here, it is checked that this is the case, and if not, warn the user # about not having Xcode already installed. -if [ x$on_mac_os = xyes ]; then +if [ $built_container = 0 ] && [ x$on_mac_os = xyes ]; then # 'which' isn't in POSIX, so we are using 'command -v' instead. xcode=$(command -v xcodebuild) @@ -341,12 +453,15 @@ web-form: https://savannah.nongnu.org/support/?func=additem&group=reproduce -The configuration will continue in 5 seconds ... +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - sleep 5 + sleep $pausesec fi + elapsed_time_from_prev_step compiler-of-mac-os fi @@ -359,14 +474,15 @@ fi # To build the software, we'll need some basic tools (the C/C++ compilers # in particular) to be present. has_compilers=no -for c in $c_compiler_list; do +if [ $built_container = 0 ]; then + for c in $c_compiler_list; do - # Set the respective C++ compiler. - if [ x$c = xcc ]; then cplus=c++; - elif [ x$c = xgcc ]; then cplus=g++; - elif [ x$c = xclang ]; then cplus=clang++; - else - cat <<EOF + # Set the respective C++ compiler. + if [ x$c = xcc ]; then cplus=c++; + elif [ x$c = xgcc ]; then cplus=g++; + elif [ x$c = xclang ]; then cplus=clang++; + else + cat <<EOF ______________________________________________________ !!!!!!! BUG !!!!!!! @@ -379,21 +495,21 @@ script (just above this error message), or contact us with this web-form: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - exit 1 - fi + exit 1 + fi - # Check if they exist. - if type $c > /dev/null 2>/dev/null; then - export CC=$c; - if type $cplus > /dev/null 2>/dev/null; then - export CXX=$cplus - has_compilers=yes - break + # Check if they exist. + if type $c > /dev/null 2>/dev/null; then + export CC=$c; + if type $cplus > /dev/null 2>/dev/null; then + export CXX=$cplus + has_compilers=yes + break + fi fi - fi -done -if [ x$has_compilers = xno ]; then - cat <<EOF + done + if [ x$has_compilers = xno ]; then + cat <<EOF ______________________________________________________ !!!!!!! C/C++ Compiler NOT FOUND !!!!!!! @@ -416,51 +532,52 @@ Xcode install are recommended. There are known problems with GCC on macOS. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - exit 1 + exit 1 + fi + elapsed_time_from_prev_step compiler-present fi - -# Special directory for compiler testing -# -------------------------------------- -# -# This directory will be deleted when the compiler testing is finished. -compilertestdir=.compiler_test_dir_please_delete -if ! [ -d $compilertestdir ]; then mkdir $compilertestdir; fi - - - - - # Check C compiler # ---------------- # -# Here we check if the C compiler works properly. About the "no warning" -# variable ('nowarnings'): -# -# -Wno-nullability-completeness: on macOS Big Sur 11.2.3 and Xcode 12.4, -# hundreds of 'nullability-completeness' warnings are printed which can -# be very annoying and even hide important errors or warnings. It is -# also harmless for our test here, so it is generally added. -testprog=$compilertestdir/test +# We are checking the C compiler before asking for the directories to let +# the user fix lower-level problems before giving inputs. +compilertestdir=.compiler_test_dir_please_delete testsource=$compilertestdir/test.c -if [ x$on_mac_os = xyes ]; then - noccwarnings="-Wno-nullability-completeness" -fi -echo; echo; echo "Checking host C compiler ('$CC')..."; -cat > $testsource <<EOF +testprog=$compilertestdir/test +if [ $built_container = 0 ]; then + + # Here we check if the C compiler works properly. We'll start by + # making a directory to keep the products. + if ! [ -d $compilertestdir ]; then mkdir $compilertestdir; fi + + # About the "no warning" variable ('nowarnings'): + # + # -Wno-nullability-completeness: on macOS Big Sur 11.2.3 and + # Xcode 12.4, hundreds of 'nullability-completeness' warnings + # are printed which can be very annoying and even hide + # important errors or warnings. It is also harmless for our + # test here, so it is generally added. + if [ x$on_mac_os = xyes ]; then + noccwarnings="-Wno-nullability-completeness" + fi + if [ $quiet = 0 ]; then + echo; echo "Checking host C compiler ('$CC')..."; + fi + cat > $testsource <<EOF #include <stdio.h> #include <stdlib.h> -int main(void){printf("...C compiler works.\n"); - return EXIT_SUCCESS;} +int main(void){printf("Good!\n"); return EXIT_SUCCESS;} EOF -if $CC $noccwarnings $testsource -o$testprog && $testprog; then - rm $testsource $testprog -else - rm $testsource - cat <<EOF + if $CC $noccwarnings $testsource -o$testprog && $testprog > /dev/null; then + if [ $quiet = 0 ]; then echo "... yes"; fi + rm $testsource $testprog + else + rm $testsource + cat <<EOF ______________________________________________________ !!!!!!! C compiler doesn't work !!!!!!! @@ -479,13 +596,14 @@ https://savannah.nongnu.org/support/?func=additem&group=reproduce !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - exit 1 + exit 1 + fi + elapsed_time_from_prev_step compiler-c-check fi - # See if we need the dynamic-linker (-ldl) # ---------------------------------------- # @@ -493,7 +611,8 @@ fi # GNU/Linux systems, we'll need the '-ldl' flag to link such programs. But # Mac OS doesn't need any explicit linking. So we'll check here to see if # it is present (thus necessary) or not. -cat > $testsource <<EOF +if [ $built_container = 0 ]; then + cat > $testsource <<EOF #include <stdio.h> #include <dlfcn.h> int @@ -502,17 +621,17 @@ main(void) { return 0; } EOF -if $CC $testsource -o$testprog 2>/dev/null > /dev/null; then - needs_ldl=no; -else - needs_ldl=yes; + if $CC $testsource -o$testprog 2>/dev/null > /dev/null; then + needs_ldl=no; + else + needs_ldl=yes; + fi + elapsed_time_from_prev_step compiler-needs-dynamic-linker fi - - # See if the C compiler can build static libraries # ------------------------------------------------ # @@ -528,32 +647,30 @@ fi # the library came from the system or our build. static_build=no - - - - # Print warning if the host CC is to be used. -if [ x$host_cc = x1 ]; then +if [ $built_container = 0 ] && [ x$host_cc = x1 ]; then cat <<EOF ______________________________________________________ !!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!! The GNU Compiler Collection (GCC, including compilers for C, C++, Fortran -and etc) is currently not built on macOS systems for this project. To build -the project's necessary software on this system, we need to use your -system's C compiler. +and etc) is not going to be built for this project. Either it is a macOS, +or you have used '--host-cc'. -Project's configuration will continue in 5 seconds. -______________________________________________________ +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - sleep 5 + sleep $pausesec fi + # Necessary C library element positions # ------------------------------------- # @@ -563,7 +680,7 @@ fi # similarly different location. sys_cpath="" sys_library_path="" -if [ x"$on_mac_os" != xyes ]; then +if [ $built_container = 0 ] && [ x"$on_mac_os" != xyes ]; then # Get the GCC target name of the compiler, when its given, special # C libraries and headers are in a sub-directory of the host. @@ -581,6 +698,7 @@ if [ x"$on_mac_os" != xyes ]; then # For a check: #echo "sys_library_path: $sys_library_path" #echo "sys_cpath: $sys_cpath" + elapsed_time_from_prev_step compiler-sys-cpath fi @@ -592,25 +710,28 @@ fi # # A static C library and the 'sys/cdefs.h' header are necessary for # building GCC. -if [ x"$host_cc" = x0 ]; then - echo; echo; echo "Checking if static C library is available..."; - cat > $testsource <<EOF +if [ $built_container = 0 ]; then + if [ x"$host_cc" = x0 ]; then + if [ $quiet = 0 ]; then + echo; echo "Checking if static C library is available..."; + fi + cat > $testsource <<EOF #include <stdio.h> #include <stdlib.h> #include <sys/cdefs.h> -int main(void){printf("...yes\n"); - return EXIT_SUCCESS;} +int main(void){printf("...yes\n"); return EXIT_SUCCESS;} EOF - cc_call="$CC $testsource $CPPFLAGS $LDFLAGS -o$testprog -static -lc" - if $cc_call && $testprog; then - gccwarning=0 - rm $testsource $testprog - else - echo; echo "Compilation command:"; echo "$cc_call" - rm $testsource - gccwarning=1 - host_cc=1 - cat <<EOF + cc_call="$CC $testsource $CPPFLAGS $LDFLAGS -o$testprog -static -lc" + if $cc_call && $testprog > /dev/null; then + gccwarning=0 + rm $testsource $testprog + if [ $quiet = 0 ]; then echo "... yes"; fi + else + echo; echo "Compilation command:"; echo "$cc_call" + rm $testsource + gccwarning=1 + host_cc=1 + cat <<EOF _______________________________________________________ !!!!!!!!!!!! Warning !!!!!!!!!!!! @@ -637,15 +758,14 @@ re-configure the project to fix this problem. $ export LDFLAGS="-L/PATH/TO/STATIC/LIBC \$LDFLAGS" $ export CPPFLAGS="-I/PATH/TO/SYS/CDEFS_H \$LDFLAGS" - _______________________________________________________ EOF + fi fi -fi -# Print a warning if GCC is not meant to be built. -if [ x"$gccwarning" = x1 ]; then + # Print a warning if GCC is not meant to be built. + if [ x"$gccwarning" = x1 ]; then cat <<EOF PLEASE SEE THE WARNINGS ABOVE. @@ -655,16 +775,38 @@ seconds and use your system's C compiler (it won't build a custom GCC). But please consider installing the necessary package(s) to complete your C compiler, then re-run './project configure'. -Project's configuration will continue in 5 seconds. +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. EOF - sleep 5 + sleep $pausesec + fi + elapsed_time_from_prev_step compiler-linkable-static fi +# Older C standard versions +# ------------------------- +# +# Some basic packages require an old standard of C compilation; for their +# list, see the 'Not working with C23' list in '../config/versions.conf'. +# Here, we first try 'gnu17', but if that fails on the host compiler, we +# fall back to 'gnu99'. +if [ $built_container = 0 ]; then + printf "int main(void){; return 0;}\n" > $testsource + if gcc -std=gnu17 $testsource -o $testprog 2> /dev/null; then + std_c_old="gnu17" + else std_c_old="gnu99" + fi + rm $testsource $testprog +fi + + + + # Fortran compiler # ---------------- # @@ -672,7 +814,7 @@ fi # have a fortran compiler: we'll build it internally for high-level # programs with GCC. However, when the host C compiler is to be used, the # user needs to have a Fortran compiler available. -if [ $host_cc = 1 ]; then +if [ $built_container = 0 ] && [ $host_cc = 1 ]; then # If a Fortran compiler is necessary, see if 'gfortran' exists and can # be used. @@ -705,8 +847,9 @@ EOF # Then, see if the Fortran compiler works testsourcef=$compilertestdir/test.f echo; echo; echo "Checking host Fortran compiler..."; - echo " PRINT *, \"... Fortran Compiler works.\"" > $testsourcef - echo " END" >> $testsourcef + echo " PRINT *, \"... Fortran Compiler works.\"" \ + > $testsourcef + echo " END" >> $testsourcef if gfortran $testsourcef -o$testprog && $testprog; then rm $testsourcef $testprog else @@ -732,6 +875,41 @@ EOF exit 1 fi fi + elapsed_time_from_prev_step compiler-fortran +fi + + + + + + + + + + +# Paths needed by the host compiler (only for 'basic.mk') +# ------------------------------------------------------- +# +# At the end of the basic build, we need to build GCC. But GCC will build +# in multiple phases, making its own simple compiler in order to build +# itself completely. The intermediate/simple compiler doesn't recognize +# some system specific locations like '/usr/lib/ARCHITECTURE' that some +# operating systems use. We thus need to tell the intermediate compiler +# where its necessary libraries and headers are. +if [ $built_container = 0 ]; then + if [ x"$sys_library_path" != x ]; then + if [ x"$LIBRARY_PATH" = x ]; then + export LIBRARY_PATH="$sys_library_path" + else + export LIBRARY_PATH="$LIBRARY_PATH:$sys_library_path" + fi + if [ x"$CPATH" = x ]; then + export CPATH="$sys_cpath" + else + export CPATH="$CPATH:$sys_cpath" + fi + fi + elapsed_time_from_prev_step compiler-paths fi @@ -743,7 +921,8 @@ fi # # Print some basic information so the user gets a feeling of what is going # on and is prepared on what will happen next. -cat <<EOF +if [ $quiet = 0 ]; then + cat <<EOF ----------------------------- Project's local configuration @@ -758,33 +937,29 @@ components from pre-defined webpages). It is STRONGLY recommended to read the description above each question before answering it. EOF +fi - -# What to do with possibly existing configuration file -# ---------------------------------------------------- +# Previous configuration +# ---------------------- # -# 'LOCAL.conf' is the top-most local configuration for the project. If it -# already exists when this script is run, we'll make a copy of it as backup -# (for example the user might have ran './project configure' by mistake). -printnotice=yes -rewritepconfig=yes -if [ -f $pconf ]; then +# 'LOCAL.conf' is the top-most local configuration for the project. At this +# point, if a LOCAL.conf exists within the '.build' symlink, we use it +# (instead of asking the user to interactively specify it). +rewritelconfig=yes +lconf=.build/software/config/LOCAL.conf +if [ -f $lconf ]; then if [ $existing_conf = 1 ]; then - printnotice=no - if [ -f $pconf ]; then rewritepconfig=no; fi + rewritelconfig=no; fi fi - - - # Make sure the group permissions satisfy the previous configuration (if it # exists and we don't want to re-write it). -if [ $rewritepconfig = no ]; then - oldgroupname=$(awk '/GROUP-NAME/ {print $3; exit 0}' $pconf) +if [ $rewritelconfig = no ]; then + oldgroupname=$(awk '/GROUP-NAME/ {print $3; exit 0}' $lconf) if [ "x$oldgroupname" = "x$maneage_group_name" ]; then just_a_place_holder_to_avoid_not_equal_test=1; else @@ -805,65 +980,9 @@ if [ $rewritepconfig = no ]; then echo " $confcommand"; echo exit 1 fi -fi - - - - -# Identify the downloader tool -# ---------------------------- -# -# After this script finishes, we will have both Wget and cURL for -# downloading any necessary dataset during the processing. However, to -# complete the configuration, we may also need to download the source code -# of some necessary software packages (including the downloaders). So we -# need to check the host's available tool for downloading at this step. -if [ $rewritepconfig = yes ]; then - if type wget > /dev/null 2>/dev/null; then - - # 'which' isn't in POSIX, so we are using 'command -v' instead. - name=$(command -v wget) - - # See if the host wget has the '--no-use-server-timestamps' option - # (for example wget 1.12 doesn't have it). If not, we'll have to - # remove it. This won't affect the analysis of Maneage in anyway, - # its just to avoid re-downloading if the server timestamps are - # bad; at the worst case, it will just cause a re-download of an - # input software source code (for data inputs, we will use our own - # wget that has this option). - tsname="no-use-server-timestamps" - tscheck=$(wget --help | grep $tsname || true) - if [ x"$tscheck" = x ]; then wgetts="" - else wgetts="--$tsname"; - fi - - # By default Wget keeps the remote file's timestamp, so we'll have - # to disable it manually. - downloader="$name $wgetts -O"; - elif type curl > /dev/null 2>/dev/null; then - name=$(command -v curl) - - # - cURL doesn't keep the remote file's timestamp by default. - # - With the '-L' option, we tell cURL to follow redirects. - downloader="$name -L -o" - else - cat <<EOF - -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -!!!!!!!!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!!!!!!!! -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - -Couldn't find GNU Wget, or cURL on this system. These programs are used for -downloading necessary programs and data if they aren't already present (in -directories that you can specify with this configure script). Therefore if -the necessary files are not present, the project will crash. - -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - -EOF - downloader="no-downloader-found" - fi; + # Report timing of this step if necessary. + elapsed_time_from_prev_step LOCAL-and-group-check fi @@ -873,7 +992,7 @@ fi # Build directory # --------------- currentdir="$(pwd)" -if [ $rewritepconfig = yes ]; then +if [ $rewritelconfig = yes ]; then cat <<EOF =============== @@ -901,12 +1020,18 @@ Do not choose any directory under the top source directory (this directory). The build directory cannot be a subdirectory of the source. --------------- +Build directory: + - Must be writable by running user. + - Not a sub-directory of the source directory. + - No meta-characters in name: SPACE ! ' @ # $ % ^ & * ( ) + ; + EOF bdir= junkname=pure-junk-974adfkj38 while [ x"$bdir" = x ] do - # Ask the user (if not already set on the command-line). + # Ask the user (if not already set on the command-line: 'build_dir' + # comes from the 'project' script). if [ x"$build_dir" = x ]; then if read -p"Please enter the top build directory: " build_dir; then @@ -948,9 +1073,11 @@ EOF # If it was newly created, it will be empty, so delete it. if ! [ "$(ls -A $bdir)" ]; then rm --dir "$bdir"; fi - # Inform the user that this is not acceptable and reset 'bdir'. + # Inform the user that this is not acceptable and reset + # 'bdir'. bdir= - echo " ** The build-directory cannot be under the source-directory." + printf " ** The build-directory cannot be under the " + printf "source-directory." fi fi @@ -959,7 +1086,8 @@ EOF # building. if ! [ x"$bdir" = x ]; then hasmeta=0; - case $bdir in *['!'\@\#\$\%\^\&\*\(\)\+\;\ ]* ) hasmeta=1 ;; esac + case $bdir in *['!'\@\#\$\%\^\&\*\(\)\+\;\ ]* ) hasmeta=1 ;; + esac if [ $hasmeta = 1 ]; then # If it was newly created, it will be empty, so delete it. @@ -967,9 +1095,10 @@ EOF # Inform the user and set 'bdir' to empty again. bdir= - echo " ** Build directory should not contain meta-characters" - echo " ** (like SPACE, %, \$, !, ;, or parenthesis, among " - echo " ** others): they can interrup the build for some software." + printf " ** Build directory should not contain " + printf "meta-characters (like SPACE, %, \$, !, ;, or " + printf "parenthesis, among others): they can interrup " + printf "the build for some software." fi fi @@ -980,16 +1109,29 @@ EOF if ! $(check_permission "$bdir"); then # Unable to handle permissions well bdir= - echo " ** File permissions can't be modified in this directory" + printf " ** File permissions can not be modified in " + printf "this directory" else # Able to handle permissions, now check for 5GB free space # in the given partition (note that the number is in units # of 1024 bytes). If this is not the case, print a warning. if $(free_space_warning 5000000 "$bdir"); then - echo " !! LESS THAN 5GB FREE SPACE IN: $bdir" - echo " !! We recommend choosing another partition." - echo " !! Build will continue in 5 seconds..." - sleep 5 + cat <<EOF + +_______________________________________________________ +!!!!!!!!!!!! Warning !!!!!!!!!!!! + +Less than 5GB free space in '$bdir'. We recommend choosing another +partition. Note that the software environment alone will take roughly +4.5GB, so if your datasets are large, it will fill up very soon. + +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +EOF + sleep $pausesec fi fi fi @@ -1003,9 +1145,42 @@ EOF echo " ** Please select another directory." echo "" else + # Set the '.build' and '.local' symbolic links (and delete + # possibly existing symbolic links). These commands are also + # present in the top-level 'project' script, but they are only + # invoked when '--build-dir' is called. When it is not called + # (the user wants to insert the directories interactively: the + # scenario here), the links need to be created from + # scratch. Furthermore, in case the given directory to + # '--build-dir' has problems (fails to pass the sanity checks + # above), the symbolic links also need to be recreated. + rm -f .build .local + ln -s $bdir .build + ln -s $bdir/software/installed .local + + # Inform the user echo " -- Build directory set to ($instring): '$bdir'" fi done + + # Report timing if necessary + elapsed_time_from_prev_step build-dir + +# The directory should be extracted from the existing LOCAL.conf, not from +# the command-line or in interactive mode. +else + + # Read the build directory from existing configuration file. It is + # assumed that 'LOCAL.conf' is created by this script (above the + # 'else') and that all the sanity checks there have already been + # applied. We'll just check if it is empty or not. + bdir=$(awk '$1=="BDIR" {print $3}' $lconf) + if [ x"$bdir" = x ]; then + printf "$scriptname: no value to 'BDIR' of '$lconf'. Please run " + printf "the project configuration again, but without " + printf "'--existing-conf' (or '-e')" + exit 1 + fi fi @@ -1014,13 +1189,10 @@ fi # Input directory # --------------- -if [ x"$input_dir" = x ]; then - indir="$optionaldir" -else - indir="$input_dir" +if [ x"$input_dir" = x ]; then indir="$optionaldir" +else indir="$input_dir" fi -noninteractive_sleep=2 -if [ $rewritepconfig = yes ] && [ x"$input_dir" = x ]; then +if [ $rewritelconfig = yes ]; then cat <<EOF ---------------------------------- @@ -1047,35 +1219,61 @@ don't want to make duplicates, you can create symbolic links to them and put those symbolic links in the given top-level directory. EOF - # Read the input directory if interactive mode is enabled. - if read -p"(OPTIONAL) Input datasets directory ($indir): " inindir; then - just_a_place_holder_to_avoid_not_equal_test=1; - else - echo "WARNING: interactive-mode seems to be disabled!" - echo "If you have a local copy of the inputs, use '--input-dir'." - echo "... project configuration will continue in $noninteractive_sleep sec ..." - sleep $noninteractive_sleep + # In case an input directory is not given, ask the user interactively. + if [ x"$input_dir" = x ]; then + + # Read the input directory if interactive mode is enabled. + if read -p"(OPTIONAL) Input datasets directory ($indir): " \ + inindir; then + just_a_place_holder_to_avoid_not_equal_test=1; + else + cat <<EOF +______________________________________________________ +!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!! + +WARNING: interactive-mode seems to be disabled! If you have a local copy of +the inputs, use '--input-dir'. Otherwise, all the data will be downloaded. + +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +EOF + sleep $pausesec + fi + else # An input directory was given. + inindir="$input_dir" fi - # In case an input-directory is given, write it in 'indir'. + # If the given string is not empty, write it in 'indir'. if [ x$inindir != x ]; then indir="$(absolute_dir "$inindir")" echo " -- Using '$indir'" fi + + # Report timing if necessary. + elapsed_time_from_prev_step input-dir + +# The directory should be extracted from the existing LOCAL.conf, not from +# the command-line or in interactive mode; similar to 'bdir' above. +else + indir=$(awk '$1=="INDIR" {print $3}' $lconf) fi + # Dependency tarball directory # ---------------------------- -if [ x"$software_dir" = x ]; then - ddir=$optionaldir -else - ddir=$software_dir +if [ x"$software_dir" = x ]; then ddir=$optionaldir +else ddir=$software_dir fi -if [ $rewritepconfig = yes ] && [ x"$software_dir" = x ]; then +if [ $rewritelconfig = yes ]; then + + # Print information. cat <<EOF --------------------------------------- @@ -1091,14 +1289,32 @@ of a dependency, it is necessary to have an internet connection because the project will download the tarballs it needs automatically. EOF - # Read the software directory if interactive mode is enabled. - if read -p"(OPTIONAL) Directory of dependency tarballs ($ddir): " tmpddir; then - just_a_place_holder_to_avoid_not_equal_test=1; + + # Ask the user for the software directory if it is not given as an + # option. + if [ x"$software_dir" = x ]; then + if read -p"(OPTIONAL) Directory of dependency tarballs ($ddir): " \ + tmpddir; then + just_a_place_holder_to_avoid_not_equal_test=1; + else + cat <<EOF +______________________________________________________ +!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!! + +WARNING: interactive-mode seems to be disabled! If you have a local copy of +the software source tarballs, use '--software-dir'. Otherwise, all the +necessary tarballs will be downloaded. + +The configuration will continue in $pausesec seconds. To avoid the +pause on such messages use the '--no-pause' option. + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +EOF + sleep $pausesec + fi else - echo "WARNING: interactive-mode seems to be disabled!" - echo "If you have a local copy of the software source, use '--software-dir'." - echo "... project configuration will continue in $noninteractive_sleep sec ..." - sleep $noninteractive_sleep + tmpddir="$software_dir" fi # If given, write the software directory. @@ -1106,105 +1322,174 @@ EOF ddir="$(absolute_dir "$tmpddir")" echo " -- Using '$ddir'" fi -fi +# The directory should be extracted from the existing LOCAL.conf, not from +# the command-line or in interactive mode; similar to 'bdir' above. +else + indir=$(awk '$1=="DEPENDENCIES-DIR" {print $3}' $lconf) +fi +elapsed_time_from_prev_step software-dir -# Write the parameters into the local configuration file. -if [ $rewritepconfig = yes ]; then - # Add commented notice. - create_file_with_notice $pconf +# Downloader +# ---------- +# +# After this script finishes, we will have both Wget and cURL for +# downloading any necessary dataset during the processing. However, to +# complete the configuration, we may also need to download the source code +# of some necessary software packages (including the downloaders). So we +# need to check the host's available tool for downloading at this step. +# +# NOTE: it is important that the name of the programs are not recorded to +# be absolute (for example 'wget' instead of '/usr/bin/wget'). This is +# because we build both 'wget' and 'curl' within 'basic.mk'. After they are +# built, Maneage's built software have higher precedence in 'PATH'. So +# without an absolute address, our own built software will be used as soon +# as they are installed/available. +if [ $rewritelconfig = yes ]; then + + # First look for wget on the host. + if type wget > /dev/null 2>/dev/null; then - # Write the values. - sed -e's|@bdir[@]|'"$bdir"'|' \ - -e's|@indir[@]|'"$indir"'|' \ - -e's|@ddir[@]|'"$ddir"'|' \ - -e's|@sys_cpath[@]|'"$sys_cpath"'|' \ - -e's|@downloader[@]|'"$downloader"'|' \ - -e's|@groupname[@]|'"$maneage_group_name"'|' \ - $pconf.in >> $pconf -else - # Read the values from existing configuration file. Note that the build - # directory may have space characters. Even though we currently check - # against it, we hope to be able to remove this condition in the - # future. - inbdir=$(awk '$1=="BDIR" { for(i=3; i<NF; i++) \ - printf "%s ", $i; \ - printf "%s", $NF }' $pconf) - - # Read the software directory (same as 'inbdir' above about space). - ddir=$(awk '$1=="DEPENDENCIES-DIR" { for(i=3; i<NF; i++) \ - printf "%s ", $i; \ - printf "%s", $NF}' $pconf) - - # The downloader command may contain multiple elements, so we'll just - # change the (in memory) first and second tokens to empty space and - # write the full line (the original file is unchanged). - downloader=$(awk '$1=="DOWNLOADER" {$1=""; $2=""; print $0}' $pconf) - - # Make sure all necessary variables have a value - err=0 - verr=0 - novalue="" - if [ x"$inbdir" = x ]; then novalue="BDIR, "; fi - if [ x"$downloader" = x ]; then novalue="$novalue"DOWNLOADER; fi - if [ x"$novalue" != x ]; then verr=1; err=1; fi - - # Make sure 'bdir' is an absolute path and it exists. - berr=0 - ierr=0 - bdir="$(absolute_dir "$inbdir")" - - if ! [ -d "$bdir" ]; then if ! mkdir "$bdir"; then berr=1; err=1; fi; fi - if [ $err = 1 ]; then - cat <<EOF + # See if the host wget has the '--no-use-server-timestamps' option + # (for example wget 1.12 doesn't have it). If not, we'll have to + # remove it. This won't affect the analysis of Maneage in anyway, + # its just to avoid re-downloading if the server timestamps are + # bad; at the worst case, it will just cause a re-download of an + # input software source code (for data inputs, we will use our own + # wget that has this option). + tsname="no-use-server-timestamps" + tscheck=$(wget --help | grep $tsname || true) + if [ x"$tscheck" = x ]; then wgetts="" + else wgetts="--$tsname"; + fi -################################################################# -######## ERORR reading existing configuration file ############ -################################################################# -EOF - if [ $verr = 1 ]; then - cat <<EOF + # By default Wget keeps the remote file's timestamp, so we'll have + # to disable it manually. + downloader="wget $wgetts -O"; -These variables have no value: $novalue. -EOF - fi - if [ $berr = 1 ]; then - cat <<EOF + # In case Wget doesn't exist, try cURL. + elif type curl > /dev/null 2>/dev/null; then -Couldn't create the build directory '$bdir' (value to 'BDIR') in -'$pconf'. -EOF - fi + # - cURL doesn't keep the remote file's timestamp by default. + # - With the '-L' option, we tell cURL to follow redirects. + downloader="curl -L -o" + # Neither Wget or cURL could be found. + else cat <<EOF -Please run the configure script again (accepting to re-write existing -configuration file) so all the values can be filled and checked. -################################################################# +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +!!!!!!!!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!!!!!!!! +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +Couldn't find GNU Wget, or cURL on this system. These programs are used for +downloading necessary programs and data if they aren't already present (in +directories that you can specify with this configure script). Therefore if +the necessary files are not present, the project will crash. + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + EOF + downloader="no-downloader-found" + fi; + +# The downloader should be extracted from the existing LOCAL.conf. +else + # The value will be a command (including white spaces), so we will read + # all the "fields" from the third to the end. + downloader=$(awk '$1=="DOWNLOADER" { for(i=3; i<NF; i++) \ + printf "%s ", $i; \ + printf "%s", $NF }' $lconf) + + if [ x"$downloader" = x ]; then + printf "$scriptname: no value to 'DOWNLOADER' of '$lconf'. " + printf "Please run the project configuration again, but " + printf "without '--existing-conf' (or '-e')" + exit 1 fi fi +elapsed_time_from_prev_step downloader -# Delete final configuration target -# --------------------------------- +# Libraries necessary for the system's shell +# ------------------------------------------ # -# We only want to start running the project later if this script has -# completed successfully. To make sure it hasn't crashed in the middle -# (without the user noticing), in the end of this script we make a file and -# we'll delete it here (at the start). Therefore if the script crashed in -# the middle that file won't exist. -sdir="$bdir"/software -finaltarget="$sdir"/configuration-done.txt -if ! [ -d "$sdir" ]; then mkdir "$sdir"; fi -rm -f "$finaltarget" +# In some cases (mostly the programs that Maneage doesn't yet build by +# itself), the programs may call the system's shell, not Maneage's +# shell. After we close-off the system environment from Maneage, this will +# cause a crash! To avoid such cases, we need to find the locations of the +# libraries that the shell needs and temporarily add them to the library +# search path. +# +# About the 'grep -v "(0x[^)]*)"' term (from bug 66847, see [1]): On some +# systems [2], the output of 'ldd /bin/sh' includes a line for the vDSO [3] +# that is different to the formats that are assumed, prior to this commit, +# by the algorithm in 'configure.sh' when evaluating the variable +# 'sys_library_sh_path'. This leads to a fatal syntax error in (at least) +# 'ncurses', because the option using 'sys_library_sh_path' contains an +# unquoted RAM address in parentheses. Even if the address were quoted, it +# would still be incorrect. This 'grep' command excludes candidate host +# path strings that look like RAM addresses to address the problem. +# +# [1] https://savannah.nongnu.org/bugs/index.php?66847 +# [2] https://stackoverflow.com/questions/34428037/how-to-interpret-the-output-of-the-ldd-program +# [3] man vdso +if [ $built_container = 0 ]; then + if [ x"$on_mac_os" = xyes ]; then + sys_library_sh_path=$(otool -L /bin/sh \ + | awk '/\/lib/{print $1}' \ + | sed 's#/[^/]*$##' \ + | sort \ + | uniq \ + | awk '{if (NR==1) printf "%s", $1; \ + else printf ":%s", $1}') + else + sys_library_sh_path=$(ldd /bin/sh \ + | awk '{if($3!="") print $3}' \ + | sed 's#/[^/]*$##' \ + | grep -v "(0x[^)]*)" \ + | sort \ + | uniq \ + | awk '{if (NR==1) printf "%s", $1; \ + else printf ":%s", $1}') + fi + elapsed_time_from_prev_step sys-library-sh-path +fi + + + + + +# When no local configuration existed, write the parameters into the local +# configuration file. +sdir=$bdir/software +sconfdir=$sdir/config +if ! [ -d "$sdir" ]; then mkdir "$sdir"; fi +if ! [ -d "$sconfdir" ]; then mkdir "$sconfdir"; fi +if [ $rewritelconfig = yes ]; then + + # Put the basic comments at the top of the file. + create_file_with_notice $lconf + + # Write the values. + lconfin=$cdir/LOCAL.conf.in + sed -e's|@bdir[@]|'"$bdir"'|' \ + -e's|@indir[@]|'"$indir"'|' \ + -e's|@ddir[@]|'"$ddir"'|' \ + -e's|@sys_cpath[@]|'"$sys_cpath"'|' \ + -e's|@downloader[@]|'"$downloader"'|' \ + -e's|@groupname[@]|'"$maneage_group_name"'|' \ + -e's|@sys_library_sh_path[@]|'"$sys_library_sh_path"'|' \ + $lconfin >> $lconf +fi +elapsed_time_from_prev_step LOCAL-write @@ -1217,123 +1502,92 @@ rm -f "$finaltarget" # avoid too many directory dependencies throughout the software and # analysis Makefiles (thus making them hard to read), we are just building # them here -# Software tarballs tardir="$sdir"/tarballs -if ! [ -d "$tardir" ]; then mkdir "$tardir"; fi - -# Installed software instdir="$sdir"/installed -if ! [ -d "$instdir" ]; then mkdir "$instdir"; fi +tmpblddir="$sdir"/build-tmp -# To record software versions and citation. +# Second-level directories. +instlibdir="$instdir"/lib +instbindir="$instdir"/bin verdir="$instdir"/version-info -if ! [ -d "$verdir" ]; then mkdir "$verdir"; fi -# Program and library versions and citation. -ibidir="$verdir"/proglib -if ! [ -d "$ibidir" ]; then mkdir "$ibidir"; fi - -# Python module versions and citation. +# Sub-directories of version-info +itidir="$verdir"/tex +ictdir="$verdir"/cite ipydir="$verdir"/python -if ! [ -d "$ipydir" ]; then mkdir "$ipydir"; fi - -# R module versions and citation. +ibidir="$verdir"/proglib ircrandir="$verdir"/r-cran -if ! [ -d "$ircrandir" ]; then mkdir "$ircrandir"; fi - -# Used software BibTeX entries. -ictdir="$verdir"/cite -if ! [ -d "$ictdir" ]; then mkdir "$ictdir"; fi - -# TeXLive versions. -itidir="$verdir"/tex -if ! [ -d "$itidir" ]; then mkdir "$itidir"; fi - -# Some software install their libraries in '$(idir)/lib64'. But all other -# libraries are in '$(idir)/lib'. Since Maneage's build is only for a -# single architecture, we can set the '$(idir)/lib64' as a symbolic link to -# '$(idir)/lib' so all the libraries are always available in the same -# place. -instlibdir="$instdir"/lib -if ! [ -d "$instlibdir" ]; then mkdir "$instlibdir"; fi -ln -fs "$instlibdir" "$instdir"/lib64 - -# Wrapper over Make as a single command so it does not default to '/bin/sh' -# during installation (needed by some programs like CMake). -instbindir=$instdir/bin -if ! [ -d $instbindir ]; then mkdir $instbindir; fi -makewshell="$instbindir/make-with-shell" -echo "$instbindir/make SHELL=$instbindir/bash \$@" > $makewshell -chmod +x $makewshell - - - - +if [ $built_container = 0 ]; then + + # Top-level directories. + if ! [ -d "$tardir" ]; then mkdir "$tardir"; fi + if ! [ -d "$instdir" ]; then mkdir "$instdir"; fi + + # Second-level directories. + if ! [ -d "$verdir" ]; then mkdir "$verdir"; fi + if ! [ -d "$instbindir" ]; then mkdir "$instbindir"; fi + + # Sub-directories of version-info + if ! [ -d "$itidir" ]; then mkdir "$itidir"; fi + if ! [ -d "$ictdir" ]; then mkdir "$ictdir"; fi + if ! [ -d "$ipydir" ]; then mkdir "$ipydir"; fi + if ! [ -d "$ibidir" ]; then mkdir "$ibidir"; fi + if ! [ -d "$ircrandir" ]; then mkdir "$ircrandir"; fi + + # Some software install their libraries in '$(idir)/lib64'. But all + # other libraries are in '$(idir)/lib'. Since Maneage's build is only + # for a single architecture, we can set the '$(idir)/lib64' as a + # symbolic link to '$(idir)/lib' so all the libraries are always + # available in the same place. + if ! [ -d "$instlibdir" ]; then mkdir "$instlibdir"; fi + ln -fs "$instlibdir" "$instdir"/lib64 + + # Wrapper over Make as a single command so it does not default to + # '/bin/sh' during installation (needed by some programs like CMake). + makewshell="$instbindir/make-with-shell" + if ! [ -f "$makewshell" ]; then + echo "$instbindir/make SHELL=$instbindir/bash \$@" > $makewshell + chmod +x $makewshell + fi -# Project's top-level built analysis directories -# ---------------------------------------------- + # Report the execution time of this step. + elapsed_time_from_prev_step subdirectories-of-build +fi -# Top-level built analysis directories. -badir="$bdir"/analysis -if ! [ -d "$badir" ]; then mkdir "$badir"; fi -# Top-level LaTeX. -texdir="$badir"/tex -if ! [ -d "$texdir" ]; then mkdir "$texdir"; fi -# LaTeX macros. -mtexdir="$texdir"/macros -if ! [ -d "$mtexdir" ]; then mkdir "$mtexdir"; fi -# TeX build directory. If built in a group scenario, the TeX build -# directory must be separate for each member (so they can work on their -# relevant parts of the paper without conflicting with each other). -if [ "x$maneage_group_name" = x ]; then - texbdir="$texdir"/build -else - user=$(whoami) - texbdir="$texdir"/build-$user -fi -if ! [ -d "$texbdir" ]; then mkdir "$texbdir"; fi - -# TiKZ (for building figures within LaTeX). -tikzdir="$texbdir"/tikz -if ! [ -d "$tikzdir" ]; then mkdir "$tikzdir"; fi - -# If 'tex/build' and 'tex/tikz' are symbolic links then 'rm -f' will delete -# them and we can continue. However, when the project is being built from -# the tarball, these two are not symbolic links but actual directories with -# the necessary built-components to build the PDF in them. In this case, -# because 'tex/build' is a directory, 'rm -f' will fail, so we'll just -# rename the two directories (as backup) and let the project build the -# proper symbolic links afterwards. -if rm -f tex/build; then - rm -f tex/tikz -else - mv tex/tikz tex/tikz-from-tarball - mv tex/build tex/build-from-tarball -fi -# Set the symbolic links for easy access to the top project build -# directories. Note that these are put in each user's source/cloned -# directory, not in the build directory (which can be shared between many -# users and thus may already exist). +# See if the linker accepts -Wl,-rpath-link +# ----------------------------------------- # -# Note: if we don't delete them first, it can happen that an extra link -# will be created in each directory that points to its parent. So to be -# safe, we are deleting all the links on each re-configure of the -# project. Note that at this stage, we are using the host's 'ln', not our -# own, so its best not to assume anything (like 'ln -sf'). -rm -f .build .local - -ln -s "$bdir" .build -ln -s "$instdir" .local -ln -s "$texdir" tex/build -ln -s "$tikzdir" tex/tikz - -# --------- Delete for no Gnuastro --------- -rm -f .gnuastro -# ------------------------------------------ +# '-rpath-link' is used to write the information of the linked shared +# library into the shared object (library or program). But some versions of +# LLVM's linker don't accept it an can cause problems. +# +# IMPORTANT NOTE: This test has to be done **AFTER** the definition of +# 'instdir'. Otherwise, the rpath-link value set within rpath_command in +# the case of a successful test compile (if $CC ...) will be "/lib", +# i.e. the host system root /lib directory, instead of the maneage library +# directory. +if [ $built_container = 0 ]; then + cat > $testsource <<EOF +#include <stdio.h> +#include <stdlib.h> +int main(void) {return EXIT_SUCCESS;} +EOF + if $CC $testsource -o$testprog -Wl,-rpath-link 2>/dev/null \ + > /dev/null; then + export rpath_command="-Wl,-rpath-link=$instdir/lib" + else + export rpath_command="" + fi + + # Delete the temporary directory for compiler checking. + rm -f $testprog $testsource + rm -r $compilertestdir + elapsed_time_from_prev_step compiler-rpath +fi @@ -1347,120 +1601,116 @@ rm -f .gnuastro # HDDs/SSDs and improve speed, it is therefore better to build them in the # RAM when possible. The RAM of most systems today (>8GB) is large enough # for the parallel building of the software. - +# # Set the top-level shared memory location. Currently there is only one # standard location (for GNU/Linux OSs), so doing this check here and the # main job below may seem redundant. However, it is written separately from # the main code below because later, we expect to add more possible # mounting locations (for other OSs). -if [ -d /dev/shm ]; then shmdir=/dev/shm -else shmdir="" -fi +if [ $built_container = 0 ]; then + if [ -d /dev/shm ]; then shmdir=/dev/shm + else shmdir="" + fi -# If a shared memory mounted directory exists and has the necessary -# conditions, set that directory to build software. -if [ x"$shmdir" != x ]; then - - # Make sure it has enough space. - needed_space=2000000 - available_space=$(df "$shmdir" | awk 'NR==2{print $4}') - if [ $available_space -gt $needed_space ]; then - - # Set the Maneage-specific directory within the shared - # memory. We'll use the names of the two parent directories to the - # current/running directory, separated by a '-' instead of - # '/'. We'll then appended that with the user's name (in case - # multiple users may be working on similar project names). - # - # Maybe later, we can use something like 'mktemp' to add random - # characters to this name and make it unique to every run (even for - # a single user). - dirname=$(pwd | sed -e's/\// /g' \ - | awk '{l=NF-1; printf("%s-%s", $l, $NF)}') - tbshmdir="$shmdir"/"$dirname"-$(whoami) - - # Try to make the directory if it does not yet exist. A failed - # directory creation will be tested for a few lines later, when - # testing for the existence and executability of a test file. - if ! [ -d "$tbshmdir" ]; then (mkdir "$tbshmdir" || true); fi - - # Some systems may protect '/dev/shm' against the right to execute - # programs by ordinary users. We thus need to check that the device - # allows execution within this directory by this user. - shmexecfile="$tbshmdir"/shm-execution-check.sh - rm -f $shmexecfile # We also don't want any existing flags. - - # Create the file to be executed, but do not fail fatally if it - # cannot be created. We will check a few lines later if the file - # really exists. - (cat > "$shmexecfile" <<EOF || true) + # If a shared memory mounted directory exists and has the necessary + # conditions, set that directory to build software. + if [ x"$shmdir" != x ]; then + + # Make sure it has enough space. + needed_space=2000000 + available_space=$(df "$shmdir" | awk 'NR==2{print $4}') + if [ $available_space -gt $needed_space ]; then + + # Set the Maneage-specific directory within the shared + # memory. We'll use the names of the two parent directories to + # the current/running directory, separated by a '-' instead of + # '/'. We'll then appended that with the user's name (in case + # multiple users may be working on similar project names). + # + # Maybe later, we can use something like 'mktemp' to add random + # characters to this name and make it unique to every run (even + # for a single user). + dirname=$(pwd | sed -e's/\// /g' \ + | awk '{l=NF-1; printf("%s-%s", $l, $NF)}') + tbshmdir="$shmdir"/"$dirname"-$(whoami) + + # Try to make the directory if it does not yet exist. A failed + # directory creation will be tested for a few lines later, when + # testing for the existence and executability of a test file. + if ! [ -d "$tbshmdir" ]; then (mkdir "$tbshmdir" || true); fi + + # Some systems may protect '/dev/shm' against the right to + # execute programs by ordinary users. We thus need to check + # that the device allows execution within this directory by + # this user. + shmexecfile="$tbshmdir"/shm-execution-check.sh + rm -f $shmexecfile # We also don't want any existing flags. + + # Create the file to be executed, but do not fail fatally if it + # cannot be created. We will check a few lines later if the + # file really exists. + (cat > "$shmexecfile" <<EOF || true) #!/bin/sh -echo "This file successfully executed." +a=b EOF - # If the file was successfully created, then make the file - # executable and see if it runs. If not, set 'tbshmdir' to an empty - # string so it is not used in later steps. In any case, delete the - # temporary file afterwards. - # - # We aren't adding '&> /dev/null' after the execution command - # because it can produce false failures randomly on some systems. - if [ -e "$shmexecfile" ]; then - - # Add the executable flag. - chmod +x "$shmexecfile" - - # The following line tries to execute the file. - if "$shmexecfile"; then - # Successful execution. The colon is a "no-op" (no - # operation) shell command. - : + # If the file was successfully created, then make the file + # executable and see if it runs. If not, set 'tbshmdir' to an + # empty string so it is not used in later steps. In any case, + # delete the temporary file afterwards. + # + # We aren't adding '&> /dev/null' after the execution command + # because it can produce false failures randomly on some + # systems. + if [ -e "$shmexecfile" ]; then + + # Add the executable flag. + chmod +x "$shmexecfile" + + # The following line tries to execute the file. + if "$shmexecfile"; then + # Successful execution. The colon is a "no-op" (no + # operation) shell command. + : + else + tbshmdir="" + fi + rm "$shmexecfile" else tbshmdir="" fi - rm "$shmexecfile" - else - tbshmdir="" fi + else + tbshmdir="" fi -else - tbshmdir="" -fi - - - + # If a shared memory directory was created, set the software building + # directory to be a symbolic link to it. Otherwise, just build the + # temporary build directory under the project's build directory. + # + # If it is a link, we need to empty its contents first, then itself. + if [ -d "$tmpblddir" ]; then empty_build_tmp; fi + + # Now that we are sure it doesn't exist, we'll make it (either as a + # directory or as a symbolic link). + if [ x"$tbshmdir" = x ]; then mkdir "$tmpblddir"; + else ln -s "$tbshmdir" "$tmpblddir"; + fi -# If a shared memory directory was created, set the software building -# directory to be a symbolic link to it. Otherwise, just build the -# temporary build directory under the project's build directory. -tmpblddir="$sdir"/build-tmp -rm -rf "$tmpblddir"/* "$tmpblddir" # If it is a link, we need to empty - # its contents first, then itself. -if [ x"$tbshmdir" = x ]; then mkdir "$tmpblddir"; -else ln -s "$tbshmdir" "$tmpblddir"; + # Report the time this step took. + elapsed_time_from_prev_step temporary-software-building-dir fi -# Make sure the temporary build directory is empty (un-finished -# source/build files from previous builds can remain there during debugging -# or software updates). -rm -rf $tmpblddir/* - - - - - # Inform the user that the build process is starting # ------------------------------------------------- # # Everything is ready, let the user know that the building is going to # start. -if [ $printnotice = yes ]; then - tsec=10 +if [ $quiet = 0 ]; then cat <<EOF ------------------------- @@ -1475,20 +1725,20 @@ NOTE: the built software will NOT BE INSTALLED in standard places of your OS (so no root access is required). They are only for local usage by this project. -**TIP**: you can see which software are being installed at every moment -with the following command. See "Inspecting status" section of -'README-hacking.md' for more. In short, run it while the project is being -configured (in another terminal, but in this same directory: -'$currentdir'): +TIP: you can see which software are being installed at every moment with +the following command. See "Inspecting status" section of +'README-hacking.md' for more. In short, run it in another terminal while +the project is being configured. $ ./project --check-config -Project's configuration will continue in $tsec seconds. +Project's configuration will continue in $tsec seconds. To avoid the pause +on such messages use the '--no-pause' option. ------------------------- EOF - sleep $tsec + sleep $pausesec fi @@ -1504,123 +1754,20 @@ fi # - On BSD-based systems (for example FreeBSD and macOS), we have a # 'hw.ncpu' in the output of 'sysctl'. # - When none of the above work, just set the number of threads to 1. -if [ $jobs = 0 ]; then - if type nproc > /dev/null 2> /dev/null; then - numthreads=$(nproc --all); - else - numthreads=$(sysctl -a | awk '/^hw\.ncpu/{print $2}') - if [ x"$numthreads" = x ]; then numthreads=1; fi - fi -else - numthreads=$jobs -fi - - - - - -# See if the linker accepts -Wl,-rpath-link -# ----------------------------------------- -# -# '-rpath-link' is used to write the information of the linked shared -# library into the shared object (library or program). But some versions of -# LLVM's linker don't accept it an can cause problems. -# -# IMPORTANT NOTE: This test has to be done **AFTER** the definition of -# 'instdir', otherwise, it is going to be used as an empty string. -cat > $testsource <<EOF -#include <stdio.h> -#include <stdlib.h> -int main(void) {return EXIT_SUCCESS;} -EOF -if $CC $testsource -o$testprog -Wl,-rpath-link 2>/dev/null > /dev/null; then - export rpath_command="-Wl,-rpath-link=$instdir/lib" -else - export rpath_command="" -fi - - - - - -# Delete the compiler testing directory -# ------------------------------------- # -# This directory was made above to make sure the necessary compilers can be -# run. -rm -f $testprog $testsource -rm -rf $compilertestdir - - - - - -# Paths needed by the host compiler (only for 'basic.mk') -# ------------------------------------------------------- -# -# At the end of the basic build, we need to build GCC. But GCC will build -# in multiple phases, making its own simple compiler in order to build -# itself completely. The intermediate/simple compiler doesn't recognize -# some system specific locations like '/usr/lib/ARCHITECTURE' that some -# operating systems use. We thus need to tell the intermediate compiler -# where its necessary libraries and headers are. -if [ x"$sys_library_path" != x ]; then - if [ x"$LIBRARY_PATH" = x ]; then - export LIBRARY_PATH="$sys_library_path" - else - export LIBRARY_PATH="$LIBRARY_PATH:$sys_library_path" - fi - if [ x"$CPATH" = x ]; then - export CPATH="$sys_cpath" +# This check is also used in 'reproduce/software/shell/docker.sh'. +if [ $built_container = 0 ]; then + if [ $jobs = 0 ]; then + if type nproc > /dev/null 2> /dev/null; then + numthreads=$(nproc --all); + else + numthreads=$(sysctl -a | awk '/^hw\.ncpu/{print $2}') + if [ x"$numthreads" = x ]; then numthreads=1; fi + fi else - export CPATH="$CPATH:$sys_cpath" + numthreads=$jobs fi -fi - - - - - -# Libraries necessary for the system's shell -# ------------------------------------------ -# -# In some cases (mostly the programs that Maneage doesn't yet build by -# itself), the programs may call the system's shell, not Maneage's -# shell. After we close-off the system environment from Maneage, this will -# cause a crash! To avoid such cases, we need to find the locations of the -# libraries that the shell needs and temporarily add them to the library -# search path. -# -# About the 'grep -v "(0x[^)]*)"' term (from bug 66847, see [1]): On some -# systems [2], the output of 'ldd /bin/sh' includes a line for the vDSO [3] -# that is different to the formats that are assumed, prior to this commit, -# by the algorithm in 'configure.sh' when evaluating the variable -# 'sys_library_sh_path'. This leads to a fatal syntax error in (at least) -# 'ncurses', because the option using 'sys_library_sh_path' contains an -# unquoted RAM address in parentheses. Even if the address were quoted, it -# would still be incorrect. This 'grep command excludes candidate host path -# strings that look like RAM addresses to address the problem. -# -# [1] https://savannah.nongnu.org/bugs/index.php?66847 -# [2] https://stackoverflow.com/questions/34428037/how-to-interpret-the-output-of-the-ldd-program -# [3] man vdso -if [ x"$on_mac_os" = xyes ]; then - sys_library_sh_path=$(otool -L /bin/sh \ - | awk '/\/lib/{print $1}' \ - | sed 's#/[^/]*$##' \ - | sort \ - | uniq \ - | awk '{if (NR==1) printf "%s", $1; \ - else printf ":%s", $1}') -else - sys_library_sh_path=$(ldd /bin/sh \ - | awk '{if($3!="") print $3}' \ - | sed 's#/[^/]*$##' \ - | grep -v "(0x[^)]*)" \ - | sort \ - | uniq \ - | awk '{if (NR==1) printf "%s", $1; \ - else printf ":%s", $1}') + elapsed_time_from_prev_step num-threads fi @@ -1644,42 +1791,39 @@ fi # which will download the DOI-resolved webpage, and extract the Zenodo-URL # of the most recent version from there (using the 'coreutils' tarball as # an example, the directory part of the URL for all the other software are -# the same). This is not done if the options '--debug' or `--offline` are used. -zenodourl="" -user_backup_urls="" -zenodocheck=.build/software/zenodo-check.html -if [ x$debug = x ] && [ x$offline = x ]; then - if $downloader $zenodocheck https://doi.org/10.5281/zenodo.3883409; then - zenodourl=$(sed -n -e'/coreutils/p' $zenodocheck \ - | sed -n -e'/http/p' \ - | tr ' ' '\n' \ - | grep http \ - | sed -e 's/href="//' -e 's|/coreutils| |' \ - | awk 'NR==1{print $1}') - fi -fi -rm -f $zenodocheck - -# Add the Zenodo URL to the user's given back software URLs. Since the user -# can specify 'user_backup_urls' (not yet implemented as an option in -# './project'), we'll give preference to their specified servers, then add -# the Zenodo URL afterwards. -user_backup_urls="$user_backup_urls $zenodourl" - - - - - -# Build core tools for project -# ---------------------------- +# the same). This is not done if the options '--debug' or `--offline` are +# used. # -# Here we build the core tools that 'basic.mk' depends on: Lzip -# (compression program), GNU Make (that 'basic.mk' is written in), Dash -# (minimal Bash-like shell) and Flock (to lock files and enable serial -# download). -export on_mac_os -./reproduce/software/shell/pre-make-build.sh \ - "$bdir" "$ddir" "$downloader" "$user_backup_urls" +# NOTE on commented region below (starting with '##'): As described [1], +# Zenodo's upload strategy does not allow a repository with more than one +# hundred files. So until the solution there has been implemented, this +# step has been commented. +# +# [1] https://savannah.nongnu.org/task/?16621 +## zenodourl="" +## user_backup_urls="" +## zenodocheck="$bdir"/software/zenodo-check.html +## if [ $built_container = 0 ]; then +## if [ x$debug = x ] && [ x$offline = x ]; then +## if $downloader $zenodocheck \ +## https://doi.org/10.5281/zenodo.3883409; then +## zenodourl=$(sed -n -e'/coreutils/p' $zenodocheck \ +## | sed -n -e'/http/p' \ +## | tr ' ' '\n' \ +## | grep http \ +## | sed -e 's/href="//' -e 's|/coreutils| |' \ +## | awk 'NR==1{print $1}') +## fi +## fi +## rm -f $zenodocheck +## +## # Add the Zenodo URL to the user's given back software URLs. Since the +## # user can specify 'user_backup_urls' (not yet implemented as an option +## # in './project'), we'll give preference to their specified servers, +## # then add the Zenodo URL afterwards. +## user_backup_urls="$user_backup_urls $zenodourl" +## elapsed_time_from_prev_step zenodo-url +## fi @@ -1707,13 +1851,29 @@ fi -# Build other basic tools our own GNU Make -# ---------------------------------------- +# Core software +# ------------- +# +# Here we build the core tools that 'basic.mk' depends on: Lzip +# (compression program), GNU Make (that 'basic.mk' is written in), Dash +# (minimal Bash-like shell) and Flock (to lock files and enable serial +# operations where necessary: mostly in download). +export on_mac_os +if [ $quiet = 0 ]; then echo "Building/validating software: pre-make"; fi +./reproduce/software/shell/pre-make-build.sh \ + "$bdir" "$ddir" "$downloader" "$user_backup_urls" +elapsed_time_from_prev_step make-software-pre-make + + + + + +# Basic software +# -------------- # -# When building these software we don't have our own un-packing software, -# Bash, Make, or AWK. In this step, we'll install such low-level basic -# tools, but we have to be very portable (and use minimal features in all). -echo; echo "Building necessary software (if necessary)..." +# Having built the core tools, we are now ready to build GCC and all its +# dependencies (the "basic" software). +if [ $quiet = 0 ]; then echo "Building/validating software: basic"; fi .local/bin/make $keepgoing -f reproduce/software/make/basic.mk \ sys_library_sh_path=$sys_library_sh_path \ user_backup_urls="$user_backup_urls" \ @@ -1723,25 +1883,22 @@ echo; echo "Building necessary software (if necessary)..." numthreads=$numthreads \ needs_ldl=$needs_ldl \ on_mac_os=$on_mac_os \ + std_c_old=$std_c_old \ host_cc=$host_cc \ -j$numthreads +elapsed_time_from_prev_step make-software-basic -# All other software -# ------------------ +# High-level software +# ------------------- # -# We will be making all the dependencies before running the top-level -# Makefile. To make the job easier, we'll do it in a Makefile, not a -# script. Bash and Make were the tools we need to run Makefiles, so we had -# to build them in this script. But after this, we can rely on Makefiles. -if [ $jobs = 0 ]; then - numthreads=$(.local/bin/nproc --all) -else - numthreads=$jobs -fi +# Having our custom GCC in place, we can now build the high-level (science) +# software: we are using our custom-built 'env' to ensure that nothing from +# the host environment leaks into the high-level software environment. +if [ $quiet = 0 ]; then echo "Building/validating software: high-level"; fi .local/bin/env -i HOME=$bdir \ .local/bin/make $keepgoing \ -f reproduce/software/make/high-level.mk \ @@ -1757,16 +1914,7 @@ fi host_cc=$host_cc \ offline=$offline \ -j$numthreads - - - - - -# Delete the temporary Make wrapper -# --------------------------------- -# -# See above for its description. -rm $makewshell +elapsed_time_from_prev_step make-software-high-level @@ -1781,22 +1929,23 @@ rm $makewshell # will just stop at the stage when all the processing is complete and it is # only necessary to build the PDF. So we don't want to stop the project's # configuration and building if its not present. -if [ -f $itidir/texlive-ready-tlmgr ]; then - texlive_result=$(cat $itidir/texlive-ready-tlmgr) -else - texlive_result="NOT!" -fi -if [ x"$texlive_result" = x"NOT!" ]; then - cat <<EOF +if [ $built_container = 0 ]; then + if [ -f $itidir/texlive-ready-tlmgr ]; then + texlive_result=$(cat $itidir/texlive-ready-tlmgr) + else + texlive_result="NOT!" + fi + if [ x"$texlive_result" = x"NOT!" ]; then + cat <<EOF -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -!!!!!!!!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!!!!!!!! -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +______________________________________________________ +!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!! TeX Live couldn't be installed during the configuration (probably because -there were downloading problems). TeX Live is only necessary in making the -final PDF (which is only done after all the analysis has been complete). It -is not used at all during the analysis. +there were downloading problems, or you used the '--offline' option). TeX +Live is only necessary in making the final PDF (which is only done after +all the analysis has been complete). It is not used at all during the +analysis. Therefore, if you don't need the final PDF, and just want to do the analysis, you can safely ignore this warning and continue. @@ -1811,18 +1960,23 @@ and re-run configure: ./project configure -e -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +The configuration will continue in $pausesec seconds. To avoid the pause on +such messages use the '--no-pause' option. + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF - sleep 10 # increase the chance that an interactive user reads this message + sleep $pausesec + fi + elapsed_time_from_prev_step check-tex-installation fi -# Citation of installed software +# Software information the paper +# ------------------------------ # # After everything is installed, we'll put all the names and versions in a # human-readable paragraph and also prepare the BibTeX citation for the @@ -1864,101 +2018,101 @@ prepare_name_version () fi } -# Import the context/sentences for placing between the list of software -# names during their acknowledgment. -. $cdir/software_acknowledge_context.sh - -# Report the different software in separate contexts (separating Python and -# TeX packages from the C/C++ programs and libraries). -proglibs=$(prepare_name_version $verdir/proglib/*) -pymodules=$(prepare_name_version $verdir/python/*) -texpkg=$(prepare_name_version $verdir/tex/texlive) - -# Acknowledge these software packages in a LaTeX paragraph. -pkgver=$mtexdir/dependencies.tex - -# Add the text to the ${pkgver} file. -.local/bin/echo "$thank_software_introduce " > $pkgver -.local/bin/echo "$thank_progs_libs $proglibs. " >> $pkgver -if [ x"$pymodules" != x ]; then - .local/bin/echo "$thank_python $pymodules. " >> $pkgver -fi -.local/bin/echo "$thank_latex $texpkg. " >> $pkgver -.local/bin/echo "$thank_software_conclude" >> $pkgver - -# Prepare the BibTeX entries for the used software (if there are any). -hasentry=0 -bibfiles="$ictdir/*" -for f in $bibfiles; do if [ -f $f ]; then hasentry=1; break; fi; done; - -# Make sure we start with an empty output file. -pkgbib=$mtexdir/dependencies-bib.tex -echo "" > $pkgbib - -# Fill it in with all the BibTeX entries in this directory. We'll just -# avoid writing any comments (usually copyright notices) and also put an -# empty line after each file's contents to make the output more readable. -if [ $hasentry = 1 ]; then - for f in $bibfiles; do - awk '!/^%/{print} END{print ""}' $f >> $pkgbib - done -fi - +# Relevant files +pkgver=$sconfdir/dependencies.tex +pkgbib=$sconfdir/dependencies-bib.tex +# Build the software LaTeX source but only when not in a container. +if [ $built_container = 0 ]; then + # Import the context/sentences for placing between the list of software + # names during their acknowledgment. + . $cdir/software_acknowledge_context.sh + # Report the different software in separate contexts (separating Python + # and TeX packages from the C/C++ programs and libraries). + proglibs=$(prepare_name_version $verdir/proglib/*) + pymodules=$(prepare_name_version $verdir/python/*) + texpkg=$(prepare_name_version $verdir/tex/texlive) -# Report machine architecture -# --------------------------- -# -# Report hardware -hwparam="$mtexdir/hardware-parameters.tex" + # Acknowledge these software packages in a LaTeX paragraph. + .local/bin/echo "$thank_software_introduce " > $pkgver + .local/bin/echo "$thank_progs_libs $proglibs. " >> $pkgver + if [ x"$pymodules" != x ]; then + .local/bin/echo "$thank_python $pymodules. " >> $pkgver + fi + .local/bin/echo "$thank_latex $texpkg. " >> $pkgver + .local/bin/echo "$thank_software_conclude" >> $pkgver + + # Prepare the BibTeX entries for the used software (if there are any). + hasentry=0 + bibfiles="$ictdir/*" + for f in $bibfiles; do if [ -f $f ]; then hasentry=1; break; fi; done; + + # Fill it in with all the BibTeX entries in this directory. We'll just + # avoid writing any comments (usually copyright notices) and also put an + # empty line after each file's contents to make the output more readable. + echo "" > $pkgbib # We don't want to inherit any pre-existing content. + if [ $hasentry = 1 ]; then + for f in $bibfiles; do + awk '!/^%/{print} END{print ""}' $f >> $pkgbib + done + fi -# Add the text to the ${hwparam} file. Since harware class might include -# underscore, it must be replaced with '\_', otherwise pdftex would -# complain and break the build process when doing ./project make. -hw_class_fixed="$(echo $hw_class | sed -e 's/_/\\_/')" -.local/bin/echo "\\newcommand{\\machinearchitecture}{$hw_class_fixed}" > $hwparam -.local/bin/echo "\\newcommand{\\machinebyteorder}{$byte_order}" >> $hwparam -.local/bin/echo "\\newcommand{\\machineaddresssizes}{$address_sizes}" >> $hwparam + # Report the time that this operation took. + elapsed_time_from_prev_step tex-macros +fi -# Clean the temporary build directory -# --------------------------------- +# Report machine architecture (has to be final created file) +# ---------------------------------------------------------- # -# By the time the script reaches here the temporary software build -# directory should be empty, so just delete it. Note 'tmpblddir' may be a -# symbolic link to shared memory. So, to work in any scenario, first delete -# the contents of the directory (if it has any), then delete 'tmpblddir'. -.local/bin/rm -rf $tmpblddir/* $tmpblddir - - - - - -# Register successful completion -# ------------------------------ -echo `.local/bin/date` > $finaltarget - +# This is the final file that is created in the configuration phase: it is +# used by the high-level project script to verify that configuration has +# been completed. If any other files should be created in the final statges +# of configuration, be sure to add them before this. +# +# Since harware class might include underscore, it must be replaced with +# '\_', otherwise pdftex would complain and break the build process when +# doing ./project make. +if [ $built_container = 0 ]; then + hw_class=$(uname -m) + hwparam="$sconfdir/hardware-parameters.tex" + hw_class_fixed="$(echo $hw_class | sed -e 's/_/\\_/')" + .local/bin/echo "\\newcommand{\\machinearchitecture}{$hw_class_fixed}" \ + > $hwparam + .local/bin/echo "\\newcommand{\\machinebyteorder}{$byte_order}" \ + >> $hwparam + .local/bin/echo "\\newcommand{\\machineaddresssizes}{$address_sizes}" \ + >> $hwparam + elapsed_time_from_prev_step hardware-params +fi -# Final notice -# ------------ +# Clean up and final notice +# ------------------------- # -# The configuration is now complete, we can inform the user on the next -# step(s) to take. -if [ x$maneage_group_name = x ]; then - buildcommand="./project make -j8" -else - buildcommand="./project make --group=$maneage_group_name -j8" -fi -cat <<EOF +# The configuration is now complete. We just need to delete the temporary +# build directory and inform the user (if '--quiet' wasn't called) on the +# next step(s). +if [ -d $tmpblddir ]; then empty_build_tmp; fi +if [ $quiet = 0 ]; then + + # Suggest the command to use. + if [ x$maneage_group_name = x ]; then + buildcommand="./project make -j8" + else + buildcommand="./project make --group=$maneage_group_name -j8" + fi + + # Print the message. + cat <<EOF ---------------- The project and its environment are configured with no errors. @@ -1976,3 +2130,10 @@ Please run the following command to start the project. $buildcommand EOF +fi + + +# Total time +if [ $check_elapsed = 1 ]; then + echo $chel_dsum | awk '{printf "Total: %-6.2f [millisec]\n", $1}' +fi diff --git a/reproduce/software/shell/docker-README.md b/reproduce/software/shell/docker-README.md new file mode 100644 index 0000000..dd46f4f --- /dev/null +++ b/reproduce/software/shell/docker-README.md @@ -0,0 +1,198 @@ +# Maneage'd projects in Docker + +Copyright (C) 2021-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ +See the end of the file for license conditions. + +For an introduction on containers, see the "Building in containers" section +of the `README.md` file within the top-level directory of this +project. Here, we focus on Docker with a simple checklist on how to use the +`docker.sh` script that we have already prepared in this directory for easy +usage in a Maneage'd project. + + + + + +## Building your Maneage'd project in Docker + +Through the steps below, you will create a Docker image that will only +contain the software environment and keep the project source and built +analysis files (data and PDF) on your host operating system. This enables +you to keep the size of the image to a minimum (only containing the built +software environment) to easily move it from one computer to another. + + 0. Add your user to the `docker` group: `usermod -aG docker + USERNAME`. This is only necessary once on an operating system. + + 1. Start the Docker daemon (root permissions required). If the operating + system uses systemd you can use the command below. If you want the + Docker daemon to be available after a reboot also (so you don't have to + restart it after turning off your computer), run this command again but + replacing `start` with `enable` (this is not recommended if you don't + regularly use Docker: it will slow the boot time of your OS). + + ```shell + systemctl start docker + ``` + + 2. At the start of the `docker.sh` script in this directory, you can see + the necessary command to run. Just activate `--build-only` on the first + run so it doesn't go onto doing the analysis (that can be done at any + later time). Set the respective directory(s) based on your filesystem + (the software directory is optional). You can put the command in a + `run.sh` at the top of the project (it is already in `.gitignore`: + because it contains local directories). + + 3. After the setup is complete, remove the `--build-only` and run the + command below to confirm that `maneage-base` (the OS of the container) + and `maneaged` (your project's full Maneage'd environment) images are + available. If you want different names for these images, add the + `--project-name` and `--base-name` options to the `docker.sh` call. You + can see the list of available images with this command: + + ```shell + docker image list + ``` + + + + + +## Script usage tips + +The `docker.sh` script introduced above has many options allowing certain +customizations that you can see when running it with the `--help` +option. The tips below are some of the more useful scenarios that we have +encountered so far. + +### Docker image in a single file + +In case you want to store the image as a single file as backup or to move +to another computer. For such cases, run the `docker.sh` script with the +`--image-file` option (for example `--image-file=myproj.tar.gz`). After +moving the file to the other system, run `docker.sh` with the same option. + +When the given file to `docker.sh` already exists, it will only be used for +loading the environment. When it doesn't exist, the script will save the +image into it. + + + + + +## Docker usage tips + +Below are some useful Docker usage scenarios that have proved to be +relevant for us in Maneage'd projects. + +### Saving and loading an image as a file + +Docker keeps its images in hard to access (by humans) location on the +operating system. Very much like Git, but with much less elegance: the +place is shared by all users and projects of the system. So they are not +easy to archive for usage on another system at a low-level. But it does +have an interface (`docker save`) to copy all the relevant files within an +image into a tar ball that you can archive externally. There is also a +separate interface to load the tarball back into docker (`docker load`). + +Both of these have been implemented as the `--image-file` option of the +`docker.sh` script. If you want to save your Maneage'd image into an image, +simply give the tarball name to this option. Alternatively, if you already +have a tarball and want to load it into Docker, give it to this option once +(until you "clean up", as explained below). In fact, docker images take a +lot of space and it is better to "clean up" regularly. And the only way you +can clean up safely is through saving your needed images as a file. + +### Cleaning up + +Docker has stored many large files in your operating system that can drain +valuable storage space. The storage of the cached files are usually orders +of magnitudes larger than what you see in `docker image list`! So after +doing your work, it is best to clean up all those files. If you feel you +may need the image later, you can save it in a single file as mentioned +above and delete all the un-necessary cached files. Afterwards, when you +load the image, only that image will be present with nothing extra. + +The easiest and most powerful way to clean up everything in Docker is the +two commands below. The first will close all open containers. The second +will remove all stopped containers, all networks not used by at least one +container, all images without at least one container associated to them, +and all build cache. + +```shell +docker ps -a -q | xargs docker rm +docker system prune -a +``` + +If you only want to delete the existing used images, run the command +below. But be careful that the cache is the largest storage consumer! So +the command above is the solution if your OS's root partition is close to +getting filled. + +```shell +docker images -a -q | xargs docker rmi -f +``` + + +### Preserving the state of an open container + +All interactive changes in a container will be deleted as soon as you exit +it. This is a very good feature of Docker in general! If you want to make +persistent changes, you should do it in the project's plain-text source and +commit them into your project's online Git repository. But in certain +situations, it is necessary to preserve the state of an interactive +container. To do this, you need to `commit` the container (and thus save it +as a Docker "image"). To do this, while the container is still running, +open another terminal and run these commands: + +```shell +# These two commands should be done in another terminal +docker container list + +# Get the 'XXXXXXX' of your desired container from the first column above. +# Give the new image a name by replacing 'NEW-IMAGE-NAME'. +docker commit XXXXXXX NEW-IMAGE-NAME +``` + + +### Interactive tests on built container + +If you later want to start a container with the built image and enter it in +interactive mode (for example for temporary tests), run the following +command. Just replace `NAME` with the same name you specified when building +the project. You can always exit the container with the `exit` command +(note that all your changes will be discarded once you exit, see below if +you want to preserve your changes after you exit). + +```shell +docker run -it NAME +``` + + +### Copying files from the Docker image to host operating system + +Except for the mounted directories, the Docker environment's file system is +indepenent of your host operating system. One easy way to copy files to and +from an open container is to use the `docker cp` command (very similar to +the shell's `cp` command). + +```shell +docker cp CONTAINER:/file/path/within/container /host/path/target +``` + + + +## Copyright information + +This file is free software: you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) +any later version. + +This file is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along +with this file. If not, see <https://www.gnu.org/licenses/>. diff --git a/reproduce/software/shell/docker.sh b/reproduce/software/shell/docker.sh new file mode 100755 index 0000000..5cd8f11 --- /dev/null +++ b/reproduce/software/shell/docker.sh @@ -0,0 +1,520 @@ +#!/usr/bin/env bash +# +# Create a Docker container from an existing image of the built software +# environment, but with the source, data and build (analysis) directories +# directly within the host file system. This script is assumed to be run in +# the top project source directory (that has 'README.md' and +# 'paper.tex'). If not, use the '--source-dir' option to specify where the +# Maneage'd project source is located. +# +# Usage: +# +# - When you are at the top Maneage'd project directory, run this script +# like the example below. Just set the build directory location on your +# system. See the items below for optional values to optimize the +# process (avoid downloading for exmaple). +# +# ./reproduce/software/shell/docker.sh --shm-size=20gb \ +# --build-dir=/PATH/TO/BUILD/DIRECTORY +# +# - Non-mandatory options: +# +# - If you already have the input data that is necessary for your +# project, use the '--input-dir' option to specify its location +# on your host file system. Otherwise the necessary analysis +# files will be downloaded directly into the build +# directory. Note that this is only necessary when '--build-only' +# is not given. +# +# - If you already have the necessary software tarballs that are +# necessary for your project, use the '--software-dir' option to +# specify its location on your host file system only when +# building the container. No problem if you don't have them, they +# will be downloaded during the configuration phase. +# +# - To avoid having to set them every time you want to start the +# apptainer environment, you can put this command (with the proper +# directories) into a 'run.sh' script in the top Maneage'd project +# source directory and simply execute that. The special name 'run.sh' +# is in Maneage's '.gitignore', so it will not be included in your +# git history by mistake. +# +# Known problems: +# +# - As of 2025-04-06 the log file containing the output of the 'docker +# build' command that configures the Maneage'd project does not keep +# all the output (which gets clipped by Docker). with a "[output +# clipped, log limit 2MiB reached]" message. We need to find a way to +# fix this (so nothing gets clipped: useful for debugging). +# +# Copyright (C) 2021-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# +# This script is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# This script is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this script. If not, see <http://www.gnu.org/licenses/>. + + + + + +# Script settings +# --------------- +# Stop the script if there are any errors. +set -e + + + + + +# Default option values +jobs=0 +quiet=0 +source_dir= +build_only= +image_file="" +shm_size=20gb +scriptname="$0" +project_shell=0 +container_shell=0 +project_name=maneaged +base_name=maneage-base +base_os=debian:stable-slim + +print_help() { + # Print the output. + cat <<EOF +Usage: $scriptname [OPTIONS] + +Top-level script to build and run a Maneage'd project within Docker. + + Host OS directories (to be mounted in the container): + -b, --build-dir=STR Dir. to build in (only analysis in host). + -i, --input-dir=STR Dir. of input datasets (optional). + -s, --software-dir=STR Directory of necessary software tarballs. + --source-dir=STR Directory of source code (default: 'pwd -P'). + + Docker images + --base-os=STR Base OS name (default: '$base_os'). + --base-name=STR Base OS docker image (default: $base_name). + --project-name=STR Project's docker image (default: $project_name). + --image-file=STR [Docker only] Load (if given file exists), or + save (if given file does not exist), the image. + For saving, the given name has to have an + '.tar.gz' suffix. + + Interactive shell + --project-shell Open the project's shell within the container. + --container-shell Open the container shell. + + Operating mode: + -q, --quiet Do not print informative statements. + -?, --help Give this help list. + --shm-size=STR Passed to 'docker build' (default: $shm_size). + -j, --jobs=INT Number of threads to use in each phase. + --build-only Just build the container, don't run it. + +Mandatory or optional arguments to long options are also mandatory or +optional for any corresponding short options. + +Maneage URL: https://maneage.org + +Report bugs to mohammad@akhlaghi.org +EOF +} + +on_off_option_error() { + if [ "x$2" = x ]; then + echo "$scriptname: '$1' doesn't take any values" + else + echo "$scriptname: '$1' (or '$2') doesn't take any values" + fi + exit 1 +} + +check_v() { + if [ x"$2" = x ]; then + printf "$scriptname: option '$1' requires an argument. " + printf "Try '$scriptname --help' for more information\n" + exit 1; + fi +} + +while [ $# -gt 0 ] +do + case $1 in + + # OS directories + -b|--build-dir) build_dir="$2"; check_v "$1" "$build_dir"; shift;shift;; + -b=*|--build-dir=*) build_dir="${1#*=}"; check_v "$1" "$build_dir"; shift;; + -b*) build_dir=$(echo "$1" | sed -e's/-b//'); check_v "$1" "$build_dir"; shift;; + -i|--input-dir) input_dir="$2"; check_v "$1" "$input_dir"; shift;shift;; + -i=*|--input-dir=*) input_dir="${1#*=}"; check_v "$1" "$input_dir"; shift;; + -i*) input_dir=$(echo "$1" | sed -e's/-i//'); check_v "$1" "$input_dir"; shift;; + -s|--software-dir) software_dir="$2"; check_v "$1" "$software_dir"; shift;shift;; + -s=*|--software-dir=*) software_dir="${1#*=}"; check_v "$1" "$software_dir"; shift;; + -s*) software_dir=$(echo "$1" | sed -e's/-s//'); check_v "$1" "$software_dir"; shift;; + --source-dir) source_dir="$2"; check_v "$1" "$source_dir"; shift;shift;; + --source-dir=*) source_dir="${1#*=}"; check_v "$1" "$source_dir"; shift;; + + # Container options. + --base-name) base_name="$2"; check_v "$1" "$base_name"; shift;shift;; + --base-name=*) base_name="${1#*=}"; check_v "$1" "$base_name"; shift;; + --project-name) project_name="$2"; check_v "$1" "$project_name"; shift;shift;; + --project-name=*) project_name="${1#*=}"; check_v "$1" "$project_name"; shift;; + + # Interactive shell. + --project-shell) project_shell=1; shift;; + --project_shell=*) on_off_option_error --project-shell;; + --container-shell) container_shell=1; shift;; + --container_shell=*) on_off_option_error --container-shell;; + + # Operating mode + -q|--quiet) quiet=1; shift;; + -q*|--quiet=*) on_off_option_error --quiet;; + -j|--jobs) jobs="$2"; check_v "$1" "$jobs"; shift;shift;; + -j=*|--jobs=*) jobs="${1#*=}"; check_v "$1" "$jobs"; shift;; + -j*) jobs=$(echo "$1" | sed -e's/-j//'); check_v "$1" "$jobs"; shift;; + --build-only) build_only=1; shift;; + --build-only=*) on_off_option_error --build-only;; + --shm-size) shm_size="$2"; check_v "$1" "$shm_size"; shift;shift;; + --shm-size=*) shm_size="${1#*=}"; check_v "$1" "$shm_size"; shift;; + -'?'|--help) print_help; exit 0;; + -'?'*|--help=*) on_off_option_error --help -?;; + + # Output file + --image-file) image_file="$2"; check_v "$1" "$image_file"; shift;shift;; + --image-file=*) image_file="${1#*=}"; check_v "$1" "$image_file"; shift;; + + # Unrecognized option: + -*) echo "$scriptname: unknown option '$1'"; exit 1;; + esac +done + + + + + +# Sanity checks +# ------------- +# +# Make sure that the build directory is given and that it exists. +if [ x$build_dir = x ]; then + printf "$scriptname: '--build-dir' not provided, this is the location " + printf "that all built analysis files will be kept on the host OS\n" + exit 1; +else + if ! [ -d $build_dir ]; then + printf "$scriptname: '$build_dir' (value to '--build-dir') doesn't " + printf "exist\n"; exit 1; + fi +fi + +# The temporary directory to place the Dockerfile. +tmp_dir="$build_dir"/temporary-docker-container-dir + + + + +# Directory preparations +# ---------------------- +# +# If the host operating system has '/dev/shm', then give Docker access +# to it also for improved speed in some scenarios (like configuration). +if [ -d /dev/shm ]; then shm_mnt="-v /dev/shm:/dev/shm"; +else shm_mnt=""; fi + +# If the following directories do not exist within the build directory, +# create them to make sure the '--mount' commands always work and +# that any file. Ideally, the 'input' directory should not be under the 'build' +# directory, but if the user hasn't given it then they don't care about +# potentially deleting it later (Maneage will download the inputs), so put +# it in the build directory. +analysis_dir="$build_dir"/analysis +if ! [ -d $analysis_dir ]; then mkdir $analysis_dir; fi + +# If no '--source-dir' was given, set it to the output of 'pwd -P' (to get +# the path without potential symbolic links) in the running directory. +if [ x"$source_dir" = x ]; then source_dir=$(pwd -P); fi + +# Only when an an input directory is given, we need the respective 'mount' +# option for the 'docker run' command. +input_dir_mnt="" +if ! [ x"$input_dir" = x ]; then + input_dir_mnt="-v $input_dir:/home/maneager/input" +fi + +# Number of threads to build software (taken from 'configure.sh'). +if [ x"$jobs" = x0 ]; then + if type nproc > /dev/null 2> /dev/null; then + numthreads=$(nproc --all); + else + numthreads=$(sysctl -a | awk '/^hw\.ncpu/{print $2}') + if [ x"$numthreads" = x ]; then numthreads=1; fi + fi +else + numthreads=$jobs +fi + +# Since the container is read-only and is run with the '--contain' option +# (which makes an empty '/tmp'), we need to make a dedicated directory for +# the container to be able to write to. This is necessary because some +# software (Biber in particular on the default branch or Ghostscript) need +# to write there! See https://github.com/plk/biber/issues/494. We'll keep +# the directory on the host OS within the build directory, but as a hidden +# file (since it is not necessary in other types of build and ultimately +# only contains temporary files of programs that need it). +toptmp=$build_dir/.docker-tmp-$(whoami) +if ! [ -d $toptmp ]; then mkdir $toptmp; fi +chmod -R +w $toptmp/ # Some software remove writing flags on /tmp files. +if ! [ x"$( ls -A $toptmp )" = x ]; then rm -r "$toptmp"/*; fi + +# [DOCKER-ONLY] Make sure the user is a member of the 'docker' group. This +# is needed only for Linux, given that other systems uses other strategies. +# (See: https://stackoverflow.com/a/70385997) +kernelname=$(uname -s) +if [ x$kernelname = xLinux ]; then + glist=$(groups $(whoami) | awk '/docker/') + if [ x"$glist" = x ]; then + printf "$scriptname: you are not a member of the 'docker' group " + printf "You can run the following command as root to fix this: " + printf "'usermod -aG docker $(whoami)'\n" + exit 1 + fi +fi + +# [DOCKER-ONLY] Function to check the temporary directory for building the +# base operating system docker image. It is necessary that this directory +# be empty because Docker will inherit the sub-directories of the directory +# that the Dockerfile is located in. +tmp_dir_check () { + if [ -d $tmp_dir ]; then + printf "$scriptname: '$tmp_dir' already exists, please " + printf "delete it and re-run this script. This is a temporary " + printf "directory only necessary when building a Docker image " + printf "and gets deleted automatically after a successful " + printf "build. The fact that it remains hints at a problem " + printf "in a previous attempt to build a Docker image\n" + exit 1 + else + mkdir $tmp_dir + fi +} + + + + + +# Base operating system +# --------------------- +# +# If the base image does not exist, then create it. If it does, inform the +# user that it will be used. +if docker image list | grep $base_name &> /dev/null; then + if [ $quiet = 0 ]; then + printf "$scriptname: info: base OS docker image ('$base_name') " + printf "already exists and will be used. If you want to build a " + printf "new base OS image, give a new name to '--base-name'. " + printf "To remove this message run with '--quiet'\n" + fi +else + + # In case an image file is given, load the environment from that (no + # need to build the environment from scratch). + if ! [ x"$image_file" = x ] && [ -f "$image_file" ]; then + docker load --input $image_file + else + + # Build the temporary directory. + tmp_dir_check + + # Build the Dockerfile. + uid=$(id -u) + cat <<EOF > $tmp_dir/Dockerfile +FROM $base_os +RUN useradd -ms /bin/sh --uid $uid maneager; \\ + printf '123\n123' | passwd maneager; \\ + printf '456\n456' | passwd root +RUN apt update; apt install -y gcc g++ wget; echo 'export PS1="[\[\033[01;31m\]\u@\h \W\[\033[32m\]\[\033[00m\]]# "' >> ~/.bashrc +USER maneager +WORKDIR /home/maneager +RUN mkdir build; mkdir build/analysis; echo 'export PS1="[\[\033[01;35m\]\u@\h \W\[\033[32m\]\[\033[00m\]]$ "' >> ~/.bashrc +EOF + + # Build the base-OS container and delete the temporary directory. + curdir="$(pwd)" + cd $tmp_dir + docker build ./ \ + -t $base_name \ + --shm-size=$shm_size + cd "$curdir" + rm -rf $tmp_dir + fi +fi + + + + + +# Maneage software configuration +# ------------------------------ +# +# Having the base operating system in place, we can now construct the +# project's docker file. +intbuild=/home/maneager/build +if docker image list | grep $project_name &> /dev/null; then + if [ $quiet = 0 ]; then + printf "$scriptname: info: project's image ('$project_name') " + printf "already exists and will be used. If you want to build a " + printf "new project image, give a new name to '--project-name'. " + printf "To remove this message run with '--quiet'\n" + fi +else + + # Build the temporary directory. + tmp_dir_check + df=$tmp_dir/Dockerfile + + # The only way to mount a directory inside the Docker build environment + # is the 'RUN --mount' command. But Docker doesn't recognize things + # like symbolic links. So we need to copy the project's source under + # this temporary directory. + sdir=source + mkdir $tmp_dir/$sdir + dsr=/home/maneager/source-raw + cp -r $source_dir/* $source_dir/.git $tmp_dir/$sdir + + # Start constructing the Dockerfile. + # + # Note on the printf's '\x5C\n' part: this will print out as a + # backslash at the end of the line to allow easy human readability of + # the Dockerfile (necessary for debugging!). + echo "FROM $base_name" > $df + printf "RUN --mount=type=bind,source=$sdir,target=$dsr \x5C\n" >> $df + + # If a software directory was given, copy it and add its line. + tsdir=tarballs-software + dts=/home/maneager/tarballs-software + if ! [ x"$software_dir" = x ]; then + + # Make the directory to host the software and copy the contents + # that the user gave there. + mkdir $tmp_dir/$tsdir + cp -r "$software_dir"/* $tmp_dir/$tsdir/ + printf " --mount=type=bind,source=$tsdir,target=$dts \x5C\n" \ + >> $df + fi + + # Construct the rest of the 'RUN' command. + printf " cp -r $dsr /home/maneager/source; \x5C\n" >> $df + printf " cd /home/maneager/source; \x5C\n" >> $df + printf " ./project configure \x5C\n" >> $df + printf " --no-pause \x5C\n" >> $df + printf " --jobs=$jobs \x5C\n" >> $df + printf " --build-dir=$intbuild \x5C\n" >> $df + printf " --input-dir=/home/maneager/input \x5C\n" >> $df + printf " --software-dir=$dts; \x5C\n" >> $df + + # We are deleting the contents of the '.build/software/tarballs' + # directory (tarballs of the sources of the software) because they are + # not relevant to the execution of the project and will consume space + # within the container. + printf " rm -rf .build/software/tarballs/*; \x5C\n" >> $df + + # We are deleting the source directory becaues later (at 'docker run' + # time), the 'source' will be mounted directly from the host operating + # system. + printf " cd /home/maneager; \x5C\n" >> $df + printf " rm -rf source\n" >> $df + + # Build the Maneage container and delete the temporary directory. + cd $tmp_dir + docker build ./ -t $project_name \ + --shm-size=$shm_size \ + --no-cache \ + 2>&1 | tee build.log + cd .. + rm -rf $tmp_dir +fi + +# If the user wants to save the container (into a file that does not +# exist), do it here. If the file exists, it will only be used for creating +# the container in the previous stages. +if ! [ x"$image_file" = x ] && ! [ -f "$image_file" ]; then + + # Save the image into a tarball + tarname=$(echo $image_file | sed -e's|.gz$||') + if [ $quiet = 0 ]; then + printf "$scriptname: info: saving docker image to '$tarname'\n" + fi + docker save -o $tarname $project_name + + # Compress the saved image + if [ $quiet = 0 ]; then + printf "$scriptname: info: compressing to '$image_file' (can " + printf "take +10 minutes, but volume decreases by more than " + printf "half!)\n" + fi + gzip --best $tarname +fi + +# If the user just wanted to build the base operating system, abort the +# script here. +if ! [ x"$build_only" = x ]; then + if [ $quiet = 0 ]; then + printf "$scriptname: info: Maneaged project has been configured " + printf "successfully in the '$project_name' image\n" + fi + exit 0 +fi + + + + + +# Run the analysis within the Maneage'd container +# ----------------------------------------------- +# +# The startup command of the container is managed though the 'shellopt' +# variable that starts here. +shellopt="" +sobase="/bin/bash -c 'cd source; " +sobase="$sobase ./project configure --build-dir=$intbuild " +sobase="$sobase --existing-conf --no-pause --offline --quiet && " +sobase="$sobase ./project MODE --build-dir=$intbuild" +if [ $container_shell = 1 ] || [ $project_shell = 1 ]; then + + # The interactive flag is necessary for both these scenarios. + interactiveopt="-it" + + # With '--project-shell' we need 'shellopt', the MODE just needs to be + # set to 'shell'. + if [ $project_shell = 1 ]; then + shellopt="$(echo $sobase | sed -e's|MODE|shell|');'" + fi + +# No interactive shell requested, just run the project. +else + interactiveopt="" + shellopt="$(echo $sobase | sed -e's|MODE|make|') --jobs=$jobs;'" +fi + +# Execute Docker. The 'eval' is because the 'shellopt' variable contains a +# single-quote that the shell should "evaluate". +eval docker run --read-only \ + -v "$analysis_dir":/home/maneager/build/analysis \ + -v "$source_dir":/home/maneager/source \ + -v $toptmp:/tmp \ + $input_dir_mnt \ + $shm_mnt \ + $interactiveopt \ + $project_name \ + $shellopt diff --git a/reproduce/software/shell/git-post-checkout b/reproduce/software/shell/git-post-checkout index 7b521a3..7bf747a 100755 --- a/reproduce/software/shell/git-post-checkout +++ b/reproduce/software/shell/git-post-checkout @@ -4,7 +4,7 @@ # controlled files (with each commit) using the 'metastore' program. # # Copyright (C) 2016 Przemyslaw Pawelczyk <przemoc@gmail.com> -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This script is taken from the 'examples/hooks/pre-commit' file of the # 'metastore' package (installed within the project, with an MIT license diff --git a/reproduce/software/shell/git-pre-commit b/reproduce/software/shell/git-pre-commit index 7b98ad0..e570346 100755 --- a/reproduce/software/shell/git-pre-commit +++ b/reproduce/software/shell/git-pre-commit @@ -4,7 +4,7 @@ # controlled files (with each commit) using the 'metastore' program. # # Copyright (C) 2016 Przemyslaw Pawelczyk <przemoc@gmail.com> -# Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # WARNING: # diff --git a/reproduce/software/shell/pre-make-build.sh b/reproduce/software/shell/pre-make-build.sh index 93d3266..3163f74 100755 --- a/reproduce/software/shell/pre-make-build.sh +++ b/reproduce/software/shell/pre-make-build.sh @@ -2,7 +2,7 @@ # # Very basic tools necessary to start Maneage's default building. # -# Copyright (C) 2020-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2020-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -120,31 +120,29 @@ download_tarball() { "$bservers" fi + # Make sure this is the correct tarball. if type sha512sum > /dev/null 2> /dev/null; then - checksum=$(sha512sum "$ucname" | awk '{print $1}') + + # On macOS, sha512sum wraps the hash in two lines. The 'tr -d '\n'' + # part joins them into one before awk extracts the first field. On + # Linux (GNU coreutils) the output is already a single line, so the + # 'tr' command has no effect. + checksum=$(sha512sum "$ucname" | tr -d '\n' | awk '{print $1}') expectedchecksum=$(awk '/^'$progname'-checksum/{print $3}' \ "$checksumsfile") - if [ x$checksum = x$expectedchecksum ]; then mv "$ucname" "$maneagetar" + if [ x$checksum = x$expectedchecksum ]; then + mv "$ucname" "$maneagetar" else - echo "ERROR: Non-matching checksum: $tarball" - echo "Checksum should be: $expectedchecksum" - echo "Checksum is: $checksum" + echo "ERROR: Non-matching checksum in $tarball" + echo " Checksum should be: $expectedchecksum" + echo " Checksum is: $checksum" + rm -f "$ucname" exit 1 fi; else mv "$ucname" "$maneagetar" fi fi - - # If the tarball is newer than the (possibly existing) program (the - # version has changed), then delete the program. When the LaTeX name is - # not given here, the software is re-built later (close to the end of - # 'basic.mk') and the name is properly placed there. - if [ -f "$ibidir/$progname" ]; then - if [ "$maneagetar" -nt "$ibidir/$progname" ]; then - rm "$ibidir/$progname" - fi - fi } @@ -159,6 +157,9 @@ build_program() { # Options configoptions=$1 + # Inform the user. + echo; echo "Pre-make building of $progname"; echo + # Go into the temporary building directory. cd "$tmpblddir" unpackdir="$progname"-"$version" @@ -177,13 +178,14 @@ build_program() { fi # Unpack the tarball and go into it. - tar xf "$intar" + tar xf "$intar" --no-same-owner --no-same-permissions if [ x$intarrm = x1 ]; then rm "$intar"; fi cd "$unpackdir" # build the project, either with Make and either without it. if [ x$progname = xlzip ]; then - ./configure --build --check --installdir="$instdir/bin" $configoptions + ./configure --build --check --installdir="$instdir/bin" \ + $configoptions else # All others accept the configure script. ./configure --prefix="$instdir" $configoptions @@ -196,7 +198,10 @@ build_program() { case $on_mac_os in yes) sed -e's/\%1u/\%d/' src/flock.c > src/flock-new.c;; no) sed -e's/\%1u/\%ld/' src/flock.c > src/flock-new.c;; - *) echo "pre-make-build.sh: '$on_mac_os' unrecognized value for on_mac_os";; + *) + printf "pre-make-build.sh: '$on_mac_os' " + printf "unrecognized value for on_mac_os" + exit 1;; esac mv src/flock-new.c src/flock.c fi @@ -218,9 +223,9 @@ build_program() { cd "$topdir" rm -rf "$tmpblddir/$unpackdir" if [ x"$progname_tex" = x ]; then - echo "" > "$ibidir/$progname" + echo "" > "$texfile" else - echo "$progname_tex $version" > "$ibidir/$progname" + echo "$progname_tex $version" > "$texfile" fi fi } @@ -238,12 +243,12 @@ build_program() { # (without compression it is just ~400Kb). So we use its '.tar' file and # won't rely on the host's compression tools at all. progname="lzip" -progname_tex="" # Lzip re-built after GCC (empty string to avoid repetition) +progname_tex="" # Lzip is re-built after GCC (empty to avoid repetition) url=$(awk '/^'$progname'-url/{print $3}' $urlfile) version=$(awk '/^'$progname'-version/{print $3}' "$versionsfile") tarball=$progname-$version.tar -download_tarball -build_program +texfile="$ibidir/$progname-$version-pre-make" +if ! [ -f $texfile ]; then download_tarball; build_program; fi @@ -268,8 +273,11 @@ progname_tex="" # Make re-built after GCC (empty string to avoid repetition) url=$(awk '/^'$progname'-url/{print $3}' $urlfile) version=$(awk '/^'$progname'-version/{print $3}' $versionsfile) tarball=$progname-$version.tar.lz -download_tarball -build_program "--disable-dependency-tracking --without-guile" +texfile="$ibidir/$progname-$version-pre-make" +if ! [ -f $texfile ]; then + download_tarball + build_program "--disable-dependency-tracking --without-guile" +fi @@ -286,13 +294,11 @@ progname_tex="Dash" url=$(awk '/^'$progname'-url/{print $3}' $urlfile) version=$(awk '/^'$progname'-version/{print $3}' $versionsfile) tarball=$progname-$version.tar.lz -download_tarball -build_program +texfile="$ibidir/$progname-$version" +if ! [ -f $texfile ]; then download_tarball; build_program; fi # If the 'sh' symbolic link isn't set yet, set it to point to Dash. -if [ -f $bindir/sh ]; then just_a_place_holder=1 -else ln -sf $bindir/dash $bindir/sh; -fi +if ! [ -f $bindir/sh ]; then ln -sf $bindir/dash $bindir/sh; fi @@ -315,12 +321,5 @@ progname_tex="Discoteq flock" url=$(awk '/^'$progname'-url/{print $3}' $urlfile) version=$(awk '/^'$progname'-version/{print $3}' $versionsfile) tarball=$progname-$version.tar.lz -download_tarball -build_program - - - - - -# Finish this script successfully -exit 0 +texfile="$ibidir/$progname-$version" +if ! [ -f $texfile ]; then download_tarball; build_program; fi diff --git a/reproduce/software/shell/prep-source.sh b/reproduce/software/shell/prep-source.sh index dcdc472..7f49b5c 100755 --- a/reproduce/software/shell/prep-source.sh +++ b/reproduce/software/shell/prep-source.sh @@ -7,7 +7,7 @@ # directory that it is run in ): # ./prep-source.sh /FULL/ADDRESS/TO/DESIRED/BIN # -# Copyright (C) 2024-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2024-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This script is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the @@ -32,6 +32,15 @@ set -e + +# Set the locale to C to make sure local language settings do not +# interefere (especially with the SED command). +export LC_ALL=C + + + + + # Read the first argument. bindir="$1" if [ x"$bindir" = x ]; then @@ -47,17 +56,18 @@ fi -# Find all the files that contain the '/bin/sh' string and correct them to -# Maneage's own Bash. We are using 'while read' to read the file names line -# by line. This is necessary to account file names that include the 'SPACE' -# character (happens in CMake for example!). +# Find all the files that contain the '/bin/sh' or '/bin/bash' strings and +# correct them to Maneage's own shell (Dash or Bash). We are using 'while +# read' to read the file names line by line. This is necessary to account +# file names that include the 'SPACE' character (happens in CMake for +# example!). # # Note that dates are important in the source directory (files depend on # each other), so we should read the original date and after making. We are # also not using GNU SED's '-i' ('--in-place') option because the host OS # may not have GNU SED. # -# Actual situation which prompted the addition of this step: a Maneage'd +# The situation which prompted the addition of this step: a Maneage'd # project (with GNU Bash 5.1.8 and Readline 8.1.1) was being built on a # system where '/bin/sh' was GNU Bash 5.2.26 and had Readline 8.2.010. The # newer version of Bash needed the newer Readline library function(s) that @@ -74,18 +84,35 @@ fi # hard-coded in the source code of almost all programs (their build # scripts); and in special programs like GNU Make, GNU M4 or CMake it is # actually hardcoded in the source code (not just build scripts). -if [ -f "$bindir/bash" ]; then shpath="$bindir"/bash +bashpath="$bindir"/bash +if [ -f "$bindir/bash" ]; then shpath="$bashpath" else shpath="$bindir"/dash fi -grep -I -r -e'/bin/sh' $(pwd)/* \ + +# On MacOS the syntax for 'stat' is a bit different, so if we are using the +# system one on mac we need a specific syntax. The one installed by maneage +# instead uses the ordinary Linux syntax. +if [ -f "$bindir/stat" ] || [ x"$on_mac_os" = xno ]; then + format="--format %a" +else + format="-f %OLp" +fi + +# On MacOS 'touch' wants the time expressed according to ISO8601 with a +# precision up to the seconds. We then use 'sed' to remove the information +# regarding the timezone, as the format is not accepted by 'touch'. +# LCTYPE and LANG are also required on macos systems by sed. +grep -I -r -e'/bin/sh' -e'/bin/bash' $(pwd)/* \ | sed -e's|:|\t|' \ | awk 'BEGIN{FS="\t"}{print $1}' \ | sort \ | uniq \ | while read filename; do \ tmp="$filename".tmp; \ - origtime="$(date -R -r "$filename")"; \ - origperm=$(stat -c '%a' "$filename"); \ + origtimex="$(date -Iseconds -r "$filename")"; \ + origtime=$(echo $origtimex | sed 's/.\{6\}$//'); \ + origperm=$(stat $format "$filename"); \ + LC_CTYPE=C; LANG=C; \ sed -e's|/bin/sh|'"$shpath"'|g' "$filename" > "$tmp"; \ mv "$tmp" "$filename"; \ chmod $origperm "$filename"; \ diff --git a/reproduce/software/shell/run-parts.in b/reproduce/software/shell/run-parts.in index 053f5f3..21c347f 100755 --- a/reproduce/software/shell/run-parts.in +++ b/reproduce/software/shell/run-parts.in @@ -10,8 +10,8 @@ # However, it didn't have a copyright statement. So one is being added # here. # -# Copyright (C) 2025 Authors mentioned above. -# Copyright (C) 2025-2024 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2026 Authors mentioned above. +# Copyright (C) 2026-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/reproduce/software/shell/tarball-prepare.sh b/reproduce/software/shell/tarball-prepare.sh index 8e30931..ed1cfd0 100755 --- a/reproduce/software/shell/tarball-prepare.sh +++ b/reproduce/software/shell/tarball-prepare.sh @@ -15,9 +15,9 @@ # # Discussion: https://savannah.nongnu.org/task/?15699 # -# Copyright (C) 2022-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2022-2025 Pedram Ashofteh Ardakani <pedramardakani@pm.me> -# Copyright (C) 2025-2025 Giacomo Lorenzetti <glorenzetti@cefca.es> +# Copyright (C) 2022-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2022-2026 Pedram Ashofteh Ardakani <pedramardakani@pm.me> +# Copyright (C) 2025-2026 Giacomo Lorenzetti <glorenzetti@cefca.es> # # This script is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the @@ -51,6 +51,9 @@ basedir=$PWD scriptname=$0 + + + # The --help output print_help() { cat <<EOF diff --git a/tex/README.md b/tex/README.md index f100c76..e94325b 100644 --- a/tex/README.md +++ b/tex/README.md @@ -1,7 +1,7 @@ Directory containing LaTeX-related files ---------------------------------------- -Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ +Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ See the end of the file for license conditions. This directory contains directories to various components the LaTeX part of diff --git a/tex/src/delete-me-image-histogram.tex b/tex/src/delete-me-image-histogram.tex index 6201c31..00ef2e1 100644 --- a/tex/src/delete-me-image-histogram.tex +++ b/tex/src/delete-me-image-histogram.tex @@ -1,6 +1,6 @@ %% Plot the demonstration image and its histogram. % -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> % %% This file is free software: you can redistribute it and/or modify it %% under the terms of the GNU General Public License as published by the @@ -42,7 +42,7 @@ ] \addplot [const plot mark mid, fill=red] table [x index=0, y index=1] - {tex/build/to-publish/wfpc2-histogram.txt} + {tex/build/figures-data/wfpc2-histogram.txt} \closedcycle; \end{axis} diff --git a/tex/src/delete-me-squared.tex b/tex/src/delete-me-squared.tex index 51e5f5b..d43b5c8 100644 --- a/tex/src/delete-me-squared.tex +++ b/tex/src/delete-me-squared.tex @@ -1,6 +1,6 @@ %% PGFPlots code to plot a random set of numbers as demo %% -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> % %% This file is free software: you can redistribute it and/or modify it %% under the terms of the GNU General Public License as published by the @@ -26,7 +26,7 @@ %% A particular plot. \addplot+[scatter, only marks] - table {tex/build/to-publish/squared.txt}; + table {tex/build/figures-data/squared.txt}; \end{axis} \end{tikzpicture} diff --git a/tex/src/preamble-biblatex.tex b/tex/src/preamble-biblatex.tex index e80d7d8..2b7b049 100644 --- a/tex/src/preamble-biblatex.tex +++ b/tex/src/preamble-biblatex.tex @@ -12,7 +12,7 @@ %% helps in clearly identifying the files and avoiding places that %% complain about '.bib' files. % -%% Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> % %% This file is free software: you can redistribute it and/or modify it %% under the terms of the GNU General Public License as published by the diff --git a/tex/src/preamble-maneage-default-style.tex b/tex/src/preamble-maneage-default-style.tex index b7aa811..adb3eb9 100644 --- a/tex/src/preamble-maneage-default-style.tex +++ b/tex/src/preamble-maneage-default-style.tex @@ -6,7 +6,7 @@ %% style defined here as a preamble. To ignore it, uncomment or delete the %% respective line in 'paper.tex'. % -%% Copyright (C) 2019-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2019-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> % %% This file is free software: you can redistribute it and/or modify it %% under the terms of the GNU General Public License as published by the diff --git a/tex/src/preamble-maneage.tex b/tex/src/preamble-maneage.tex index 9e6cc5a..a2a3a74 100644 --- a/tex/src/preamble-maneage.tex +++ b/tex/src/preamble-maneage.tex @@ -1,6 +1,6 @@ %% Preamble for Maneage-related features. % -%% Copyright (C) 2020-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2020-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> % %% This LaTeX file is part of Maneage. Maneage is free software: you can %% redistribute it and/or modify it under the terms of the GNU General diff --git a/tex/src/preamble-pgfplots.tex b/tex/src/preamble-pgfplots.tex index 74f5290..8a2a98f 100644 --- a/tex/src/preamble-pgfplots.tex +++ b/tex/src/preamble-pgfplots.tex @@ -40,7 +40,7 @@ %% hasn't changed since the last build. TiKZ/PGFPlots won't rebuild the %% plot. % -%% Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> % %% This file is part of Maneage (https://maneage.org). % diff --git a/tex/src/preamble-project.tex b/tex/src/preamble-project.tex index adb99b7..bcc10e5 100644 --- a/tex/src/preamble-project.tex +++ b/tex/src/preamble-project.tex @@ -4,8 +4,8 @@ %% LaTeX usages. However, if any are not needed in your work, please feel %% free to remove them. % -%% Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> -%% Copyright (C) YYYY Your Name <your@email.address> +%% Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) YYYY-YYYY Your Name <your@email.address> % %% This file is free software: you can redistribute it and/or modify it %% under the terms of the GNU General Public License as published by the diff --git a/tex/src/references.tex b/tex/src/references.tex index ae3e5d1..fefb636 100644 --- a/tex/src/references.tex +++ b/tex/src/references.tex @@ -1,7 +1,7 @@ %% Non-software BibTeX entries. The software-specific BibTeX entries are %% stored in a '*.tex' file under the 'tex/dependencies' directory. % -%% Copyright (C) 2018-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Copyright (C) 2018-2026 Mohammad Akhlaghi <mohammad@akhlaghi.org> % %% Copying and distribution of this file, with or without modification, %% are permitted in any medium without royalty provided the copyright |