From ab5e0aa9fa8a6b94b1306fa120736b6fbb1a2f70 Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Mon, 20 Jul 2026 20:20:56 +0200 Subject: IMPORTANT: better names for to-publish dirs and portability fixes Summary: this commit only affects the analysis of your project in case you used the 'tex-publish-dir' or 'data-publish-dir' directories. They need to be renamed to 'figdir' and 'drdir' respectively. Until now, the directories containing the to-be-published data were described deep in the middle of 'initialize.mk' (away from the other shared directories), they also had hard-to-use (long) names. Furthermore, 'README-hacking.md' that is the main documentation of Maneage had grown organically, such that the checklists were scattered across the file and some things were repeated or not classified clearly. Finally, some portability issues were reported by Stergios Amarantidis and Diana Korotun. With this commit, the to-be-published directories have been renamed to the more simple names of 'figdir' and 'drdir' respectively and their definition in 'initialize.mk' has come right under the rest of the top-level directories (to be easily found). Furthermore, the 'README-hacking.md' has been fully restructured to be easier to navigate and follow for a new reader. Finally, the portability issues mentioned above have been addressed: - Within the '--help' output of './project', the '--keep-going' option was listed under "Configure and Make options", while it is only relevant as a Make option. It was therefore moved under Make options. - We had not explicitly set the locale of Maneage; causing a crash on one of the newly tested computers (a macOS with a non-C locale) during the execution of 'prep-source.sh'. With this commit, all the top-level Makefiles and scripts of Maneage set 'LC_ALL' to 'C' to ensure the same locale on any system. - The 'download-multi-try.sh' script would crash when building basic software in the following scenario: the downloader was still from the host, but one of its linked libraries had already been installed in Maneage and conflicted with the host. A step has been added to catch such cases and remove the Maneage'd library when this was the problem. - On macOS, we discovered that 'pkg-config' and 'wget' need a Python installation, so the necessary conditions have been added. - The 'apptainer.sh' script did not allow passing the '--debug' or '--host-cc' options to the execution of './project' within the container. - Within 'configure.sh', we wrote the absolute path of the downloader to use in 'LOCAL.conf'. However, we install both downloaders and it is safer to use Maneage's own downloader when it has been installed. So the downloader name in 'LOCAL.conf' now only has the name (without its absolute path). --- README-hacking.md | 3487 ++++++++++++++++++++++++++++------------------------- 1 file changed, 1816 insertions(+), 1671 deletions(-) (limited to 'README-hacking.md') diff --git a/README-hacking.md b/README-hacking.md index dc6e623..929b14c 100644 --- a/README-hacking.md +++ b/README-hacking.md @@ -1,1059 +1,949 @@ -Maneage: managing data lineage -============================== +# Maneage development Copyright (C) 2018-2026 Mohammad Akhlaghi \ -Copyright (C) 2020-2026 Raul Infante-Sainz \ See the end of the file for license conditions. -Maneage is a **fully working template** for doing reproducible research (or -writing a reproducible paper) as defined in the link below. If the link -below is not accessible at the time of reading, please see the appendix at -the end of this file for a portion of its introduction. Some -[slides](http://akhlaghi.org/pdf/reproducible-paper.pdf) are also available -to help demonstrate the concept implemented here. - - http://akhlaghi.org/reproducible-science.html - -Maneage is created with the aim of supporting reproducible research by -making it easy to start a project in this framework. As shown below, it is -very easy to customize Maneage for any particular (research) project and -expand it as it starts and evolves. It can be run with no modification (as -described in `README.md`) as a demonstration and customized for use in any -project as fully described below. - -A project designed using Maneage will download and build all the necessary -libraries and programs for working in a closed environment (highly -independent of the host operating system) with fixed versions of the -necessary dependencies. The tarballs for building the local environment are -also collected in a [separate -repository](http://git.maneage.org/tarballs-software.git/tree/). The final -output of the project is [a -paper](http://git.maneage.org/output-raw.git/plain/paper.pdf). Notice the -last paragraph of the Acknowledgments where all the necessary software are -mentioned with their versions. - -Below, we start with a discussion of why Make was chosen as the high-level -language/framework for project management and how to learn and master Make -easily (and freely). The general architecture and design of the project is -then discussed to help you navigate the files and their contents. This is -followed by a checklist for the easy/fast customization of Maneage to your -exciting research. We continue with some tips and guidelines on how to -manage or extend your project as it grows based on our experiences with it -so far. There is also a publication checklist, describing the recommended -steps to publish your data/code. The main body concludes with a description -of possible future improvements that are planned for Maneage (but not yet -implemented). As discussed above, we end with a short introduction on the -necessity of reproducible science in the appendix. +Maneage (Managing data lineage) is a customizable workflow controller +program, providing low-level control over a project's data lineage and its +history for a designing and operating fully reproducible project. Maneage +is formally defined in [Akhlaghi et +al. 2021](https://scixplorer.org/abs/2021CSE....23c..82A) (Computing in +Science & Engineering, vol. 23, issue 3, pp. 82-91; +DOI:[10.1109/MCSE.2021.3072860](https://doi.org/10.1109/MCSE.2021.3072860)). The +Maneage webpage at https://maneage.org contains a 30 minute video +presentation of Maneage that you can also watch for a fast introduction. + +This `README-hacking.md` file is targeted at those who want to develop +their projects within Maneage (by "hacking", or modifying, it), not just +execute it. The `README.md` file that is also in this directory contains +the instructions for those who only need to run/execute it. Please don't forget to share your thoughts, suggestions and criticisms. Maintaining and designing Maneage is itself a separate project, -so please join us if you are interested. Once it is mature enough, we will -describe it in a paper (written by all contributors) for a formal -introduction to the community. +so please join us if you are interested. The "Development Workflow" section +below describes our general development workflow. One of our planned tasks +is to write a fully featured documentation (similar to the [Gnuastro +documentation](https://www.gnu.org/software/gnuastro/manual)). +[[_TOC_]] -Why Make? ---------- -When batch processing is necessary (no manual intervention, as in a -reproducible project), shell scripts are usually the first solution that -come to mind. However, the inherent complexity and non-linearity of -progress in a scientific project (where experimentation is key) make it -hard to manage the script(s) as the project evolves. For example, a script -will start from the top/start every time it is run. So if you have already -completed 90% of a research project and want to run the remaining 10% that -you have newly added, you have to run the whole script from the start -again. Only then will you see the effects of the last new steps (to find -possible errors, or better solutions and etc). -It is possible to manually ignore/comment parts of a script to only do a -special part. However, such checks/comments will only add to the complexity -of the script and will discourage you to play-with/change an already -completed part of the project when an idea suddenly comes up. It is also -prone to very serious bugs in the end (when trying to reproduce from -scratch). Such bugs are very hard to notice during the work and frustrating -to find in the end. -The Make paradigm, on the other hand, starts from the end: the final -*target*. It builds a dependency tree internally, and finds where it should -start each time the project is run. Therefore, in the scenario above, a -researcher that has just added the final 10% of steps of her research to -her Makefile, will only have to run those extra steps. With Make, it is -also trivial to change the processing of any intermediate (already written) -*rule* (or step) in the middle of an already written analysis: the next -time Make is run, only rules that are affected by the changes/additions -will be re-run, not the whole analysis/project. -This greatly speeds up the processing (enabling creative changes), while -keeping all the dependencies clearly documented (as part of the Make -language), and most importantly, enabling full reproducibility from scratch -with no changes in the project code that was working during the -research. This will allow robust results and let the scientists get to what -they do best: experiment and be critical to the methods/analysis without -having to waste energy and time on technical problems that come up as a -result of that experimentation in scripts. -Since the dependencies are clearly demarcated in Make, it can identify -independent steps and run them in parallel. This further speeds up the -processing. Make was designed for this purpose. It is how huge projects -like all Unix-like operating systems (including GNU/Linux or Mac OS -operating systems) and their core components are built. Therefore, Make is -a highly mature paradigm/system with robust and highly efficient -implementations in various operating systems perfectly suited for a complex -non-linear research project. -Make is a small language with the aim of defining *rules* containing -*targets*, *prerequisites* and *recipes*. It comes with some nice features -like functions or automatic-variables to greatly facilitate the management -of text (filenames for example) or any of those constructs. For a more -detailed (yet still general) introduction see the article on Wikipedia: +# Maneage checklists - https://en.wikipedia.org/wiki/Make_(software) +This section contians some checklists to help in various stages of your +project: from the first commit, to the final publication. If you notice +anything missing or any in-correct part (probably a change that has not +been explained here), please let us know to correct it. -Make is a +40 year old software that is still evolving, therefore many -implementations of Make exist. The only difference in them is some extra -features over the [standard -definition](https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html) -(which is shared in all of them). Maneage is primarily written in GNU Make -(which it installs itself, you don't have to have it on your system). GNU -Make is the most common, most actively developed, and most advanced -implementation. Just note that Maneage downloads, builds, internally -installs, and uses its own dependencies (including GNU Make), so you don't -have to have it installed before you try it out. +## First custom commit + 1. **Get this repository and its history** (if you don't already have it): + Arguably the easiest way to start is to clone Maneage and prepare for + your customizations as shown below. After the cloning first you rename + the default `origin` remote server to specify that this is Maneage's + remote server. This will allow you to use the conventional `origin` + name for your own project as shown in the next steps. Second, you will + create and go into the conventional `main` branch to start + committing in your project later. + ```shell + $ git clone https://git.maneage.org/project.git # Clone/copy the project and its history. + $ mv project my-project # Change the name to your project's name. + $ cd my-project # Go into the cloned directory. + $ git remote rename origin origin-maneage # Rename current/only remote to "origin-maneage". + $ git checkout -b main # Create and enter your own "main" branch. + $ pwd # Just to confirm where you are. + ``` -How can I learn Make? ---------------------- + 2. The final job of Maneage is to create your paper's PDF. By default it + uses a custom LaTeX style that resembles that of the Astrophysical + Journal (because the precursor of Maneage was for [Akhlaghi & Ichikawa + 2015](https://ui.adsabs.harvard.edu/abs/2015ApJS..220....1A)). The + journal you plan to submit your paper to will have its own separate + style. So it is best that you start your project by writing in the + desired style. We have already customized Maneage for the official + styles of some journals. To find them, run `git branch -r | grep + journal`. If your planned journal is one of them, you can take the + following steps to start your project based on that journal's style. If + it is not in these, you can ignore this step for now and customize the + style later (you can model based on these branches). In the commands + below, we'll assume you want to prepare for the Astronomy and + Astrophysics journal (A&A). -The GNU Make book/manual (links below) is arguably the best place to learn -Make. It is an excellent and non-technical book to help get started (it is -only non-technical in its first few chapters to get you started easily). It -is freely available and always up to date with the current GNU Make -release. It also clearly explains which features are specific to GNU Make -and which are general in all implementations. So the first few chapters -regarding the generalities are useful for all implementations. + ```shell + $ git checkout -b journal origin-maneage/journal-a-and-a + $ git log -1 --oneline | awk '{print $1}' # To keep the commit hash + $ git rebase -i main # See description below + ``` -The first link below points to the GNU Make manual in various formats and -in the second, you can download it in PDF (which may be easier for a first -time reading). + In the first text editor that opens after the last command, change all + (except the first) `pick`s into `squash`, then save the change and + close the editor. In case there is no conflict, the second editor will + be pre-filled with all the commit messages in that branch. You do not + need those, so you can delete everything and write a commit message + like the following: `A&A journal (commit XXXXX of Maneage's + journal-a-and-a branch)`. Just replace the `XXXXX` with the output of + the second command above. The commit hash is important to be stored + here since it allows you to later check if any updates have been made + in that branch in the future. After completing the git rebase operation + (last command above), run the following commands below to put the new + commit in your `main` branch (and continue working based on that). - https://www.gnu.org/software/make/manual/ + ```shell + $ git checkout main + $ git merge journal + $ git branch -D journal + ``` - https://www.gnu.org/software/make/manual/make.pdf + 3. **Prepare for configuration**: The `./harvester configure` command of + the next step will build the different software packages within the + "build" directory (that you will specify). Nothing else on your system + will be touched. However, since it takes long, it is useful to see + what it is being built at every instant (its almost impossible to tell + from the torrent of commands that are produced!). So open another + terminal on your desktop and navigate to the same project directory + that you cloned (output of last command above). Then run the following + command. Once every second, this command will just print the date + (possibly followed by a non-existent directory notice). But as soon as + the next step starts building software, you'll see the names of + software get printed as they are being built. Once any software is + installed in the project build directory it will be removed. Again, + don't worry, nothing will be installed outside the build directory. -If you use GNU Make, you also have the whole GNU Make manual on the -command-line with the following command (you can come out of the "Info" -environment by pressing `q`). + ```shell + # On another terminal (go to top project source directory, last command above) + $ ./harvester --check-config + ``` -```shell - $ info make -``` + 4. **Test Maneage**: Before making any changes, it is important to test it + and see if everything works properly with the commands below. If there + is any problem in the `./harvester configure` or `./harvester make` steps, + please contact us to fix the problem before continuing. Since the + building of dependencies in configuration can take long, you can take + the next few steps (editing the files) while its working (they don't + affect the configuration). After `./harvester make` is finished, open + `paper.pdf`. If it looks fine, you are ready to start customizing the + Maneage for your project. But before that, clean all the extra Maneage + outputs with `make clean` as shown below. -If you aren't familiar with the Info documentation format, we strongly -recommend running `$ info info` and reading along. In less than an hour, -you will become highly proficient in it (it is very simple and has a great -manual for itself). Info greatly simplifies your access (without taking -your hands off the keyboard!) to many manuals that are installed on your -system, allowing you to be much more efficient as you work. If you use the -GNU Emacs text editor (or any of its variants), you also have access to all -Info manuals while you are writing your projects (again, without taking -your hands off the keyboard!). + ```shell + $ ./harvester configure # Build the project's software environment (can take an hour or so). + $ ./harvester make # Do the processing and build paper (just a simple demo). + # Open 'paper.pdf' and see if everything is ok. + ``` + 5. **Setup the remote**: You can use any [hosting + facility](https://en.wikipedia.org/wiki/Comparison_of_source_code_hosting_facilities) + that supports Git to keep an online copy of your project's version + controlled history. We recommend [GitLab](https://gitlab.com) because + it is [more ethical (although not + perfect)](https://www.gnu.org/software/repo-criteria-evaluation.html), + and later you can also host GitLab on your own server. Anyway, create + an account in your favorite hosting facility (if you don't already + have one), and define a new project there. Please make sure *the newly + created project is empty* (some services ask to include a `README` in + a new project which is bad in this scenario, and will not allow you to + push to it). It will give you a URL (usually starting with `git@` and + ending in `.git`), put this URL in place of `XXXXXXXXXX` in the first + command below. With the second command, "push" your `main` branch to + your `origin` remote, and (with the `--set-upstream` option) set them + to track/follow each other. However, the `maneage` branch is currently + tracking/following your `origin-maneage` remote (automatically set + when you cloned Maneage). So when pushing the `maneage` branch to your + `origin` remote, you _shouldn't_ use `--set-upstream`. With the last + command, you can actually check this (which local and remote branches + are tracking each other). + ```shell + git remote add origin XXXXXXXXXX # Newly created repo is now called 'origin'. + git push --set-upstream origin main # Push 'main' branch to 'origin' (with tracking). + git push origin maneage # Push 'maneage' branch to 'origin' (no tracking). + ``` + 6. **Title**, **short description** and **author**: You can start adding + your name (with your possible coauthors) and tentative abstract in + `paper.tex`. You should see the relevant place in the preamble (prior + to `\begin{document}`. Just note that some core project metadata like + the project title are actually set in + `reproduce/analysis/config/metadata.conf`. So set your project title + in there. After you are done, run the `./harvester make` command again + to see your changes in the final PDF and make sure that your changes + don't cause a crash in LaTeX. Of course, if you use a different LaTeX + package/style for managing the title and authors (in particular a + specific journal's style), please feel free to use it your own methods + after finishing this checklist and doing your first commit. -Published works using Maneage ------------------------------ + 7. **Delete dummy parts**: Maneage contains some parts that are only for + the initial/test run, mainly as a demonstration of important steps, + which you can use as a reference to use in your own project. But they + not for any real analysis, so you should remove these parts as + described below: -The list below shows some of the works that have already been published -with (earlier versions of) Maneage, and some that have been recently -submitted for peer review. Maneage is evolving rapidly, so some details -will differ between the different versions. The more recent papers will -tend to be the most useful as good working examples if you are just -starting. + - `paper.tex`: 1) Delete the text of the abstract (from + `\includeabstract{` to `\vspace{0.25cm}`) and write your own (a + single sentence can be enough now, you can complete it later). 2) + Add some keywords under it in the keywords part. 3) Delete + everything between `%% Start of main body.` and `%% End of main + body.`. 4) Remove the notice in the "Acknowledgments" section (in + `\new{}`) and Acknowledge your funding sources (this can also be + done later). Just don't delete the existing acknowledgment + statement: Maneage is possible thanks to funding from several + grants. Since Maneage is being used in your work, it is necessary to + acknowledge them in your work also. -In the list below, by clicking on the paper's publication year, you will be -taken to the paper's indexed page on the [SciX Digital -Library](https://scixplorer.org) (successor to the [NASA -ADS](https://ui.adsabs.harvard.edu)) which contains further bibliographic -information like title, DOI and etc, in a unified format (independent of -which journal the paper is published in). + - `reproduce/analysis/make/top-make.mk`: Delete the `delete-me` line + in the `makesrc` definition. Just make sure there is no empty line + between the `initialize \` and `verify \` lines (they should be + directly under each other). - - Eskandarlou et - al. ([2026](https://scixplorer.org/abs/2026A%26A...709A.210E), A&A - Volume 709, id.A210, 17 pp; - arXiv:[2603.22166](https://arxiv.org/abs/2603.22166)). The - project's version controlled source is on - [Gitlab](https://gitlab.com/sepideh.esk/alba), necessary software, - outputs and backup of history are available at - [zenodo.17674805](http://doi.org/10.5281/zenodo.17674805); and the - archived git history is available at - [swh:1:dir:7c203a565ec6ef40d8419dbdf70e8639b78f4b99](https://archive.softwareheritage.org/swh:1:dir:7c203a565ec6ef40d8419dbdf70e8639b78f4b99;origin=https://gitlab.com/Sepideh.Esk/alba;visit=swh:1:snp:adc584ff3540ea3151de6ebc8462805e8818b1f4;anchor=swh:1:rev:f9d0f9cb25c80415107c962cc53d369a650e9b82). - - Eskandarlou et - al. ([2026](https://scixplorer.org/abs/2026A%26A...705A..67E), A&A, - Volume 705, id.A67, 14 pp; - arXiv:[2510.12940](https://arxiv.org/abs/2510.12940)). The project's - version controlled source is on - [Gitlab](https://gitlab.com/sepideh.esk/psf-j-plus), necessary software, - outputs and backup of history are available at - [zenodo.17348653](http://doi.org/10.5281/zenodo.17348653); and the - archived git history is available at - [swh:1:dir:fc95ed9ad173de6fa64690e3d71ab041c630f32c](https://archive.softwareheritage.org/swh:1:dir:fc95ed9ad173de6fa64690e3d71ab041c630f32c;origin=https://gitlab.com/Sepideh.Esk/psf-j-plus;visit=swh:1:snp:31bb4550902f7902a0a65d0674106271de776d3e;anchor=swh:1:rev:4860c70d6285a60300e429889e8637e98568a915). - - Saremi et al. ([2025](https://scixplorer.org/abs/2025A%26A...701A.116S), - A&A, Volume 701, id.A116, 18 pp.; - arXiv:[2508.02780](https://arxiv.org/abs/2508.02780)). The project's - version controlled source is on - [Gitlab](https://gitlab.com/nasim-projects/pipeline), necessary - software, outputs and backup of history are available at - [zenodo.16152699](https://doi.org/10.5281/zenodo.16152699); and the - archived git history is available at - [swh:1:dir:b3657cfb6053fd976695bd63c15cb99e5095648a](https://archive.softwareheritage.org/swh:1:dir:b3657cfb6053fd976695bd63c15cb99e5095648a;origin=https://gitlab.com/nasim-projects/pipeline;visit=swh:1:snp:ab7c6f0b9999f42d77154103c1bc082fa23b325c;anchor=swh:1:rev:afeb282c01983cba2a11eb4b2f25d5a40d35c164). - - Eskandarlou & Akhlaghi - ([2024](https://scixplorer.org/abs/2024RNAAS...8..168E), RNAAS, Volume - 8, Issue 6, id.168; - arXiv:[2406.14619](https://arxiv.org/abs/2406.14619)). The project's - version controlled source is on - [Codeberg](https://codeberg.org/gnuastro/papers) (the `polar-plot` - branch). Necessary software, outputs and backup of history are available - at [zenodo.11403643](https://doi.org/10.5281/zenodo.11403643); and the - archived git history is available at - [swh:1:dir:4e09bf85f9f87336fa55920bf67e7bcf6d58bbd5](https://archive.softwareheritage.org/swh:1:dir:4e09bf85f9f87336fa55920bf67e7bcf6d58bbd5;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:557ee1a90de465659659ecc46df0c5ce29d0bb61;anchor=swh:1:rev:375e12e52080006be6a28e10980e79ef54d13d1d). - - Infante-Sainz et - al. ([2024](https://scixplorer.org/abs/2024RNAAS...8...22I), RNAAS, - Volume 8, Issue 1, id.22; - arXiv:[2401.05303](https://arxiv.org/abs/2401.05303)). The project's - version controlled source is on - [Codeberg](https://codeberg.org/gnuastro/papers) (the `radial-profile` - branch). Necessary software, outputs and backup of history are available - at [zenodo.10124582](https://doi.org/10.5281/zenodo.10124582); and the - archived git history is available at - [swh:1:dir:d5029e066916cb64f0d95d20eb88294acc78b2b1](https://archive.softwareheritage.org/swh:1:dir:d5029e066916cb64f0d95d20eb88294acc78b2b1;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:b065324c2ef3b48bc26e8f30e48102a1abd2052f;anchor=swh:1:rev:61764447b16da44538e5ddbf7fb69937ba138e81). - - Infante-Sainz & Akhlaghi - ([2024](https://scixplorer.org/abs/2024RNAAS...8...10I), RNAAS, Volume - 8, Issue 1, id.10; - arXiv:[2401.03814](https://arxiv.org/abs/2401.03814)). The project's - version controlled source is on - [Codeberg](https://codeberg.org/gnuastro/papers) (the `color-faint-gray` - branch). Necessary software, outputs and backup of history are available - at [zenodo.10058165](https://doi.org/10.5281/zenodo.10058165); and the - archived git history is available at - [swh:1:dir:1064a48d4bb58d6684c3df33c6633a04d4141d2d](https://archive.softwareheritage.org/swh:1:dir:1064a48d4bb58d6684c3df33c6633a04d4141d2d;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:a083ff647c571f895d1ccc9f7432fa1b9a1d03a8;anchor=swh:1:rev:ff77b619daa50b05ddd83206d979d1f8a53d040b). - - Peper, Roukema & Bolejko - ([2023](https://scixplorer.org/abs/2023MNRAS.525...91P), MNRAS, 525, 91, - DOI:10.1093/mnras/stad2246, - arXiv:[2304.00591](https://arxiv.org/abs/2304.00591)): The live version - of the controlled source is [at - Codeberg](https://codeberg.org/mpeper/lensing); the main input - dataset, a software snapshot, the software tarballs, the project outputs - and editing history are available at - [zenodo.8103985](https://zenodo.org/record/8103985); and the archived - git history is available at - [swh:1:dir:f57fd0e254bcb976c1ab34b6c9c9708e891f7914](https://archive.softwareheritage.org/swh:1:dir:f57fd0e254bcb976c1ab34b6c9c9708e891f7914;origin=https://codeberg.org/mpeper/lensing;visit=swh:1:snp:984413c49bb02ee0eec88c3ac01614808d57beb7;anchor=swh:1:rev:185c36f1b6263588526be3881078626869041882). - - Eskandarlou et - al. ([2023](https://scixplorer.org/abs/2023RNAAS...7..269E), RNAAS, - Volume 7, Issue 12, id.269; - arXiv:[2312.04263](https://arxiv.org/abs/2312.04263)). The project's - version controlled source is on - [Codeberg](https://codeberg.org/gnuastro/papers) (the `zeropoint` - branch). Necessary software, outputs and backup of history are available - at [zenodo.10256845](https://doi.org/10.5281/zenodo.10256845); and the - archived git history is available at - [swh:1:dir:8b2d1f63be96de3de03aa3e2bb68fa7fa52df56f](https://archive.softwareheritage.org/swh:1:dir:8b2d1f63be96de3de03aa3e2bb68fa7fa52df56f;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:e37e226bab517eef24d854467682b2fcf5d7dc32;anchor=swh:1:rev:ea682783d83707c0e1d114a5de74a100be9f545d). - - Akhlaghi ([2023](https://scixplorer.org/abs/2023RNAAS...7..211A), RNAAS), Volume 7, Issue - 10, id.211; arXiv:[2310.15006](https://arxiv.org/abs/2310.15006)). The - project's version controlled source is on - [Codeberg](https://codeberg.org/gnuastro/papers) (the - `pointing-simulate` branch). - - Borkowska & Roukema - ([2022](https://scixplorer.org/abs/2022CQGra..39u5007B), Classical and - Quantum Gravity, Volume 39, Issue 21, id.215007, 29 pp; - arXiv:[2112.14174](https://arxiv.org/abs/2112.14174)): The live version - of the controlled source is [at - Codeberg](https://codeberg.org/boud/gevcurvtest); the main input - dataset, a software snapshot, the software tarballs, the project outputs - and editing history are available at - [zenodo.5806027](https://doi.org/10.5281/zenodo.5806027); and the - archived git history is available at - [swh:1:rev:54398b720ddbac269ede30bf1e27fe27f07567f7](https://archive.softwareheritage.org/browse/revision/54398b720ddbac269ede30bf1e27fe27f07567f7). - - Peper & Roukema ([2021](https://scixplorer.org/abs/2021MNRAS.505.1223P), - MNRAS, 505, 1223; arXiv:[2010.03742](https://arxiv.org/abs/2010.03742)): - The live version of the controlled source is [at - Codeberg](https://codeberg.org/boud/elaphrocentre); the main input - dataset, a software snapshot, the software tarballs, the project outputs - and editing history are available at - [zenodo.4699702](https://zenodo.org/record/4699702); and the archived - git history is available at - [swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e](https://archive.softwareheritage.org/swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e). - - Roukema ([2021](https://scixplorer.org/abs/2021PeerJ...911856R), PeerJ, - Vol. 9, article id. e11856; - arXiv:[2007.11779](https://arxiv.org/abs/2007.11779)): The live version - of the controlled source is [at - Codeberg](https://codeberg.org/boud/subpoisson); the main input dataset, - a software snapshot, the software tarballs, the project outputs and - editing history are available at - [zenodo.4765705](https://zenodo.org/record/4765705); and the archived - git history is available at - [swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37](https://archive.softwareheritage.org/swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37). - - Akhlaghi et al. ([2021](https://scixplorer.org/abs/2021CSE....23c..82A), - CiSE, 23(3), 82; arXiv:[2006.03018](https://arxiv.org/abs/2006.03018)): - The project's version controlled source is [on - Gitlab](https://gitlab.com/makhlaghi/maneage-paper), necessary software, - outputs and backup of history are available at - [zenodo.3872248](https://doi.org/10.5281/zenodo.3872248); and the - archived git history is available at - [swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717](https://archive.softwareheritage.org/swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717). - - Infante-Sainz et - al. ([2020](https://scixplorer.org/abs/2020MNRAS.491.5317I), MNRAS, 491, - 5317; arXiv:[1911.01430](https://arxiv.org/abs/1911.01430)): The version - controlled project source is available [on - GitLab](https://gitlab.com/infantesainz/sdss-extended-psfs-paper) and is - also archived on Zenodo with all the necessary software tarballs: - [zenodo.3524937](https://zenodo.org/record/3524937). - - Akhlaghi ([2019](https://scixplorer.org/abs/2019arXiv190911230A), IAU - Symposium 355; - arXiv:[1909.11230](https://arxiv.org/abs/1909.11230)). The version - controlled project source is available [on - GitLab](https://gitlab.com/makhlaghi/iau-symposium-355) and is also - archived on Zenodo with all the necessary software tarballs: - [zenodo.3408481](https://doi.org/10.5281/zenodo.3408481). - -The items below were the precursors to Maneage. In other words, the work to -have reproducible results in the items below lead to what is now -Maneage. They are therefore much less complete than what Maneage is today, -but are important from a historical perspective. - - Section 7.3 of Bacon et - al. ([2017](https://scixplorer.org/abs/2017A%26A...608A...1B), A&A 608, - A1; arXiv:[1710.03002](https://arxiv.org/abs/1710.03002)): The version - controlled project source is available [on - GitLab](https://gitlab.com/makhlaghi/muse-udf-origin-only-hst-magnitudes) - and a snapshot of the project along with all the necessary input - datasets and outputs is available in - [zenodo.1164774](https://doi.org/10.5281/zenodo.1164774). - - Section 4 of Bacon et - al. ([2017](https://scixplorer.org/abs/2017A%26A...608A...1B), A&A, 608, - A1; arXiv:[1710.03002](https://arxiv.org/abs/1710.03002)): The version - controlled project is available [on - GitLab](https://gitlab.com/makhlaghi/muse-udf-photometry-astrometry) and - a snapshot of the project along with all the necessary input datasets is - available in [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746). - - Akhlaghi & Ichikawa - ([2015](https://scixplorer.org/abs/2015ApJS..220....1A), ApJS, 220, 1; - arXiv:[1505.01664](https://arxiv.org/abs/1505.01664)): The version - controlled project is available [on - GitLab](https://gitlab.com/makhlaghi/NoiseChisel-paper). This is the - very first (and much less mature!) incarnation of Maneage: the history - of Maneage started more than two years after this paper was - published. It is a very rudimentary/initial implementation, thus it is - only included here for historical reasons. However, the project source - is complete, accurate and uploaded to arXiv along with the paper. + - `reproduce/analysis/make/initialize.mk`: in the very end of this + file, you will see a set of lines between `delete the lines below + this` and `delete the lines above this`. Delete this whole group of + lines (including the two instruction lines). + - `reproduce/analysis/config/verify-outputs.conf`: Disable + verification of outputs by changing the `yes` (the value of + `verify-outputs`) to `no`. Later, when you are ready to submit your + paper, or publish the dataset, activate verification and make the + proper corrections in this file (described under the "Other basic + customizations" section below). This is a critical step and only + takes a few minutes when your project is finished. So DON'T FORGET + to activate it in the end. + - Delete all `delete-me*` files in the following directories: + ```shell + $ rm tex/src/delete-me* + $ rm reproduce/analysis/make/delete-me* + $ rm reproduce/analysis/config/delete-me* + ``` + - Re-make the project (after a cleaning) to see if you haven't + introduced any errors. -Citation --------- + ```shell + $ ./harvester make clean + $ ./harvester make + ``` -If you use Maneage in your project please cite Akhlaghi et -al. ([2020](https://arxiv.org/abs/2006.03018), arXiv:2006.03018). It has -been submitted and is under peer review. + 8. **Ignore changes in some Maneage files**: One of the main advantages of + Maneage is that you can later update your infra-structure by merging + your `main` branch with the `maneage` branch. This is good for many + low-level features that you will likely never modify yourself. But it + is not desired for some files like `paper.tex` (you don't want changes + in Maneage's default `paper.tex` to cause conflicts with all the text + you have already written for your project). You need to tell Git to + ignore changes in such files from the `maneage` branch during the + merge, and just keep your own branch's version. The commands below + show how you can avert such future conflicts and frustrations with + some known files. Note that only the first `echo` command has a `>` + (to write over the file), the rest are `>>` (to append to it). If you + want to avoid any other set of files to be imported from Maneage into + your project's branch, you can follow a similar strategy (it should + happen rarely, if at all!). Generally be very careful about adding + files to `.gitattributes` because it affects the whole file and if a + wrong file is ignored, Maneage may break after a merge (some + inter-dependent files may not get updated together). We recommend only + doing it when you encounter the same conflict in more than one merge, + and are sure that it won't affect other files. In such cases please + let us know so we can improve the design of Maneage and modularize + those components to be easily added here. -Also, when your paper is published, don't forget to add a notice in your -own paper (in coordination with the publishing editor) that the paper is -fully reproducible and possibly add a sentence or paragraph in the end of -the paper shortly describing the concept. This will help spread the word -and encourage other scientists to also manage and publish their projects in -a reproducible manner. + ```shell + $ echo "paper.tex merge=ours" > .gitattributes + $ echo "tex/src/*.tex merge=ours" >> .gitattributes + $ echo "reproduce/analysis/config/*.conf merge=ours" >> .gitattributes + $ echo "reproduce/software/config/TARGETS.conf merge=ours" >> .gitattributes + $ echo "reproduce/software/config/texlive-packages.conf merge=ours" >> .gitattributes + $ git add .gitattributes + ``` + 9. **Copyright and License notice**: It is necessary that _all_ the + "copyright-able" files in your project (those larger than 10 lines) + have a copyright and license notice. Please take a moment to look at + several existing files to see a few examples. The copyright notice is + usually close to the start of the file, it is the line starting with + `Copyright (C)` and containing a year and the author's name (like the + examples below). The License notice is a short description of the + copyright license, usually one or two paragraphs with a URL to the + full license. Don't forget to add these _two_ notices to *any new + file* you add in your project (you can just copy-and-paste). When you + modify an existing Maneage file (which already has the notices), just + add a copyright notice in your name under the existing one(s), like + the line with capital letters below. To start with, add this line with + your name and email address to `paper.tex`, + `tex/src/preamble-project.tex`, `reproduce/analysis/make/top-make.mk`, + and generally, all the files you modified in the previous step. + ``` + Copyright (C) 2018-2026 Existing Name + Copyright (C) 2026-2026 YOUR NAME + ``` + 10. **Configure Git for fist time**: If this is the first time you are + running Git on this system, then you have to configure it with some + basic information in order to have essential information in the commit + messages (ignore this step if you have already done it). Git will + include your name and e-mail address information in each commit. You + can also specify your favorite text editor for making the commit + (`emacs`, `vim`, `nano`, and etc.). + ```shell + $ git config --global user.name "YourName YourSurname" + $ git config --global user.email your-email@example.com + $ git config --global core.editor nano + ``` + 11. **Your first commit**: You have already made some small and basic + changes in the steps above and you are in your project's `main` + branch. So, you can officially make your first commit in your + project's history and push it. But before that, you need to make sure + that there are no problems in the project. This is a good habit to + always re-build the system before a commit to be sure it works as + expected. + ```shell + $ git status # See which files you have changed. + $ git diff # Check the lines you have added/changed. + $ ./harvester make # Make sure everything builds successfully. + $ git add -u # Put all tracked changes in staging area. + $ git status # Make sure everything is fine. + $ git diff --cached # Confirm all the changes that will be committed. + $ git commit # Your first commit: put a good description! + $ git push # Push your commit to your remote. + ``` + 12. **Read the publication checklist**: The publication checklist below is + very similar to this one, but for the final phase of your project. For + now, you don't have to do any of its steps, but reading it will give + you good insight into the later stages of your project. If you already + know how you want to publish your project, you can implement many of + those steps from the start and during the actual project (in + particular how to organize your data files that go into the plots). + Making it much easier to complete that checklist when you are ready + for submission. + 13. **Start your exciting research**: You are now ready to add flesh and + blood to this raw skeleton by further modifying and adding your + exciting research steps. You can use the "published works" section in + the introduction (above) as some fully working models to learn + from. Also, don't hesitate to contact us if you have any + questions. -Project architecture -==================== -In order to customize Maneage to your research, it is important to first -understand its architecture so you can navigate your way in the directories -and understand how to implement your research project within its framework: -where to add new files and which existing files to modify for what -purpose. But if this the first time you are using Maneage, before reading -this theoretical discussion, please run Maneage once from scratch without -any changes (described in `README.md`). You will see how it works (note that -the configure step builds all necessary software, so it can take long, but -you can continue reading while its working). - -The project has two top-level directories: `reproduce` and -`tex`. `reproduce` hosts all the software building and analysis -steps. `tex` contains all the final paper's components to be compiled into -a PDF using LaTeX. - -The `reproduce` directory has two sub-directories: `software` and -`analysis`. As the name says, the former contains all the instructions to -download, build and install (independent of the host operating system) the -necessary software (these are called by the `./project configure` -command). The latter contains instructions on how to use those software to -do your project's analysis. - -After it finishes, `./project configure` will create the following symbolic -links in the project's top source directory: `.build` which points to the -top build directory and `.local` for easy access to the custom built -software installation directory. With these you can easily access the build -directory and project-specific software from your top source directory. For -example if you run `.local/bin/ls` you will be using the `ls` of Maneage, -which is probably different from your system's `ls` (run them both with -`--version` to check). - -Once the project is configured for your system, `./project make` will do -the basic preparations and run the project's analysis with the custom -version of software. The `project` script is just a wrapper, and with the -`make` argument, it will first call `top-prepare.mk` and `top-make.mk` -(both are in the `reproduce/analysis/make` directory). +## Other basic customizations -In terms of organization, `top-prepare.mk` and `top-make.mk` have an -identical design, only minor differences. So, let's continue Maneage's -architecture with `top-make.mk`. Once you understand that, you'll clearly -understand `top-prepare.mk` also. These very high-level files are -relatively short and heavily commented so hopefully the descriptions in -each comment will be enough to understand the general details. As you read -this section, please also look at the contents of the mentioned files and -directories to fully understand what is going on. +The checklist for the first commit above does not include other basic +customizations that will be necessary once you start using maneaging your +project. Here, we list some of the other steps that are usually necessary. -Before starting to look into the top `top-make.mk`, it is important to -recall that Make defines dependencies by files. Therefore, the -input/prerequisite and output of every step/rule must be a file. Also -recall that Make will use the modification date of the prerequisite(s) and -target files to see if the target must be re-built or not. Therefore during -the processing, _many_ intermediate files will be created (see the tips -section below on a good strategy to deal with large/huge files). + - **High-level software**: Maneage installs all the software that your + project needs. You can specify which software your project needs in + `reproduce/software/config/TARGETS.conf`. The necessary software are + classified into two classes: 1) programs or libraries (usually written + in C/C++) which are run directly by the operating system. 2) Python + modules/libraries that are run within Python. By default + `TARGETS.conf` only has GNU Astronomy Utilities (Gnuastro) as one + scientific program and Astropy as one scientific Python module. Both + have many dependencies which will be installed into your project + during the configuration step. To see a list of software that are + currently ready to be built in Maneage, see + `reproduce/software/config/versions.conf` (which has their versions + also), the comments in `TARGETS.conf` describe how to use the software + name from `versions.conf`. Currently the raw pipeline just uses + Gnuastro to make the demonstration plots. Therefore if you don't need + Gnuastro, go through the analysis steps in `reproduce/analysis` and + remove all its use cases (clearly marked). -To keep the source and (intermediate) built files separate, the user _must_ -define a top-level build directory variable (or `$(BDIR)`) to host all the -intermediate files (you defined it during `./project configure`). This -directory doesn't need to be version controlled or even synchronized, or -backed-up in other servers: its contents are all products, and can be -easily re-created any time. As you define targets for your new rules, it is -thus important to place them all under sub-directories of `$(BDIR)`. As -mentioned above, you always have fast access to this "build"-directory with -the `.build` symbolic link. Also, beware to *never* make any manual change -in the files of the build-directory, just delete them (so they are -re-built). + - **Input datasets**: The input datasets are managed through the + `reproduce/analysis/config/INPUTS.conf` file. It is best to gather the + following information regarding all the input datasets into this one + central file: 1) the SHA256 checksum of the file, 2) the URL where the + file can be downloaded online. Please read the comments at the start + of `reproduce/analysis/config/INPUTS.conf` carefully. -In this architecture, we have two types of Makefiles that are loaded into -the top `Makefile`: _configuration-Makefiles_ (only independent -variables/configurations) and _workhorse-Makefiles_ (Makefiles that -actually contain analysis/processing rules). - -The configuration-Makefiles are those that satisfy these two wildcards: -`reproduce/software/config/*.conf` (for building the necessary software -when you run `./project configure`) and `reproduce/analysis/config/*.conf` -(for the high-level analysis, when you run `./project make`). These -Makefiles don't actually have any rules, they just have values for various -free parameters throughout the configuration or analysis. Open a few of -them to see for yourself. These Makefiles must only contain raw Make -variables (project configurations). By "raw" we mean that the Make -variables in these files must not depend on variables in any other -configuration-Makefile. This is because we don't want to assume any order -in reading them. It is also very important to *not* define any rule, or -other Make construct, in these configuration-Makefiles. + - **`README.md`**: Correct all the `XXXXX` place holders (name of your + project, your own name, address of your project's online/remote + repository, link to download dependencies and etc). Generally, read + over the text and update it where necessary to fit your project. Don't + forget that this is the first file that is displayed on your online + repository and also your colleagues will first be drawn to read this + file. Therefore, make it as easy as possible for them to start + with. Also check and update this file one last time when you are ready + to publish your project's paper/source. -Following this rule-of-thumb enables you to set these configure-Makefiles -as a prerequisite to any target that depends on their variable -values. Therefore, if you change any of their values, all targets that -depend on those values will be re-built. This is very convenient as your -project scales up and gets more complex. + - **Verify outputs**: During the initial customization checklist, you + disabled verification. This is natural because during the project you + need to make changes all the time and its a waste of time to enable + verification every time. But at significant moments of the project + (for example before submission to a journal, or publication) it is + necessary. When you activate verification, before building the paper, + all the specified datasets will be compared with their respective + checksum and if any file's checksum is different from the one recorded + in the project, it will stop and print the problematic file and its + expected and calculated checksums. First set the value of + `verify-outputs` variable in + `reproduce/analysis/config/verify-outputs.conf` to `yes`. Then go to + `reproduce/analysis/make/verify.mk`. The verification of all the files + is only done in one recipe. First the files that go into the + plots/figures are checked, then the LaTeX macros. Validation of the + former (inputs to plots/figures) should be done manually. If its the + first time you are doing this, you can see two examples of the dummy + steps (with `delete-me`, you can use them if you like). These two + examples should be removed before you can run the project. For the + latter, you just have to update the checksums. The important thing to + consider is that a simple checksum can be problematic because some + file generators print their run-time date in the file (for example as + commented lines in a text table). When checking text files, this + Makefile already has this function: + `verify-txt-no-comments-leading-space`. As the name suggests, it will + remove comment lines and empty lines before calculating the MD5 + checksum. For FITS formats (common in astronomy, fortunately there is + a `DATASUM` definition which will return the checksum independent of + the headers. You can use the provided function(s), or define one for + your special formats. -The workhorse-Makefiles are those satisfying this wildcard -`reproduce/software/make/*.mk` and `reproduce/analysis/make/*.mk`. They -contain the details of the processing steps (Makefiles containing -rules). Therefore, in this phase *order is important*, because the -prerequisites of most rules will be the targets of other rules that will be -defined prior to them (not a fixed name like `paper.pdf`). The lower-level -rules must be imported into Make before the higher-level ones. + - **Feedback**: As you use Maneage you will notice many things that if + implemented from the start would have been very useful for your + work. This can be in the actual scripting and architecture of Maneage, + or useful implementation and usage tips, like those below. In any + case, please share your thoughts and suggestions with us, so we can + add them here for everyone's benefit. -All processing steps are assumed to ultimately (usually after many rules) -end up in some number, image, figure, or table that will be included in the -paper. The writing of these results into the final report/paper is managed -through separate LaTeX files that only contain macros (a name given to a -number/string to be used in the LaTeX source, which will be replaced when -compiling it to the final PDF). So the last target in a workhorse-Makefile -is a `.tex` file (with the same base-name as the Makefile, but in -`$(BDIR)/tex/macros`). As a result, if the targets in a workhorse-Makefile -aren't directly a prerequisite of other workhorse-Makefile targets, they -can be a prerequisite of that intermediate LaTeX macro file and thus be -called when necessary. Otherwise, they will be ignored by Make. + - **Re-preparation**: Automatic preparation is only run in the first run + of the project on a system, to re-do the preparation you have to use + the option below. Here is the reason for this: when its necessary, the + preparation process can be slow and will unnecessarily slow down the + whole project while the project is under development (focus is on the + analysis that is done after preparation). Because of this, preparation + will be done automatically for the first time that the project is run + (when `.build/software/preparation-data.mk` doesn't exist). After the + preparation process completes once, future runs of `./harvester make` + will not do the preparation process anymore (will not call + `top-prepare.mk`). They will only call `top-make.mk` for the + analysis. To manually invoke the preparation process after the first + attempt, the `./harvester make` script should be run with the + `--prepare-redo` option, or you can delete the special file above. -Maneage also has a mode to share the build directory between several -users of a Unix group (when working on large computer clusters). In this -scenario, each user can have their own cloned project source, but share the -large built files between each other. To do this, it is necessary for all -built files to give full permission to group members while not allowing any -other users access to the contents. Therefore the `./project configure` and -`./project make` steps must be called with special conditions which are -managed in the `--group` option. + ```shell + $ ./harvester make --prepare-redo + ``` -Let's see how this design is implemented. Please open and inspect -`top-make.mk` it as we go along here. The first step (un-commented line) is -to import the local configuration (your answers to the questions of -`./project configure`). They are defined in the configuration-Makefile -`reproduce/software/config/LOCAL.conf` which was also built by `./project -configure` (based on the `LOCAL.conf.in` template of the same directory). + - **Pre-publication**: add notice on reproducibility**: Add a notice + somewhere prominent in the first page within your paper, informing the + reader that your research is fully reproducible. For example in the + end of the abstract, or under the keywords with a title like + "reproducible paper". This will encourage them to publish their own + works in this manner also and also will help spread the word. -The next non-commented set of the top `Makefile` defines the ultimate -target of the whole project (`paper.pdf`). But to avoid mistakes, a sanity -check is necessary to see if Make is being run with the same group settings -as the configure script (for example when the project is configured for -group access using the `./for-group` script, but Make isn't). Therefore we -use a Make conditional to define the `all` target based on the group -permissions. -Having defined the top/ultimate target, our next step is to include all the -other necessary Makefiles. However, order matters in the importing of -workhorse-Makefiles and each must also have a TeX macro file with the same -base name (without a suffix). Therefore, the next step in the top-level -Makefile is to define the `makesrc` variable to keep the base names -(without a `.mk` suffix) of the workhorse-Makefiles that must be imported, -in the proper order. -Finally, we import all the necessary remaining Makefiles: 1) All the -analysis configuration-Makefiles with a wildcard. 2) The software -configuration-Makefile that contains their version (just in case its -necessary). 3) All workhorse-Makefiles in the proper order using a Make -`foreach` loop. +## Upgrading the Maneage branch -In short, to keep things modular, readable and manageable, follow these -recommendations: 1) Set clear-to-understand names for the -configuration-Makefiles, and workhorse-Makefiles, 2) Only import other -Makefiles from top Makefile. These will let you know/remember generally -which step you are taking before or after another. Projects will scale up -very fast. Thus if you don't start and continue with a clean and robust -convention like this, in the end it will become very dirty and hard to -manage/understand (even for yourself). As a general rule of thumb, break -your rules into as many logically-similar but independent steps as -possible. +In time, Maneage is going to become more and more mature and robust (thanks +to your feedback and the feedback of other users). Bugs will be fixed and +new/improved features will be added. So every once and a while, you can run +the commands below to pull new work that is done in Maneage. If the changes +are useful for your work, you can merge them with your project to benefit +from them. -The `reproduce/analysis/make/paper.mk` Makefile must be the final Makefile -that is included. This workhorse Makefile ends with the rule to build -`paper.pdf` (final target of the whole project). If you look in it, you -will notice that this Makefile starts with a rule to create -`$(mtexdir)/project.tex` (`mtexdir` is just a shorthand name for -`$(BDIR)/tex/macros` mentioned before). As you see, the only dependency of -`$(mtexdir)/project.tex` is `$(mtexdir)/verify.tex` (which is the last -analysis step: it verifies all the generated results). Therefore, -`$(mtexdir)/project.tex` is _the connection_ between the -processing/analysis steps of the project, and the steps to build the final -PDF. +0. Before going into the technicalities of upgrading your project's +Maneage, it may happen that you don't have the `maneage` branch and +`origin-maneage` remote any more! This can happen when you clone your own +project on a different system, or a colleague clones it to collaborate with +you: the clone won't have the `origin-maneage` remote that you started the +project with. If this is the case, you can add the `origin-maneage` remote +and define the `maneage` branch from it using the steps below: -During the research, it often happens that you want to test a step that is -not a prerequisite of any higher-level operation. In such cases, you can -(temporarily) define that processing as a rule in the most relevant -workhorse-Makefile and set its target as a prerequisite of its TeX -macro. If your test gives a promising result and you want to include it in -your research, set it as prerequisites to other rules and remove it from -the list of prerequisites for TeX macro file. In fact, this is how a -project is designed to grow in this framework. +```shell +$ git remote add origin-maneage https://git.maneage.org/project.git +$ git fetch origin-maneage maneage +$ git checkout -b maneage --track origin-maneage/maneage +``` +1. Pull the latest changes on the `maneage` branch and read the commit +message (full description) of all new changes with the commands below. Just +be sure that you have already committed any changes in your branch +(otherwise the checkout command will fail). +```shell +# Go to the 'maneage' branch and import updates. +$ git checkout maneage +$ git pull # Get recent work in Maneage +``` +2. Read all the commit messages of the newly imported features/changes. In +particular pay close attention to the ones starting with 'IMPORTANT': these +may cause a crash in your project (changing something fundamental in +Maneage). Replace the `XXXXXXX..YYYYYYY` with hashs mentioned close to +start of the `git pull` command outputs (previous step). +```shell +$ git log XXXXXXX..YYYYYYY --reverse +``` -File modification dates (meta data) ------------------------------------ +3. If you find the updates useful, go to your project's `main` branch and +import all the updates into it with the commands below. Don't worry about +the printed outputs (in particular the `CONFLICT`s), we'll clean them up in +the next step. -While Git does an excellent job at keeping a history of the contents of -files, it makes no effort in keeping the file meta data, and in particular -the dates of files. Therefore when you checkout to a different branch, -files that are re-written by Git will have a newer date than the other -project files. However, file dates are important in the current design of -Maneage: Make checks the dates of the prerequisite files and target files -to see if the target should be re-built. +```shell +$ git checkout main +$ git merge maneage -This is relevant, for example, in the `analysis/` side of calculations when -a rule in a `reproduce/analysis/make/SOMETHING.mk` is dependent on -`reproduce/analysis/conf/SOMETHING.conf` (where `SOMETHING` is one of the -science pipeline steps that you have prepared). Switching back and forth -between git branches will update the timestamp on `SOMETHING.conf`, -incorrectly implying, according to the Make rules, that a rule in -SOMETHING.mk has to be run again. +# Ignore conflicting Maneage files that you had previously deleted +# in the customization checklist (mostly demonstration files). +$ git status # Just for a check +$ git status --porcelain | awk '/^DU/{system("git rm "$NF)}' +$ git status # Just for a check +``` -To address this problem, you can use the `touch` command to backdate the -timestamp of files that you want to be considered to be old. For example -with the command below: +4. Files with conflicts will be visible from the output of the last command +above) with the classification `both modified:`. Open one of these files +with your favorite text editor and correct the conflict (placed in between +`<<<<<<<`, `=======` and `>>>>>>>`). Once all conflicts in a file are +removed, the file will be automatically removed from the "Un-merged paths" +of `git status`. So run `git status` after correcting the conflicts of each +file just to make sure things are clean. TIP: If you want the changes in +one file to be only from a special branch (`maneage` or `main`, completely +ignoring changes in the other), use this command: ```shell -$ touch --date=2000-01-01 reproduce/analysis/conf/SOMETHING.conf -$ ls -l reproduce/analysis/conf/SOMETHING.conf # to check +$ git checkout -- ``` -You may want to use a date that is obviously (to you) fake, so that you -don't accidentally believe the date later on. - - +5. Once all the Git conflicts are fixed, it is important to make sure that +"semantic conflicts" (that don't show up in Git, but can potentially break +your project) are also fixed. For example updates to software versions +(their behavior may have changed), or to internal Maneage structure. Hence +read the commit messages of `git log` carefully to see what has changed. In +case you see a commit with `IMPORTANT` in its title, the best way is to +delete your build directory and let the software and project be executed +from scratch. +```shell +$ ./harvester make distclean # will DELETE ALL your build-directory!! +$ ./harvester configure -e +$ ./harvester make +``` +6. Once your final product is created at the end of the previous step and +its contents are what you expect, you are ready to commit the merge. In the +commit message, Explain any conflicts that you fixed. -Summary -------- +```shell +$ git add -u +$ git commit +``` -Based on the explanation above, some major design points you should have in -mind are listed below. +7. When everything is OK, before continuing with your project's work, don't +forget to push both your `main` branch and your updated `maneage` branch to +your remote server. - - Define new `reproduce/analysis/make/XXXXXX.mk` workhorse-Makefile(s) - with good and human-friendly name(s) replacing `XXXXXX`. +```shell +$ git push +$ git push origin maneage +``` - - Add `XXXXXX`, as a new line, to the values in `makesrc` of the top-level - `Makefile`. - - Do not use any constant numbers (or important names like filter names) - in the workhorse-Makefiles or paper's LaTeX source. Define such - constants as logically-grouped, separate configuration-Makefiles in - `reproduce/analysis/config/XXXXX.conf`. Then set this - configuration-Makefiles file as a prerequisite to any rule that uses - the variable defined in it. - - Through any number of intermediate prerequisites, all processing steps - should end in (be a prerequisite of) `$(mtexdir)/verify.tex` (defined in - `reproduce/analysis/make/verify.mk`). `$(mtexdir)/verify.tex` is the sole - dependency of `$(mtexdir)/project.tex`, which is the bridge between the - processing steps and PDF-building steps of the project. +## Citing Maneage +If you use Maneage in your project please cite Akhlaghi et al. 2025, +published in Computing in Science & Engineering, vol. 23, issue 3, +pp. 82-91; +arXiv: [2006.03018](https://arxiv.org/abs/2006.03018); +Bibcode: +[2021CSE....23c..82A](https://scixplorer.org/abs/2021CSE....23c..82A); +DOI:[10.1109/MCSE.2021.3072860](https://doi.org/10.1109/MCSE.2021.3072860). +The default `paper.tex` file contains example citation and how to +acknowledge all the software that are used. +Also, when your paper is published, don't forget to add a notice that you +paper is fully reproducible and add a sentence or paragraph in the end of +the paper shortly describing the concept. This will help spread the word +and encourage other scientists to also manage and publish their projects in +a correctly scientific (reproducible) manner. -Customization checklist -======================= +## Publication checklist -Take the following steps to fully customize Maneage for your research -project. After finishing the list, be sure to run `./project configure` and -`project make` to see if everything works correctly. If you notice anything -missing or any in-correct part (probably a change that has not been -explained here), please let us know to correct it. +Once your project is complete and you are ready to submit/publish the +project, we recommend the following steps to ensure the maximum FAIRness of +all your hard work (Findability, Accessibility, Interoperability, and +Reusability). This list may seem long, and may take a day or so to +complete, but please consider the fact that you have spent months/years on +your project, so it is a very small step in your over-all project! Most of +it is about organizing things that you can do during your project. So its +good to have a look at these from the start of your project. -As described above, the concept of reproducibility (during a project) -heavily relies on [version -control](https://en.wikipedia.org/wiki/Version_control). Currently Maneage -uses Git as its main version control system. If you are not already -familiar with Git, please read the first three chapters of the [ProGit -book](https://git-scm.com/book/en/v2) which provides a wonderful practical -understanding of the basics. You can read later chapters as you get more -advanced in later stages of your work. +As you will notice, when you complete this checklist, your projects source +will be present in multiple places: Zenodo, SoftwareHeritage, arXiv, your +own Git repositories. This is a major advantage of Maneaged(!) projects: +because their source is very small (a few hundred kilobytes), there is +effectively no cost in keeping multiple redundancies on different servers, +just in case one (or more) of them are discontinued in the (near/far) +future. -First custom commit -------------------- + - **Reserve a DOI for your datasets**: There are multiple data servers + that give this functionality, one of the most well known and + (currently!) well-funded is [Zenodo](https://zenodo.org) so we'll focus + on it here. Of course, you can use any other service that provides a + similar functionality. Once you complete these steps, you can start + using/citing your dataset's DOI in the source of your project to + finalize the rest of the points. With Zenodo, you can even use the given + identifier for things like downloading. - 1. **Get this repository and its history** (if you don't already have it): - Arguably the easiest way to start is to clone Maneage and prepare for - your customizations as shown below. After the cloning first you rename - the default `origin` remote server to specify that this is Maneage's - remote server. This will allow you to use the conventional `origin` - name for your own project as shown in the next steps. Second, you will - create and go into the conventional `main` branch to start - committing in your project later. + * *Start new upload*: After you log in to Zenodo, you can start a new + upload by clicking on the "New Upload button". - ```shell - $ git clone https://git.maneage.org/project.git # Clone/copy the project and its history. - $ mv project my-project # Change the name to your project's name. - $ cd my-project # Go into the cloned directory. - $ git remote rename origin origin-maneage # Rename current/only remote to "origin-maneage". - $ git checkout -b main # Create and enter your own "main" branch. - $ pwd # Just to confirm where you are. - ``` + * *Reserve DOI*: Under the "Basic information" --> "Digital Object + Identifier", click on the "Reserve DOI" button. - 2. The final job of Maneage is to create your paper's PDF. By default it - uses a custom LaTeX style that resembles that of the Astrophysical - Journal (because the precusor of Maneage was for [Akhlaghi & Ichikawa - 2015](https://ui.adsabs.harvard.edu/abs/2015ApJS..220....1A)). The - journal you plan to submit your paper to will have its own separate - style. So it is best that you start your project by writing in the - desired style. We have already customized Maneage for the official - styles of some journals. To find them, run `git branch -r | grep - journal`. If your planned journal is one of them, you can take the - following steps to start your project based on that journal's style. If - it is not in these, you can ignore this step for now and customize the - style later (you can model based on these branchs). In the commands - below, we'll assume you want to prepare for the Astronomy and - Astrophysics journal (A&A). + * *Fill basic info*: You need to at least fill in the "required fields" + (marked with a red star). You will always be able to change any + metadata (even after you "Publish"), so don't worry too much about + values in the fields, at this phase, its just important that they are + not empty. - ```shell - $ git checkout -b journal origin-maneage/journal-a-and-a - $ git log -1 --oneline | awk '{print $1}' # To keep the commit hash - $ git rebase -i main # See description below - ``` + * *Save your project but do not yet publish*: Press the "Save" button + (at the top or bottom of the page). Do not yet press "Publish" though, + since that would make the project public, and freeze the DOI with any + possible file you may have uploaded already. We will get to the + publication phase in the next steps. - In the first text editor that opens after the last command, change all - (except the first) `pick`s into `squash`, then save the change and - close the editor. In case there is no conflict, the second editor will - be pre-filled with all the commit messages in that branch. You do not - need those, so you can delete everything and write a commit message - like the following: `A&A journal (commit XXXXX of Maneage's - journal-a-and-a branch)`. Just replace the `XXXXX` with the output of - the second command above. The commit hash is important to be stored - here since it allows you to later check if any updates have been made - in that branch in the future. After completing the git rebase operation - (last command above), run the following commands below to put the new - commit in your `main` branch (and continue working based on that). + - **Record the metadata**: Maneage comes with a file to store all the + project's metadata: `reproduce/analysis/config/metadata.conf`. Open this + file and store all the information that you currently have: for example + the Zenodo DOI, project's Git repository, Copyright owner and license of + the data after it becomes public. Keep the empty fields in mind and + after obtaining them, don't forget to fill them up. - ```shell - $ git checkout main - $ git merge journal - $ git branch -D journal - ``` + - **Request archival on SoftwareHeritage**: [Software + Heritage](https://archive.softwareheritage.org/save/) is an online + project to archive source code and their development histories. It + provides wonderful features for archiving source code (not data!) and + also for citing special parts of a project's source in any point of its + history. So it blends elegantly with the purpose of Maneage. Once you + make your project's Git repository publicly accessible (no login + required to clone it), you can request that SoftwareHeritage archives + it. Its good if you do this as soon as you make your Git repository + public. When you are ready, just register your repository's address (the + same one you give to `git clone`) to in [SoftwareHeritage's save + form](https://archive.softwareheritage.org/save). - 3. **Prepare to build project**: The `./project configure` command of the - next step will build the different software packages within the - "build" directory (that you will specify). Nothing else on your system - will be touched. However, since it takes long, it is useful to see - what it is being built at every instant (its almost impossible to tell - from the torrent of commands that are produced!). So open another - terminal on your desktop and navigate to the same project directory - that you cloned (output of last command above). Then run the following - command. Once every second, this command will just print the date - (possibly followed by a non-existent directory notice). But as soon as - the next step starts building software, you'll see the names of - software get printed as they are being built. Once any software is - installed in the project build directory it will be removed. Again, - don't worry, nothing will be installed outside the build directory. + - **Run a spell-check on `paper.tex`**: we all forget ;-)! - ```shell - # On another terminal (go to top project source directory, last command above) - $ ./project --check-config - ``` + - **Zenodo/SoftwareHeritage links in paper**: put links to the Zenodo-DOI + (and SoftwareHeritage source when you make it public) in your + paper. Somewhere close to the start, maybe under the keywords/abstract, + highlighting that they are supplements for reproducibility. These help + readers easily access these resources for supplementary material + directly from your PDF paper (sources on SoftwareHeritage and + data/software on Zenodo). These links are more trusted/reliable in terms + of longevity than Git repositories or private web pages. - 4. **Test Maneage**: Before making any changes, it is important to test it - and see if everything works properly with the commands below. If there - is any problem in the `./project configure` or `./project make` steps, - please contact us to fix the problem before continuing. Since the - building of dependencies in configuration can take long, you can take - the next few steps (editing the files) while its working (they don't - affect the configuration). After `./project make` is finished, open - `paper.pdf`. If it looks fine, you are ready to start customizing the - Maneage for your project. But before that, clean all the extra Maneage - outputs with `make clean` as shown below. + - **Identify and properly format output data**: If you have a plot, figure + or table in your paper, you need to verify that data later and publish + that data with the paper (see the steps below for both). But before + going to those steps, its good if you polish your datasets with the + recommendations below: - ```shell - $ ./project configure # Build the project's software environment (can take an hour or so). - $ ./project make # Do the processing and build paper (just a simple demo). + * *Keep published data in a special place*: it helps if you keep the + to-be-published data files in a special sub-directory under your build + directory. In this way, irrespective of which subMakefile builds a + published dataset, they won't be lost/scattered in the middle of all + the project's intermediate-built files. - # Open 'paper.pdf' and see if everything is ok. - ``` + * *In plain-text*: If the data are in tabular form (for example the X + and Y values in your plots), store them as a simple plain-text file + (for example with columns separated by white-space characters) or in + the more formal [Comma-separated + values](https://en.wikipedia.org/wiki/Comma-separated_values) or CSV, + format). Generally, its best to set the suffixes to `.txt` (because + most browsers/OSs will automatically know they are plain-text and open + them without needing any other software). If you have other types of + data (for example images, or very large tables with millions of + rows/columns that can be inconvenient in plain-text), feel free to use + custom binary formats, but later, in the description of your project + on the server, add a note, explaining what software they should use to + open them. - 5. **Setup the remote**: You can use any [hosting - facility](https://en.wikipedia.org/wiki/Comparison_of_source_code_hosting_facilities) - that supports Git to keep an online copy of your project's version - controlled history. We recommend [GitLab](https://gitlab.com) because - it is [more ethical (although not - perfect)](https://www.gnu.org/software/repo-criteria-evaluation.html), - and later you can also host GitLab on your own server. Anyway, create - an account in your favorite hosting facility (if you don't already - have one), and define a new project there. Please make sure *the newly - created project is empty* (some services ask to include a `README` in - a new project which is bad in this scenario, and will not allow you to - push to it). It will give you a URL (usually starting with `git@` and - ending in `.git`), put this URL in place of `XXXXXXXXXX` in the first - command below. With the second command, "push" your `main` branch to - your `origin` remote, and (with the `--set-upstream` option) set them - to track/follow each other. However, the `maneage` branch is currently - tracking/following your `origin-maneage` remote (automatically set - when you cloned Maneage). So when pushing the `maneage` branch to your - `origin` remote, you _shouldn't_ use `--set-upstream`. With the last - command, you can actually check this (which local and remote branches - are tracking each other). + * *Descriptive names*: In some papers there are many files and having + cryptic names will only confuse your readers (actually, yourself in + two years!). So set the names of the files to be as descriptive as + possible, so simply by reading the name of the file, someone who has + read the paper will understand what figure it corresponds to. In + particular, don't set names like `figure-3.txt`! In a few months you + will forget the order of the figures! Even worse, after the referee + report, you may need to re-arrange some figures and you will be forced + to rename everything related to each figure (which is very frustrating + and prone to errors). - ```shell - git remote add origin XXXXXXXXXX # Newly created repo is now called 'origin'. - git push --set-upstream origin main # Push 'main' branch to 'origin' (with tracking). - git push origin maneage # Push 'maneage' branch to 'origin' (no tracking). - ``` + * *Good metadata*: Raw data are not too useful merely as a series of raw + numbers! So don't forget to have **good metadata in every file**. If + its a plain-text file, usually lines starting with a `#` are + ignored. So in the command that generates each dataset, add some extra + information (the more the better!) about the dataset as lines starting + with `#`. Based on `reproduce/analysis/config/metadata.conf`, in + `initialize.mk`, Maneage will produce a default set of basic + information for plain-text data and will put it in the + `$(print-general-metadata)` variable. It is thus recommended to print + this variable into your plain-text file before printing the actual + data (so it shows on top of the file). For a real-world example, see + its usage in `reproduce/analysis/make/delete-me.mk` (in the `maneage` + branch). If you are publishing your data in binary formats, please add + all the metadata you see in `$(print-general-metadata)` into each + dataset file (for example keywords in the FITS format). If there are + many files, its easy to define a tiny shell-script to do the job on + each dataset. - 6. **Title**, **short description** and **author**: You can start adding - your name (with your possible coauthors) and tentative abstract in - `paper.tex`. You should see the relevant place in the preamble (prior - to `\begin{document}`. Just note that some core project metadata like - the project title are actually set in - `reproduce/analysis/config/metadata.conf`. So set your project title - in there. After you are done, run the `./project make` command again - to see your changes in the final PDF and make sure that your changes - don't cause a crash in LaTeX. Of course, if you use a different LaTeX - package/style for managing the title and authors (in particular a - specific journal's style), please feel free to use it your own methods - after finishing this checklist and doing your first commit. + - **Link to figure datasets in caption**: all the datasets that go into + the plots should be uploaded directly to Zenodo so they can be + viewed/downloaded with a simple link in the caption. For example see the + last sentence of the caption of Figure 1 in + [Akhlaghi+2021](https://scixplorer.org/abs/2021CSE....23c..82A), it + points to [the + data](https://zenodo.org/record/3872248/files/tools-per-year.txt) that + was used to create that figure's left-side plot. A more complete example + is [Eskandarlou+2026](https://scixplorer.org/abs/2026A&A...709A.210E); + where the underlying data of all figures are linked in Appendix E. As + you see, this will allow your paper's readers (again, most probably your + future-self!) to directly access the numbers of each visualization + (plot/figure) with a simple click in a trusted server. This also shows + the major advantage of having your data as simple plain-text where + possible, as described above. To help you keep all your to-be-visualized + datasets in a single place, Maneage has the two `figdir` (for + figures/plots) and `drdir` (for data release) directories that are + defined in `reproduce/analysis/make/initialize.mk`, see the comments + above their definition for more. Upon publication, you can just publish + the contents of these two directories. Therefore, be sure to only keep + the actual data in these directories, not intermediate files. - 7. **Delete dummy parts**: Maneage contains some parts that are only for - the initial/test run, mainly as a demonstration of important steps, - which you can use as a reference to use in your own project. But they - not for any real analysis, so you should remove these parts as - described below: + - **Verification step**: It is very important to automatically verify the + outputs of your project. Recall from the customization checklist (above) + that you can activate verification by setting the `verify-outputs` + variable to `yes` in `reproduce/analysis/config/verify-outputs.conf`. So + please activate it and look into the `reproduce/analysis/make/verify.mk` + to add the necessary steps to automatically verify your outputs. *Tip*: + you don't have to generate the checksums manually, just give a wrong + value (for example `XXXX`) so Maneage crashes! In the error message it + will then print the actual and expected checksums and you can take the + value from there. Outputs that must be verified can be listed as: - - `paper.tex`: 1) Delete the text of the abstract (from - `\includeabstract{` to `\vspace{0.25cm}`) and write your own (a - single sentence can be enough now, you can complete it later). 2) - Add some keywords under it in the keywords part. 3) Delete - everything between `%% Start of main body.` and `%% End of main - body.`. 4) Remove the notice in the "Acknowledgments" section (in - `\new{}`) and Acknowledge your funding sources (this can also be - done later). Just don't delete the existing acknowledgment - statement: Maneage is possible thanks to funding from several - grants. Since Maneage is being used in your work, it is necessary to - acknowledge them in your work also. + * *subMakefile LaTeX macro files*: these LaTeX macros put numbers into + the text. You don't want your readers (actually: yourself in two + years!) to have to painfully find and check, by eye, all those tiny + numbers buried deep in the ocean of words! - - `reproduce/analysis/make/top-make.mk`: Delete the `delete-me` line - in the `makesrc` definition. Just make sure there is no empty line - between the `initialize \` and `verify \` lines (they should be - directly under each other). + * *Final data files* (for tables, figures, or plots, or as data + release). These are the same files described above. If you have + followed the guidelines above and stored them as plain-text with + comments on top, you can use the provided function + `verify-txt-no-comments-leading-space` which takes the filename and + checksum as arguments to avoid the commented lines (which may change) + and only verify the data. If your data are in other formats, be sure + to verify them without metadata that may change (like date and etc). - - `reproduce/analysis/make/initialize.mk`: in the very end of this - file, you will see a set of lines between `delete the lines below - this` and `delete the lines above this`. Delete this whole group of - lines (including the two instruction lines). + - **Fill `README.md`**: The `README.md` is *the first place* your readers + are going to look into. It already has a default text with place-holders + in the form of `XXXXXX`. Please go through its first few paragraphs and + replace the place-holders with the relevant information/links or feel + free to add/remove anything else. The rest is just basic information + that is useful for any Maneage'd project. Just don't forget to tell your + readers in `README.md` that they can learn about this system in the + `README-hacking.md` file (ideally close to the top). - - `reproduce/analysis/config/verify-outputs.conf`: Disable - verification of outputs by changing the `yes` (the value of - `verify-outputs`) to `no`. Later, when you are ready to submit your - paper, or publish the dataset, activate verification and make the - proper corrections in this file (described under the "Other basic - customizations" section below). This is a critical step and only - takes a few minutes when your project is finished. So DON'T FORGET - to activate it in the end. + - **Confirm if your project builds from scratch**: Before publishing + anything, you should see if your project can indeed reproduce itself! + You may be mistakenly using temporarily created files that aren't built + when the project is built from scratch (this happens a lot and is very + dangerous for the integrity of your project!). So, go to a temporary + directory, clone your project from its repository and try configuring + and building it from scratch in a new-temporary build-directory. It is + important to ignore the original directory you developed your project on + (source and build): you may have files there that you forgot to import + into Git or depended on in the build (it happens!). Ideally, it would be + good to try it on a different computer. - - Delete all `delete-me*` files in the following directories: + - **Confirm if `./harvester make dist` works**: The special target `dist` + tells the project to build a tarball that is ready to compile the LaTeX + PDF without having to do the analysis and build software. This is very + useful for servers like arXiv, or some journals. This tarball is also + one of the deliverables you want to publish on Zenodo. Once the tarball + is created, copy it to a temporary directory outside of Maneage, unpack + it and run `make` (completely ignoring Maneage's `./harvester` + script). If you plan to submit your paper to arXiv, the best test is to + actually start a test submission on arXiv to upload the tarball there to + see if it can build your PDF. Once it works, you can delete that + temporary submission for now. Afterwards, try configuring and building + it with the tarball by running its `./harvester` (from scratch and + without the Git history!). If there is a problem in any of these tests, + you can modify what goes into this tarball in + `reproduce/analysis/make/initialize.mk`: go through the steps and add + the necessary components until the checks pass. - ```shell - $ rm tex/src/delete-me* - $ rm reproduce/analysis/make/delete-me* - $ rm reproduce/analysis/config/delete-me* - ``` + - **Upload all deliverables to Zenodo**: With the datasets ready, you can + now upload the following deliverables to Zenodo. Except for the data + files, put the Git hash of your Maneaged project at the moment of + publication in the filename of other uploaded files. The output files + shouldn't have a hash in their names because their URL (that goes in the + caption of the figures/tables) should be known prior to a commit, + creating a cyclic dependency! Ideally the hash should be placed just + before the final suffix, for example `paper-XXXXXXX.pdf` (where + `XXXXXXX` is the Git hash). This will clearly identify the point in + history that your file was created. - - Re-make the project (after a cleaning) to see if you haven't - introduced any errors. + * **paper-XXXXXXX.pdf**: you shouldn't just download data to the data + server, also upload your paper's PDF so its there with the other raw + formats. It will greatly help yourself and others. Most data centers + (like Zenodo) actually also have a PDF viewer that will load + automatically before the list of data files. For example see + [zenodo.3408481](https://doi.org/10.5281/zenodo.3408481). + * **`project-XXXXXXX.tar.gz`**: Or the output of `make dist` as + described above. + + * **`project-git.bundle`** This is the full Git history of the project + in one file (which you can actually clone from later!). Its + necessary to publish this with your dataset too because Git + repositories make no promise on longevity. The way to "bundle" a Git + history is described below, in summary, its this command: ```shell - $ ./project make clean - $ ./project make + $ git bundle create my-project-git.bundle --all ``` - 8. **Ignore changes in some Maneage files**: One of the main advantages of - Maneage is that you can later update your infra-structure by merging - your `main` branch with the `maneage` branch. This is good for many - low-level features that you will likely never modify yourself. But it - is not desired for some files like `paper.tex` (you don't want changes - in Maneage's default `paper.tex` to cause conflicts with all the text - you have already written for your project). You need to tell Git to - ignore changes in such files from the `maneage` branch during the - merge, and just keep your own branch's version. The commands below - show how you can avert such future conflicts and frustrations with - some known files. Note that only the first `echo` command has a `>` - (to write over the file), the rest are `>>` (to append to it). If you - want to avoid any other set of files to be imported from Maneage into - your project's branch, you can follow a similar strategy (it should - happen rarely, if at all!). Generally be very careful about adding - files to `.gitattributes` because it affects the whole file and if a - wrong file is ignored, Maneage may break after a merge (some - inter-dependent files may not get updated together). We recommend only - doing it when you encounter the same conflict in more than one merge, - and are sure that it won't affect other files. In such cases please - let us know so we can improve the design of Maneage and modularize - those components to be easily added here. + * **`software-XXXXXXX.tar.gz`**: This is effectively a copy of all the + software source code tarballs in your project's + `.build/software/tarballs`. It is necessary to upload these with + your project to avoid relying on third party servers. In the future + any one of those servers may go down and if so, your project won't + be buildable. You can generate this tarball easily with `./harvester + make dist-software`. - ```shell - $ echo "paper.tex merge=ours" > .gitattributes - $ echo "tex/src/*.tex merge=ours" >> .gitattributes - $ echo "reproduce/analysis/config/*.conf merge=ours" >> .gitattributes - $ echo "reproduce/software/config/TARGETS.conf merge=ours" >> .gitattributes - $ echo "reproduce/software/config/texlive-packages.conf merge=ours" >> .gitattributes - $ git add .gitattributes - ``` + * All the figure (and other) output datasets of the project. Don't + rename these files, let them have the same descriptive name + mentioned above. Also recall that a link to all these files is also + in the caption of the respective figure. - 9. **Copyright and License notice**: It is necessary that _all_ the - "copyright-able" files in your project (those larger than 10 lines) - have a copyright and license notice. Please take a moment to look at - several existing files to see a few examples. The copyright notice is - usually close to the start of the file, it is the line starting with - `Copyright (C)` and containing a year and the author's name (like the - examples below). The License notice is a short description of the - copyright license, usually one or two paragraphs with a URL to the - full license. Don't forget to add these _two_ notices to *any new - file* you add in your project (you can just copy-and-paste). When you - modify an existing Maneage file (which already has the notices), just - add a copyright notice in your name under the existing one(s), like - the line with capital letters below. To start with, add this line with - your name and email address to `paper.tex`, - `tex/src/preamble-project.tex`, `reproduce/analysis/make/top-make.mk`, - and generally, all the files you modified in the previous step. + - **Upload to [arXiv](https://arxiv.org)**: or to any other pre-print + server (if you want to). Of course, you can also do this after the + initial/final submission to your desired journal. But we'll just add the + necessary points for arXiv submission here: - ``` - Copyright (C) 2018-2026 Existing Name - Copyright (C) 2026-2026 YOUR NAME - ``` + * *Necessary links in comments*: put a link to your project's Git + repository, Zenodo-DOI (this is not your paper's DOI, its the + data/resources DOI), and/or SoftwareHeritage link in the comments. - 10. **Configure Git for fist time**: If this is the first time you are - running Git on this system, then you have to configure it with some - basic information in order to have essential information in the commit - messages (ignore this step if you have already done it). Git will - include your name and e-mail address information in each commit. You - can also specify your favorite text editor for making the commit - (`emacs`, `vim`, `nano`, and etc.). + - *Update `metadata.conf`*: Once you have your final arXiv ID (formatted + as: `1234.56789`) put it in `reproduce/analysis/config/metadata.conf`. - ```shell - $ git config --global user.name "YourName YourSurname" - $ git config --global user.email your-email@example.com - $ git config --global core.editor nano - ``` + - **Submission to a journal**: different journals accept submissions in + different formats, some accept LaTeX, some only want a PDF, or etc. It + would be good if you highlight in the cover-letter that your work is + reproducible and provide the Zenodo and Software Heritage links (if they + are public). If not, you can mention that everything is ready for such a + submission after acceptance. - 11. **Your first commit**: You have already made some small and basic - changes in the steps above and you are in your project's `main` - branch. So, you can officially make your first commit in your - project's history and push it. But before that, you need to make sure - that there are no problems in the project. This is a good habit to - always re-build the system before a commit to be sure it works as - expected. + * Some journals (like the "Astronomy and Astrophysics" journal) + require that only a single LaTeX file be submitted without any + commens (along with all the figures). In other words, they do not + want any sub-directories: only a single LaTeX file, one `.bbl` file + and one PDF per figure. Here is how you can do this: + ```shell + # If you don't already have 'latexpand' in your OS, install it: + $ tlmgr install latexpand - ```shell - $ git status # See which files you have changed. - $ git diff # Check the lines you have added/changed. - $ ./project make # Make sure everything builds successfully. - $ git add -u # Put all tracked changes in staging area. - $ git status # Make sure everything is fine. - $ git diff --cached # Confirm all the changes that will be committed. - $ git commit # Your first commit: put a good description! - $ git push # Push your commit to your remote. - ``` + # Unpack the tarball and make a directory for a single-LaTeX source. + $ tar -xf maneaged-XXXXXXX.tar.gz + $ cd maneaged-XXXXXXX.tar.gz + $ mkdir one-latex - 12. **Read the publication checklist**: The publication checklist below is - very similar to this one, but for the final phase of your project. For - now, you don't have to do any of its steps, but reading it will give - you good insight into the later stages of your project. If you already - know how you want to publish your project, you can implement many of - those steps from the start and during the actual project (in - particular how to organize your data files that go into the plots). - Making it much easier to complete that checklist when you are ready - for submission. + ## The '--empty-comments' and subsequent call to AWK (that removes + ## lines starting with '%') is because we need the lines ending in a + ## '%' for some LaTeX macros). + $ latexpand paper.tex --output=one-latex/paper-0.tex \ + --empty-comments + $ awk '!/^%/' one-latex/paper-0.tex > one-latex/paper.tex + $ rm one-latex/paper-0.tex - 13. **Start your exciting research**: You are now ready to add flesh and - blood to this raw skeleton by further modifying and adding your - exciting research steps. You can use the "published works" section in - the introduction (above) as some fully working models to learn - from. Also, don't hesitate to contact us if you have any - questions. + ## Copy the other necessary files (if you are using specific journal + ## style files, also copy them). + $ cp paper.bbl one-latex/ + $ cp tex/tikz/*.pdf one-latex/ + ## Correct the location of the images. + $ sed -i -e's|tex/tikz/|./|' one-latex/paper.tex + ``` -Other basic customizations --------------------------- + * You can confirm the contents of the `one-latex` directory made + above by running `pdflatex paper` inside that directory. Your + operating system's LaTeX may not have all the necessary packages, + in that case run `tlmgr install XXXXX` (where `XXXXX` is the + value of `texlive-packages` in + `reproduce/software/config/texlive-packages.conf`. Afterwards, + clean up all the temporary files with this command: + ```shell + $ rm *.log *.aux *.auxlock *.bcf *.out *.xml + ``` - - **High-level software**: Maneage installs all the software that your - project needs. You can specify which software your project needs in - `reproduce/software/config/TARGETS.conf`. The necessary software are - classified into two classes: 1) programs or libraries (usually written - in C/C++) which are run directly by the operating system. 2) Python - modules/libraries that are run within Python. By default - `TARGETS.conf` only has GNU Astronomy Utilities (Gnuastro) as one - scientific program and Astropy as one scientific Python module. Both - have many dependencies which will be installed into your project - during the configuration step. To see a list of software that are - currently ready to be built in Maneage, see - `reproduce/software/config/versions.conf` (which has their versions - also), the comments in `TARGETS.conf` describe how to use the software - name from `versions.conf`. Currently the raw pipeline just uses - Gnuastro to make the demonstration plots. Therefore if you don't need - Gnuastro, go through the analysis steps in `reproduce/analysis` and - remove all its use cases (clearly marked). + * Upload the contents of this `one-latex` directory to the journal. - - **Input datasets**: The input datasets are managed through the - `reproduce/analysis/config/INPUTS.conf` file. It is best to gather the - following information regarding all the input datasets into this one - central file: 1) the SHA256 checksum of the file, 2) the URL where the - file can be downloaded online. Please read the comments at the start - of `reproduce/analysis/config/INPUTS.conf` carefully. + - **Future versions**: Both Zenodo and arXiv allow uploading new versions + after your first publication. So it is recommended to put more recent + versions of your published projects later (for example after applying + the changes suggested by the referee). In Zenodo (unlike arXiv), you + only need to publish a new version if the uploaded files have + changed. You can always update the project's metadata with no effect on + the DOI (so you don't need to upload a new version if you just want to + update the metadata). - - **`README.md`**: Correct all the `XXXXX` place holders (name of your - project, your own name, address of your project's online/remote - repository, link to download dependencies and etc). Generally, read - over the text and update it where necessary to fit your project. Don't - forget that this is the first file that is displayed on your online - repository and also your colleagues will first be drawn to read this - file. Therefore, make it as easy as possible for them to start - with. Also check and update this file one last time when you are ready - to publish your project's paper/source. + - **After acceptance (before publication)**: Congratulations on the + acceptance! The main science content of your paper can't be changed any + more, but the paper will now go to the publication editor (for language + and style). Your approval of the final proof is necessary before the + paper is finally published. Use this period to finalize the final + metadata of your project: the journal's DOI. Some journals associate + your paper's DOI during this process. So before approving the final + proof do these steps: - - **Verify outputs**: During the initial customization checklist, you - disabled verification. This is natural because during the project you - need to make changes all the time and its a waste of time to enable - verification every time. But at significant moments of the project - (for example before submission to a journal, or publication) it is - necessary. When you activate verification, before building the paper, - all the specified datasets will be compared with their respective - checksum and if any file's checksum is different from the one recorded - in the project, it will stop and print the problematic file and its - expected and calculated checksums. First set the value of - `verify-outputs` variable in - `reproduce/analysis/config/verify-outputs.conf` to `yes`. Then go to - `reproduce/analysis/make/verify.mk`. The verification of all the files - is only done in one recipe. First the files that go into the - plots/figures are checked, then the LaTeX macros. Validation of the - former (inputs to plots/figures) should be done manually. If its the - first time you are doing this, you can see two examples of the dummy - steps (with `delete-me`, you can use them if you like). These two - examples should be removed before you can run the project. For the - latter, you just have to update the checksums. The important thing to - consider is that a simple checksum can be problematic because some - file generators print their run-time date in the file (for example as - commented lines in a text table). When checking text files, this - Makefile already has this function: - `verify-txt-no-comments-leading-space`. As the name suggests, it will - remove comment lines and empty lines before calculating the MD5 - checksum. For FITS formats (common in astronomy, fortunately there is - a `DATASUM` definition which will return the checksum independent of - the headers. You can use the provided function(s), or define one for - your special formats. + * Add the Journal DOI in `reproduce/analysis/config/metadata.conf`, + and re-build your final data products, so this important metadata is + added. - - **Feedback**: As you use Maneage you will notice many things that if - implemented from the start would have been very useful for your - work. This can be in the actual scripting and architecture of Maneage, - or useful implementation and usage tips, like those below. In any - case, please share your thoughts and suggestions with us, so we can - add them here for everyone's benefit. + * Once you get the final proof, and if everything is OK for you, + implement all the good language corrections/edits they have made + inside your own copy here and commit it into your project. This will + be the final commit of your project before publication. - - **Re-preparation**: Automatic preparation is only run in the first run - of the project on a system, to re-do the preparation you have to use - the option below. Here is the reason for this: when its necessary, the - preparation process can be slow and will unnecessarily slow down the - whole project while the project is under development (focus is on the - analysis that is done after preparation). Because of this, preparation - will be done automatically for the first time that the project is run - (when `.build/software/preparation-done.mk` doesn't exist). After the - preparation process completes once, future runs of `./project make` - will not do the preparation process anymore (will not call - `top-prepare.mk`). They will only call `top-make.mk` for the - analysis. To manually invoke the preparation process after the first - attempt, the `./project make` script should be run with the - `--prepare-redo` option, or you can delete the special file above. + * Submit your final project as a new version to Zenodo (and + arXiv). The Zenodo one is most important because your plots will + link to it and you want the commit hash in the data files that + readers will get from Zenodo to be the same hash as the paper. - ```shell - $ ./project make --prepare-redo - ``` + * Tell the journal's publication editor to correct the hash and Zenodo + ID in your final proof confirmation (so the links point to the + correct place). Recall that on every new version upload in Zenodo, + you get a new DOI (or Zenodo ID). - - **Pre-publication**: add notice on reproducibility**: Add a notice - somewhere prominent in the first page within your paper, informing the - reader that your research is fully reproducible. For example in the - end of the abstract, or under the keywords with a title like - "reproducible paper". This will encourage them to publish their own - works in this manner also and also will help spread the word. @@ -1063,817 +953,1102 @@ Other basic customizations +# Maneage development tools -Upgrading the Maneage branch of your project -============================================ +Maneage does not define a new programming language. By design, it uses +existing and time-tested tools/software. In this section, we review the +purpose of each tool and include recommendations on how to learn them. The +"Published works using Maneage" section below lists published papers that +have been written within Maneage. Once you have a good knowledge of the +necessary tools listed below, you can try reading the source of previously +published papers as demos to help in the optimal design your own project. -In time, Maneage is going to become more and more mature and robust (thanks -to your feedback and the feedback of other users). Bugs will be fixed and -new/improved features will be added. So every once and a while, you can run -the commands below to pull new work that is done in Maneage. If the changes -are useful for your work, you can merge them with your project to benefit -from them. -0. Before going into the technicalities of upgrading your project's -Maneage, it may happen that you don't have the `maneage` branch and -`origin-maneage` remote any more! This can happen when you clone your own -project on a different system, or a colleague clones it to collaborate with -you: the clone won't have the `origin-maneage` remote that you started the -project with. If this is the case, you can add the `origin-maneage` remote -and define the `maneage` branch from it using the steps below: -```shell -$ git remote add origin-maneage https://git.maneage.org/project.git -$ git fetch origin-maneage maneage -$ git checkout -b maneage --track origin-maneage/maneage -``` +### GNU Info -1. Pull the latest changes on the `maneage` branch and read the commit -message (full description) of all new changes with the commands below. Just -be sure that you have already committed any changes in your branch -(otherwise the checkout command will fail). +The distinguishing feature of a good developer is not memorizing more of a +tool's features, but to know where to look (for the details) when they need +something. Documentation is therefore very important in any development +environment. As part of Maneage's software environment configuration, we +also install any documentation that each software installs (mostly in the +Info documentation format). This includes many of the basic software in +Maneage like Make, AWK, SED, Coreutils (that provides commands such as +`ls`, `cp` and etc), as well as high-level software like GNU Scientific +Library (GSL) and GNU Astronomy Utilities (Gnuastro). -```shell -# Go to the 'maneage' branch and import updates. -$ git checkout maneage -$ git pull # Get recent work in Maneage -``` +Info greatly simplifies your access (without taking your hands off the +keyboard!) to many manuals that are installed in your Maneage'd software +environment, allowing you to be much more efficient as you work. Another +nice thing with the Info documentation is that it corresponds the same +version that you are using in your analysis and you do not need any +internet connection. -2. Read all the commit messages of the newly imported features/changes. In -particular pay close attention to the ones starting with 'IMPORTANT': these -may cause a crash in your project (changing something fundamental in -Maneage). Replace the `XXXXXXX..YYYYYYY` with hashs mentioned close to -start of the `git pull` command outputs (prevoius step). +If you are not familiar with this documentation format, the good news is +that it has a very nice manual for itself that you can easily enter with +the second command below. The first command takes you into the interactive +Maneage'd software environment. In case you have the software environment +within a container, include the `--sif` option (just like the `./harvester +make` command). ```shell -$ git log XXXXXXX..YYYYYYY --reverse +$ ./harvester shell +$ info info ``` -3. If you find the updates useful, go to your project's `main` branch and -import all the updates into it with the commands below. Don't worry about -the printed outputs (in particular the `CONFLICT`s), we'll clean them up in -the next step. +We strongly recommend reading this manual: it will open a world of +knowledge for you that is always available, even when you are offline. In +less than an hour, you will become familiar and proficient in the Info +documentation. You can then use Info when reading the documentation of +other software you may want to learn when using Maneage (like Make, AWK, +SED or Gnuastro), and you'll see how powerful and convenient it is. -```shell -$ git checkout main -$ git merge maneage +If you use the GNU Emacs text editor (or any of its variants), you also +have access to all Info manuals while you are writing your code (again, +without taking your hands off the keyboard!): simply run `Ctrl-h i` inside +Emacs. Emacs (along with Vim) is available within the Harvester software +environment (which you can access through `./harvester shell`). -# Ignore conflicting Maneage files that you had previously deleted -# in the customization checklist (mostly demonstration files). -$ git status # Just for a check -$ git status --porcelain | awk '/^DU/{system("git rm "$NF)}' -$ git status # Just for a check -``` -4. Files with conflicts will be visible from the output of the last command -above) with the classification `both modified:`. Open one of these files -with your favorite text editor and correct the conflict (placed in between -`<<<<<<<`, `=======` and `>>>>>>>`). Once all conflicts in a file are -removed, the file will be automatically removed from the "Unmerged paths" -of `git status`. So run `git status` after correcting the conflicts of each -file just to make sure things are clean. TIP: If you want the changes in -one file to be only from a special branch (`maneage` or `main`, completely -ignoring changes in the other), use this command: -```shell -$ git checkout -- -``` +### GNU Make -5. Once all the Git conflicts are fixed, it is important to make sure that -"semantic conflicts" (that don't show up in Git, but can potentially break -your project) are also fixed. For example updates to software versions -(their behavior may have changed), or to internal Maneage structure. Hence -read the commit messages of `git log` carefully to see what has changed. In -case you see a commit with `IMPORTANT` in its title, the best way is to -delete your build directory and let the software and project be executed -from scratch. +When batch processing is necessary (no manual intervention, as in a +reproducible project), shell (or Python) scripts are usually the first +solution for most people. However, the inherent complexity and +non-linearity of progress in a scientific project (where experimentation is +key) make it hard to manage them as the project evolves. For example, a +script will start from the top/start every time it is run. So if you have +already completed 90% of the project and want to run the remaining 10% that +you have newly added, you have to run the whole script from the start +again. -```shell -$ ./project make distclean # will DELETE ALL your build-directory!! -$ ./project configure -e -$ ./project make -``` +It is possible to manually ignore/comment parts of a script to only do a +special part. However, such checks/comments will only add to the complexity +of the script and will discourage you from experimenting on an already +completed part of the project when an idea suddenly comes up. It is also +prone to very serious bugs in the end (when trying to reproduce from +scratch); for example you forget to un-comment or remove a temporary +change. Such bugs are very hard to notice during the work and frustrating +to find in the end. -6. Once your final product is created at the end of the previous step and -its contents are what you expect, you are ready to commit the merge. In the -commit message, Explain any conflicts that you fixed. +Make is the solution to these problems. It starts from the end: the final +*target*. Dependencies and recipes (shell scripts) are defined for every +target and using them, Make builds a dependency tree internally. Using +that, Make finds where it should start each time the project is +run. Therefore, in the scenario above, a researcher that has just added the +final 10% of the project, will only have to run those extra steps: without +any modification to previously executed code. With Make, it is also trivial +to change the processing of any intermediate (already written) *rule* (or +step) in the middle of an already written analysis: the next time Make is +run, only rules that are affected by the changes/additions will be re-run, +not the whole analysis/project. -```shell -$ git add -u -$ git commit -``` +This greatly speeds up the processing (enabling creative changes), while +keeping all the dependencies clearly documented (as part of the Make +language), and most importantly, enabling full reproducibility from scratch +with no changes in the project code that was working during the +research. This will allow robust results and let the scientists get to what +they do best: experiment and be critical to the methods/analysis without +having to waste energy and time on technical problems that come up as a +result of using scripts alone. -7. When everything is OK, before continuing with your project's work, don't -forget to push both your `main` branch and your updated `maneage` branch to -your remote server. +Furthermore, since the dependencies are clearly demarcated in Make, it can +identify independent steps and run them in parallel. This further speeds up +the processing. Make was designed for this purpose. It is how huge projects +like all Unix-like operating systems (including GNU/Linux or Mac OS +operating systems) and their core components are built. Therefore, Make is +a highly mature paradigm/system with robust and highly efficient +implementations in various operating systems perfectly suited for a complex +non-linear research project. -```shell -$ git push -$ git push origin maneage -``` +Make is a small language with the aim of defining *rules* that are composed +of *targets*, *prerequisites* and *recipes*. It comes with some nice +features like functions or automatic-variables to greatly facilitate the +management of text (filenames for example) or any of those constructs. For +a more detailed (yet still general) introduction see the [Make article on +Wikipedia](https://en.wikipedia.org/wiki/Make_(software)). +As of 2026, Make is a 50 year old software that is still getting better, +therefore many implementations of Make exist. The only difference in them +is some extra features over the [standard +definition](https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html) +(which is shared in all of them). Maneage is primarily written in GNU Make +that is the most common, most actively developed, and most advanced +implementation. Just note that Maneage downloads, builds, internally +installs, and uses its own dependencies (including GNU Make), so you don't +have to have it installed before you try it out. +The recipes (operations) that are applied on the dependencies of each +target to build the target are simple shell scripts in Maneage. In case you +are more comfortable with Python, Perl or R scripts, you can save them in +your project's source and simply call them in the recipe. +The [GNU Make book](https://www.gnu.org/software/make/manual) is arguably +the best place to learn Make. As mentioned in the "Info documentation" +section above, you also have the manual within your Maneage'd software +environment. The main caveat of the GNU Make manual is that all its +examples and demonstrations are based on compiling programs in C; not data +analysis. But fortunately beyond the first chapter, the rest of the +examples in the manual are sufficiently generic. +To replace the first chapter (and give examples of how Make can be used in +data analysis), along with some early adopters of Maneage and as part of +the IAC's Short Meetings on Astro-Computing Knowledge (SMACK) series, we +prepared [a hands-on video +demonstration](https://peertube.stream/w/iJitjS3r232Z8UPMxKo6jq) that you +can watch. In it we show how you can write your Make recipes in any +high-level scripting language. +Once you have watched the video above, read the GNU Make manual at least +until the end of Chapter 8 (called "Functions for transforming text"). You +can use the Info format manual within Maneage (as described in the "Info +documentation" section above) with `info make`. +### GNU Bash +The main shell that is installed within Maneage is GNU Bash. Like GNU Make, +it also has an Info manual, but unfortunately it is not written with +beginners in mind: it digs deep into very technical details from the +start. In case you are using Gnuastro in your analysis, its [general +tutorial](https://www.gnu.org/software/gnuastro/manual/html_node/General-program-usage-tutorial.html) +also includes many useful tips on the shell. We have also prepared the +following hands-on SMACK videos to introduce the shell -Publication checklist -===================== +- SMACK 2: [More on the command-line](https://peertube.stream/w/d16286ee-fc46-444f-acb8-645f9de6c7b7). +- SMACK 4: [Shell scripting](https://peertube.stream/w/18d2e386-2ca2-4a1f-a949-c7ca5dcee8ff). -Once your project is complete and you are ready to submit/publish the -project, we recommend the following steps to ensure the maximum FAIRness of -all your hard work (Findability, Accessibility, Interoperability, and -Reusability). This list may seem long, and may take a day or so to -complete, but please consider the fact that you have spent months/years on -your project, so it is a very small step in your over-all project! Most of -it is about organizing things that you can do during your project. So its -good to have a look at these from the start of your project. -As you will notice, when you complete this checklist, your projects source -will be present in multiple places: Zenodo, SoftwareHeritage, arXiv, your -own Git repositories. This is a major advantage of Maneaged(!) projects: -because their source is very small (a few hundred kilobytes), there is -effectively no cost in keeping multiple redundancies on different servers, -just in case one (or more) of them are discontinued in the (near/far) -future. - - **Reserve a DOI for your datasets**: There are multiple data servers - that give this functionality, one of the most well known and - (currently!) well-funded is [Zenodo](https://zenodo.org) so we'll focus - on it here. Of course, you can use any other service that provides a - similar functionality. Once you complete these steps, you can start - using/citing your dataset's DOI in the source of your project to - finalize the rest of the points. With Zenodo, you can even use the given - identifier for things like downloading. +### Git - * *Start new upload*: After you log in to Zenodo, you can start a new - upload by clicking on the "New Upload button". +Git is the version control system used in Maneage to allow easy tracking of +changes during the development of the project, collaborate with colleagues +and also to import improvements from the core branch into individual +projects. Without Git, such operations become so tedious and prone to human +error that the development of any software-based project with more than one +developer would be practically impossible. - * *Reserve DOI*: Under the "Basic information" --> "Digital Object - Identifier", click on the "Reserve DOI" button. +If you are not yet familiar with Git, the best resource is the free and +open source [Pro Git](https://git-scm.com/book/en) book. Don't be alarmed +by the number of chapters in the Pro Git book. The majority of the book is +for very advanced usages of Git, including setting up servers (that is not +relevant in Maneage). To comfortably use Maneage, mastering the following +chapters is enough - * *Fill basic info*: You need to at least fill in the "required fields" - (marked with a red star). You will always be able to change any - metadata (even after you "Publish"), so don't worry too much about - values in the fields, at this phase, its just important that they are - not empty. +- Chapter 1: Getting Started +- Chapter 2: Git Basics +- Chapter 3: Git Branching +- Chapter 5: Distributed Git - * *Save your project but do not yet publish*: Press the "Save" button - (at the top or bottom of the page). Do not yet press "Publish" though, - since that would make the project public, and freeze the DOI with any - possible file you may have uploaded already. We will get to the - publication phase in the next steps. +After mastering these chapters, you will get a very good feeling of any +further chapters in case you need them. Of course, [Git's own +manual](https://git-scm.com/docs/user-manual.html) is also available for +further details when necessary. - - **Record the metadata**: Maneage comes with a file to store all the - project's metadata: `reproduce/analysis/config/metadata.conf`. Open this - file and store all the information that you currently have: for example - the Zenodo DOI, project's Git repository, Copyright owner and license of - the data after it becomes public. Keep the empty fields in mind and - after obtaining them, don't forget to fill them up. - - **Request archival on SoftwareHeritage**: [Software - Heritage](https://archive.softwareheritage.org/save/) is an online - project to archive source code and their development histories. It - provides wonderful features for archiving source code (not data!) and - also for citing special parts of a project's source in any point of its - history. So it blends elegantly with the purpose of Maneage. Once you - make your project's Git repository publicly accessible (no login - required to clone it), you can request that SoftwareHeritage archives - it. Its good if you do this as soon as you make your Git repository - public. When you are ready, just register your repository's address (the - same one you give to `git clone`) to in [SoftwareHeritage's save - form](https://archive.softwareheritage.org/save). - - **Run a spell-check on `paper.tex`**: we all forget ;-)! - - **Zenodo/SoftwareHeritage links in paper**: put links to the Zenodo-DOI - (and SoftwareHeritage source when you make it public) in your - paper. Somewhere close to the start, maybe under the keywords/abstract, - highlighting that they are supplements for reproducibility. These help - readers easily access these resources for supplementary material - directly from your PDF paper (sources on SoftwareHeritage and - data/software on Zenodo). These links are more trusted/reliable in terms - of longevity than Git repositories or private webpages. - - **Identify and properly format output data**: If you have a plot, figure - or table in your paper, you need to verify that data later and publish - that data with the paper (see the steps below for both). But before - going to those steps, its good if you polish your datasets with the - recommendations below: - * *Keep published data in a special place*: it helps if you keep the - to-be-published data files in a special sub-directory under your build - directory. In this way, irrespective of which subMakefile builds a - published dataset, they won't be lost/scatterred in the middle of all - the project's intermediate-built files. - * *In plain-text*: If the data are in tabular form (for example the X - and Y values in your plots), store them as a simple plain-text file - (for example with columns separated by white-space characters) or in - the more formal [Comma-separated - values](https://en.wikipedia.org/wiki/Comma-separated_values) or CSV, - format). Generally, its best to set the suffixes to `.txt` (because - most browsers/OSs will automatically know they are plain-text and open - them without needing any other software). If you have other types of - data (for example images, or very large tables with millions of - rows/columns that can be inconvenient in plain-text), feel free to use - custom binary formats, but later, in the description of your project - on the server, add a note, explaining what software they should use to - open them. - * *Descriptive names*: In some papers there are many files and having - cryptic names will only confuse your readers (actually, yourself in - two years!). So set the names of the files to be as descriptive as - possible, so simply by reading the name of the file, someone who has - read the paper will understand what figure it corresponds to. In - particular, don't set names like `figure-3.txt`! In a few months you - will forget the order of the figures! Even worse, after the referee - report, you may need to re-arrange some figures and you will be forced - to rename everything related to each figure (which is very frustrating - and prone to errors). - * *Good metadata*: Raw data are not too useful merely as a series of raw - numbers! So don't forget to have **good metadata in every file**. If - its a plain-text file, usually lines starting with a `#` are - ignored. So in the command that generates each dataset, add some extra - information (the more the better!) about the dataset as lines starting - with `#`. Based on `reproduce/analysis/config/metadata.conf`, in - `initialize.mk`, Maneage will produce a default set of basic - information for plain-text data and will put it in the - `$(print-general-metadata)` variable. It is thus recommended to print - this variable into your plain-text file before printing the actual - data (so it shows on top of the file). For a real-world example, see - its usage in `reproduce/analysis/make/delete-me.mk` (in the `maneage` - branch). If you are publishing your data in binary formats, please add - all the metadata you see in `$(print-general-metadata)` into each - dataset file (for example keywords in the FITS format). If there are - many files, its easy to define a tiny shell-script to do the job on - each dataset. - - **Link to figure datasets in caption**: all the datasets that go into - the plots should be uploaded directly to Zenodo so they can be - viewed/downloaded with a simple link in the caption. For example see the - last sentence of the caption of Figure 1 in - [arXiv:2006.03018](https://arxiv.org/pdf/2006.03018.pdf), it points to - [the data](https://zenodo.org/record/3872248/files/tools-per-year.txt) - that was used to create that figure's left-side plot. As you see, this - will allow your paper's readers (again, most probably your future-self!) - to directly access the numbers of each visualization (plot/figure) with - a simple click in a trusted server. This also shows the major advantage - of having your data as simple plain-text where possible, as described - above. To help you keep all your to-be-visualized datasets in a single - place, Maneage has the two `tex-publish-dir` and `data-publish-dir` - directories that are defined in `reproduce/analysis/make/initialize.mk`, - see the comments above their definition for more. +# Maneage directory architecture + +In order to customize Maneage to your research, it is important to first +understand its souce and build directory architectures. This will help +navigate your way in the directories and understand how to implement your +research project: where to add new files and which existing files to modify +for what purpose. But if this the first time you are using Maneage, before +reading this theoretical discussion, please run Maneage once from scratch +without any changes (described in `README.md`). This will show you how it +works and will allow you to make test changes/edits in the code to test +what you read here. + +A core concept in Maneage is that that there is a very strong distinction +between the "source" (hand-written, plain-text files: most valuable of them +all) and "build" (for downloaded, or automatically generated +datasets/files) directories. + +## Source directory (hand-written files) + +The directory that is hosting this file in your operating system is the +"source" directory. This is where all hand-written files of your project +(the most valuable) are saved and it is fully under version control. The +source directory has two top-level directories: + +- `reproduce/`: contains all the "code" (assuming LaTeX is not "code") and + has two high-level sub-directories for the two high-level phases of + Maneage: + + - `software/`: all the instructions to download, build and install + (independent of the host operating system) the necessary + software. These are executed when you ran the `./harvester configure`. + + - `analysis/`: all the analysis steps of your project: that are executed + with the `./project make` command. + +- `tex/` contains all the final paper's components to be compiled into a + PDF using LaTeX. After building Maneage once, two symbolic links to the + build directory are placed here that are described in the corresponding + sub-section below. But they are not under version control, the place to + keep any extra hand-written LaTeX source is under the `src/` + sub-directory here. + +At the end of its configuration, Maneage also creates some symbolic links +for easy access to the build directory from your top source directory. See +the corresponding sub-section below for more on those: they are very useful +while you are developing your project. + +Once the project is configured for your system, `./harvester make` will do +the basic preparations and run the project's analysis with the custom +version of software. The `project` script is just a wrapper, and with the +`make` argument, it will first call `top-prepare.mk` and `top-make.mk` +(both are in the `reproduce/analysis/make` directory). + +In terms of organization, `top-prepare.mk` and `top-make.mk` have an +identical design, only minor differences. So, let's continue Maneage's +architecture with `top-make.mk`. Once you understand that, you'll clearly +understand `top-prepare.mk` also. These very high-level files are +relatively short and heavily commented so hopefully the descriptions in +each comment will be enough to understand the general details. As you read +this section, please also look at the contents of the mentioned files and +directories to fully understand what is going on. + +Before starting to look into the top `top-make.mk`, it is important to +recall that Make defines dependencies by files. Therefore, the +input/prerequisite and output of every step/rule must be a file. Also +recall that Make will use the modification date of the prerequisite(s) and +target files to see if the target must be re-built or not. Therefore during +the processing, _many_ intermediate files will be created (see the tips +section below on a good strategy to deal with large/huge files). + +To keep the source and (intermediate) built files separate, the user _must_ +define a top-level build directory variable (or `$(BDIR)`) to host all the +intermediate files (you defined it during `./harvester configure`). This +directory doesn't need to be version controlled or even synchronized, or +backed-up in other servers: its contents are all products, and can be +easily re-created any time. As you define targets for your new rules, it is +thus important to place them all under sub-directories of `$(BDIR)`. As +mentioned above, you always have fast access to this "build"-directory with +the `.build` symbolic link. Also, beware to *never* make any manual change +in the files of the build-directory, just delete them (so they are +re-built). + +In this architecture, we have two types of Makefiles that are loaded into +the top `Makefile`: _configuration-Makefiles_ (only independent +variables/configurations) and _workhorse-Makefiles_ (Makefiles that +actually contain analysis/processing rules). + +The configuration-Makefiles are those that satisfy these two wildcards: +`reproduce/software/config/*.conf` (for building the necessary software +when you run `./harvester configure`) and `reproduce/analysis/config/*.conf` +(for the high-level analysis, when you run `./harvester make`). These +Makefiles don't actually have any rules, they just have values for various +free parameters throughout the configuration or analysis. Open a few of +them to see for yourself. These Makefiles must only contain raw Make +variables (project configurations). By "raw" we mean that the Make +variables in these files must not depend on variables in any other +configuration-Makefile. This is because we don't want to assume any order +in reading them. It is also very important to *not* define any rule, or +other Make construct, in these configuration-Makefiles. + +Following this rule-of-thumb enables you to set these configure-Makefiles +as a prerequisite to any target that depends on their variable +values. Therefore, if you change any of their values, all targets that +depend on those values will be re-built. This is very convenient as your +project scales up and gets more complex. + +The workhorse-Makefiles are those satisfying this wildcard +`reproduce/software/make/*.mk` and `reproduce/analysis/make/*.mk`. They +contain the details of the processing steps (Makefiles containing +rules). Therefore, in this phase *order is important*, because the +prerequisites of most rules will be the targets of other rules that will be +defined prior to them (not a fixed name like `paper.pdf`). The lower-level +rules must be imported into Make before the higher-level ones. + +All processing steps are assumed to ultimately (usually after many rules) +end up in some number, image, figure, or table that will be included in the +paper. The writing of these results into the final report/paper is managed +through separate LaTeX files that only contain macros (a name given to a +number/string to be used in the LaTeX source, which will be replaced when +compiling it to the final PDF). So the last target in a workhorse-Makefile +is a `.tex` file (with the same base-name as the Makefile, but in +`$(BDIR)/tex/macros`). As a result, if the targets in a workhorse-Makefile +aren't directly a prerequisite of other workhorse-Makefile targets, they +can be a prerequisite of that intermediate LaTeX macro file and thus be +called when necessary. Otherwise, they will be ignored by Make. + +Maneage also has a mode to share the build directory between several +users of a Unix group (when working on large computer clusters). In this +scenario, each user can have their own cloned project source, but share the +large built files between each other. To do this, it is necessary for all +built files to give full permission to group members while not allowing any +other users access to the contents. Therefore the `./harvester configure` and +`./harvester make` steps must be called with special conditions which are +managed in the `--group` option. + +Let's see how this design is implemented. Please open and inspect +`top-make.mk` it as we go along here. The first step (un-commented line) is +to import the local configuration (your answers to the questions of +`./harvester configure`). They are defined in the configuration-Makefile +`reproduce/software/config/LOCAL.conf` which was also built by `./harvester +configure` (based on the `LOCAL.conf.in` template of the same directory). + +The next non-commented set of the top `Makefile` defines the ultimate +target of the whole project (`paper.pdf`). But to avoid mistakes, a sanity +check is necessary to see if Make is being run with the same group settings +as the configure script (for example when the project is configured for +group access using the `./for-group` script, but Make isn't). Therefore we +use a Make conditional to define the `all` target based on the group +permissions. + +Having defined the top/ultimate target, our next step is to include all the +other necessary Makefiles. However, order matters in the importing of +workhorse-Makefiles and each must also have a TeX macro file with the same +base name (without a suffix). Therefore, the next step in the top-level +Makefile is to define the `makesrc` variable to keep the base names +(without a `.mk` suffix) of the workhorse-Makefiles that must be imported, +in the proper order. + +Finally, we import all the necessary remaining Makefiles: 1) All the +analysis configuration-Makefiles with a wildcard. 2) The software +configuration-Makefile that contains their version (just in case its +necessary). 3) All workhorse-Makefiles in the proper order using a Make +`foreach` loop. + +In short, to keep things modular, readable and manageable, follow these +recommendations: 1) Set clear-to-understand names for the +configuration-Makefiles, and workhorse-Makefiles, 2) Only import other +Makefiles from top Makefile. These will let you know/remember generally +which step you are taking before or after another. Projects will scale up +very fast. Thus if you don't start and continue with a clean and robust +convention like this, in the end it will become very dirty and hard to +manage/understand (even for yourself). As a general rule of thumb, break +your rules into as many logically-similar but independent steps as +possible. + +The `reproduce/analysis/make/paper.mk` Makefile must be the final Makefile +that is included. This workhorse Makefile ends with the rule to build +`paper.pdf` (final target of the whole project). If you look in it, you +will notice that this Makefile starts with a rule to create +`$(mtexdir)/project.tex` (`mtexdir` is just a shorthand name for +`$(BDIR)/tex/macros` mentioned before). As you see, the only dependency of +`$(mtexdir)/project.tex` is `$(mtexdir)/verify.tex` (which is the last +analysis step: it verifies all the generated results). Therefore, +`$(mtexdir)/project.tex` is _the connection_ between the +processing/analysis steps of the project, and the steps to build the final +PDF. - - **Verification step**: It is very important to automatically verify the - outptus of your project. Recall from the customization checklist (above) - that you can activate verification by setting the `verify-outputs` - variable to `yes` in `reproduce/analysis/config/verify-outputs.conf`. So - please activate it and look into the `reproduce/analysis/make/verify.mk` - to add the necessary steps to automatically verify your outputs. *Tip*: - you don't have to generate the checksums manually, just give a wrong - value (for example `XXXX`) so Maneage crashes! In the error message it - will then print the actual and expected checksums and you can take the - value from there. Outputs that must be verified can be listed as: +During the research, it often happens that you want to test a step that is +not a prerequisite of any higher-level operation. In such cases, you can +(temporarily) define that processing as a rule in the most relevant +workhorse-Makefile and set its target as a prerequisite of its TeX +macro. If your test gives a promising result and you want to include it in +your research, set it as prerequisites to other rules and remove it from +the list of prerequisites for TeX macro file. In fact, this is how a +project is designed to grow in this framework. - * *subMakefile LaTeX macro files*: these LaTeX macros put numbers into - the text. You don't want your readers (actually: yourself in two - years!) to have to painfully find and check, by eye, all those tiny - numbers buried deep in the ocean of words! +## Build directory (automatically generated files) + +The build directory has the same sub-directories as the `reproduce/` +sub-directory of the source. + +- `software/`: all the automatically built products during the Maneage + configuration phase. It includes the following sub-directories: + - `config/`: all the local configurations for your system. The + `LOCAL.conf` file of this directory can contain your server user ID + and password (used when your input data are in servers that need + authentication). + - `installed/`: the top-level software environment of Maneage. Just like + your Unix-based operating system, it has sub-directories like `bin/` + (for executable software), `lib/` (for libraries) and etc. + - `locks/`: This directory is used to keep file locks when necessary (it + may be removed later). File locks are useful to stop the + parallelization. for example when we want all the steps to be done in + parallel, but not for steps like downloading (where the + network/servers can be a bottleneck and actually decrease the speed of + a parallelized operation compared to its serial operation). + - `tarballs/`: The source tarballs of all the software that were used to + build the software environment. If they were in the directory given to + `--software-dir`, they will be symbolic links to the files in that + directory. Otherwise (if they were downloaded), they will be normal + files. +- `analysis/`: all the analysis outputs. The default (in the `maneage` + branch) will create two sub-directories here when you run `./project + make`: + - `inputs/`: all the input files to the project. Similar to the + `tarballs/` directory above, if the files were present on the host + file system (in a directory given to `--input-dir`), a symbolic link + will be placed. Otherwise (if they were downloaded), the raw file will + be present. You can use this distinction (being a symbolic link or + not) to see which files were not on your system and copy them into a + dedicated directory to avoid needing the internet on every new run. + + - `tex/`: This directory contains the built files that are necessary for + LaTeX aspect of you final PDF. It is recommended to not keep large + data files here, only that absolute minimum that you need to create + the final PDF (the contents of this directory will need to be uploaded + to publication severs like arXiv). The following sub-directories are + assumed to always be present here: + - `build\`: the directory that LaTeX actually executes in. The + `paper.tex` (of the top source directory) will be copied here and + the directories containing the other LaTeX files are included into + as directories under the `tex/` directory of the top-level source. + - `macros\`: all the macros from the various stages of the software + and analysis will be present as different files here. As described + in the source directory descriptions, these are the backbone of + Maneage's operation (one for every sub-Makefile). + - `to-publish\`: will contain the raw tables behind every figure. For + example, see Appendix E of [Eskandarlou et + al. 2026](https://scixplorer.org/abs/2026A%26A...709A.210E). + + +## Easy access to build directory from source + +After it finishes, `./harvester configure` will create the following +symbolic links (also directories) in the project's top source directory for +easy access to the build directory. + +- `.build` points to the top build directory. This is very useful for fast + access to the build directory without having to worry about which system + is hosting your maneage'd project: in one system, your build directory + can be in one location and on another system in another directory. But + thanks to this symbolic link, you do not have to worry which system you + are developing on. + +- `.local` for easy access to the custom built software installation + directory. For example the `ls` command of your project can be accessed + outside of Maneage with `.local/bin/ls`. This is useful when you want to + run your Maneage'd software outside of maneage (although using `./project + shell` is a much better strategy for that). + +There are also two symbolic links under the `tex/` directory of the source +that are listed below. When running `./project make dist`, the contents of +these symbolic links will be copied into the tarball. In this way, +publishers (like arXiv) do not have to run Maneage, they just run LaTeX and +have all the products that are necessary). + +- `tex/build` for direct access to the `analysis/tex/build` directory. As + described above, this directory will contain all the files that go into + the TeX source of the paper. + +- `tex/tikz` for the outputs of [PGF & TiKZ](https://ctan.org/pkg/pgf) (or + [PGFPLots](https://ctan.org/pkg/pgfplots), that is based on it). These + packages can generate plots within LaTeX. Therefore, they use the same + font, line width and colors of the paper, making your plots fit very + nicely (visually) in the final paper. Another nice aspect is that with + these software, you do not need to generate the plots/figures separate + from the paper (PGFPlots can directly read the tables you put in the + `to-publish/` subdirectory mentioned under the build directory. + + + + + + + + + + +# Maneagement tips - * *Final data files* (for tables, figures, or plots, or as data - release). These are the same files described above. If you have - followed the guidelines above and stored them as plain-text with - comments on top, you can use the provided function - `verify-txt-no-comments-leading-space` which takes the filename and - checksum as arguments to avoid the commented lines (which may change) - and only verify the data. If your data are in other formats, be sure - to verify them without metadata that may change (like date and etc). +The following is a list of design points, tips, or recommendations that +have been learned after some experience with this type of project +management. Please don't hesitate to share any experience you gain after +using it with us. In this way, we can add it here (with full giving credit) +for the benefit of others. - - **Fill `README.md`**: The `README.md` is *the first place* your readers - are going to look into. It already has a default text with place-holders - in the form of `XXXXXX`. Please go through its first few paragraphs and - replace the place-holders with the relevant information/links or feel - free to add/remove anything else. The rest is just basic information - that is useful for any Maneage'd project. Just don't forget to tell your - readers in `README.md` that they can learn about this system in the - `README-hacking.md` file (ideally close to the top). - - **Confirm if your project builds from scratch**: Before publishing - anything, you should see if your project can indeed reproduce itself! - You may be mistakenly using temporarily created files that aren't built - when the project is built from scratch (this happens a lot and is very - dangerous for the integrity of your project!). So, go to a temporary - directory, clone your project from its repository and try configuring - and building it from scratch in a new-temporary build-directory. It is - important to ignore the original directory you developed your project on - (source and build): you may have files there that you forgot to import - into Git or depended on in the build (it happens!). Ideally, it would be - good to try it on a different computer. - - **Confirm if `./project make dist` works**: The special target `dist` - tells the project to build a tarball that is ready to compile the LaTeX - PDF without having to do the analysis and build software. This is very - useful for servers like arXiv, or some journals. This tarball is also - one of the deliverables you want to publish on Zenodo. Once the tarball - is created, copy it to a temporary directory outside of Maneage, unpack - it and run `make` (completely ignoring Maneage's `./project` script). If - you plan to submit your paper to arXiv, the best test is to actually - start a test submission on arXiv to upload the tarball there to see if - it can build your PDF. Once it works, you can delete that temporary - submission for now. Afterwards, try configuring and building it with the - tarball by running its `./project` (from scratch and without the Git - history!). If there is a problem in any of these tests, you can modify - what goes into this tarball in `reproduce/analysis/make/initialize.mk`: - go through the steps and add the necessary components until the checks - pass. +## Modularity + +Modularity is the key to easy and clean growth of a project. So it is +always best to break up a job into as many sub-components as +reasonable. Here are some tips to stay modular. + +- *Short recipes*: if you see the recipe of a rule becoming more than a + handful of lines which involve significant processing, it is probably a + good sign that you should break up the rule into its main + components. Try to only have one major processing step per rule. + +- *Context-based (many) Makefiles*: For maximum modularity, this design + allows easy inclusion of many Makefiles: in + `reproduce/analysis/make/*.mk` for analysis steps, and + `reproduce/software/make/*.mk` for building software. So keep the rules + for closely related parts of the processing in separate Makefiles. + +- *Descriptive names*: Be very clear and descriptive with the naming of the + files and the variables because a few months after the processing, it + will be very hard to remember what each one was for. Also this helps + others (your collaborators or other people reading the project source + after it is published) to more easily understand your work and find + their way around. + +- *Naming convention*: As the project grows, following a single standard or + convention in naming the files is very useful. Try best to use multiple + word filenames for anything that is non-trivial (separating the words + with a `-`). For example if you have a Makefile for creating a catalog + and another two for processing it under models A and B, you can name + them like this: `catalog-create.mk`, `catalog-model-a.mk` and + `catalog-model-b.mk`. In this way, when listing the contents of + `reproduce/analysis/make` to see all the Makefiles, those related to the + catalog will all be close to each other and thus easily found. This also + helps in auto-completions by the shell or text editors like Emacs. + +- *Source directories*: If you need to add files in other languages for + example in shell, Python, AWK or C, keep the files in the same language + in a separate directory under `reproduce/analysis`, with the appropriate + name. + +- *Configuration files*: If your research uses special programs as part of + the processing, put all their configuration files in a devoted directory + (with the program's name) within `reproduce/software/config`. It is much + cleaner and readable (thus less buggy) to avoid mixing the configuration + files, even if there is no technical necessity. + + + +## Contents of source files + +It is good practice to follow the following recommendations on the contents +of your files, whether they are source code for a program, Makefiles, +scripts or configuration files (copyrights aren't necessary for the +latter). + +- *Copyright*: Always start a file containing programming constructs with a + copyright statement like the ones that Maneage starts with (for example + in the top level `Makefile`). + +- *Comments*: Comments are vital for readability (by yourself in two + months, or others). Describe everything you can about why you are doing + something, how you are doing it, and what you expect the result to + be. Write the comments as if it was what you would say to describe the + variable, recipe or rule to a friend sitting beside you. When writing + the project it is very tempting to just steam ahead with commands and + codes, but be patient and write comments before the rules or + recipes. This will also allow you to think more about what you should be + doing. Also, in several months when you come back to the code, you will + appreciate the effort of writing them. Just don't forget to also read + and update the comment first if you later want to make changes to the + code (variable, recipe or rule). As a general rule of thumb: first the + comments, then the code. + +- *File title*: In general, it is good practice to start all files with a + single line description of what that particular file does. If further + information about the totality of the file is necessary, add it after a + blank line. This will help a fast inspection where you don't care about + the details, but just want to remember/see what that file is (generally) + for. This information must of course be commented (its for a human), but + this is kept separate from the general recommendation on comments, + because this is a comment for the whole file, not each step within it. + + + +## Make programming + +Here are some experiences that we have come to learn over the years in +using Make and are useful/handy in research contexts. + +### Environment of each recipe + +If you need to define a special environment (or aliases, or scripts to run) +for all the recipes in your Makefiles, you can use a Bash startup file +`reproduce/software/shell/bashrc.sh`. This file is loaded before every Make +recipe is run, just like the `.bashrc` in your home directory is loaded +every time you start a new interactive, non-login terminal. See the +comments in that file for more. + +### Automatic variables + +These are wonderful and very useful Make constructs that greatly shrink the +text, while helping in read-ability, robustness (less bugs in typos for +example) and generalization. For example even when a rule only has one +target or one prerequisite, always use `$@` instead of the target's name, +`$<` instead of the first prerequisite, `$^` instead of the full list of +prerequisites and etc. You can see the full list of automatic variables +[here](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html). If +you use GNU Make, you can also see this page on your command-line: - - **Upload all deliverables to Zenodo**: With the datasets ready, you can - now upload the following deliverables to Zenodo. Except for the data - files, put the Git hash of your Maneaged project at the moment of - publication in the filename of other uploaded files. The output files - shouldn't have a hash in their names because their URL (that goes in the - caption of the figures/tables) should be known prior to a commit, - creating a cyclic dependency! Ideally the hash should be placed just - before the final suffix, for example `paper-XXXXXXX.pdf` (where - `XXXXXXX` is the Git hash). This will clearly identify the point in - history that your file was created. +```shell +$ info make "automatic variables" +``` - * **paper-XXXXXXX.pdf**: you shouldn't just download data to the data - server, also upload your paper's PDF so its there with the other raw - formats. It will greatly help yourself and others. Most datacenters - (like Zenodo) actually also have a PDF viewer that will load - automatically before the list of data files. For example see - [zenodo.3408481](https://doi.org/10.5281/zenodo.3408481). +### Debugging Make dependencies + +Since Make doesn't follow the common top-down paradigm, it can be a little +hard to get accustomed to why you get an error or un-expected behavior. In +such cases, run Make with the `-d` option. With this option, Make prints a +full list of exactly which prerequisites are being checked for which +targets. Looking (patiently) through this output and searching for the +faulty file/step will clearly show you any mistake you might have made in +defining the targets or prerequisites. + +### Large files + +If you are dealing with very large files (thus having multiple copies of +them for intermediate steps is not possible), one solution is the following +strategy (Also see the next item on "Fast access to temporary files"). Set +a small plain text file as the actual target and delete the large file when +it is no longer needed by the project (in the last rule that needs +it). Below is a simple demonstration of doing this. In it, we use +Gnuastro's Arithmetic program to add all pixels of the input image with 2 +and create `large1.fits`. We then subtract 2 from `large1.fits` to create +`large2.fits` and delete `large1.fits` in the same rule (when its no longer +needed). We can later do the same with `large2.fits` when it is no longer +needed and so on. + +```make +large1.fits.txt: input.fits + astarithmetic $< 2 + --output=$(subst .txt,,$@) + echo "done" > $@ +large2.fits.txt: large1.fits.txt + astarithmetic $(subst .txt,,$<) 2 - --output=$(subst .txt,,$@) + rm $(subst .txt,,$<) + echo "done" > $@ +``` - * **`project-XXXXXXX.tar.gz`**: Or the output of `make dist` as - described above. +A more advanced Make +programmer will use Make's [call +function](https://www.gnu.org/software/make/manual/html_node/Call-Function.html) +to define a wrapper in `reproduce/analysis/make/initialize.mk`. This +wrapper will replace `$(subst .txt,,XXXXX)`. Therefore, it will be possible +to greatly simplify this repetitive statement and make the code even more +readable throughout the whole project. + +### Fast access to temporary files + +Most Unix-like operating systems will give you a special shared-memory +device (directory): on systems using the GNU C Library (all GNU/Linux +system), it is `/dev/shm`. The contents of this directory are actually in +your RAM, not in your persistence storage like the HDD or SSD. Reading and +writing from/to the RAM is much faster than persistent storage, so if you +have enough RAM available, it can be very beneficial for large temporary +files to be put there. You can use the `mktemp` program to give the +temporary files a randomly-set name, and use text files as targets to keep +that name (as described in the item above under "Large files") for later +deletion. For example, see the minimal working example Makefile below +(which you can actually put in a `Makefile` and run if you have an +`input.fits` in the same directory, and Gnuastro is installed). + +```make +.ONESHELL: +.SHELLFLAGS = -ec +all: mean-std.txt +shm-maneage := /dev/shm/$(shell whoami)-maneage-XXXXXXXXXX +large1.txt: input.fits + out=$$(mktemp $(shm-maneage)) + astarithmetic $< 2 + --output=$$out.fits + echo "$$out" > $@ +large2.txt: large1.txt + input=$$(cat $<) + out=$$(mktemp $(shm-maneage)) + astarithmetic $$input.fits 2 - --output=$$out.fits + rm $$input.fits $$input + echo "$$out" > $@ +mean-std.txt: large2.txt + input=$$(cat $<) + aststatistics $$input.fits --mean --std > $@ + rm $$input.fits $$input +``` - * **`project-git.bundle`** This is the full Git history of the project - in one file (which you can actually clone from later!). Its - necessary to publish this with your dataset too because Git - repositories make no promise on longevity. The way to "bundle" a Git - history is described below, in summary, its this command: - ```shell - $ git bundle create my-project-git.bundle --all - ``` +The important point here is that the temporary name template +(`shm-maneage`) has no suffix. So you can add the suffix corresponding to +your desired format afterwards (for example `$$out.fits`, or +`$$out.txt`). But more importantly, when `mktemp` sets the random name, it +also checks if no file exists with that name and creates a file with that +exact name at that moment. So at the end of each recipe above, you'll have +two files in your `/dev/shm`, one empty file with no suffix one with a +suffix. The role of the file without a suffix is just to ensure that the +randomly set name will not be used by other calls to `mktemp` (when running +in parallel) and it should be deleted with the file containing a +suffix. This is the reason behind the `rm $$input.fits $$input` command +above: to make sure that first the file with a suffix is deleted, then the +core random file (note that when working in parallel on powerful systems, +in the time between deleting two files of a single `rm` command, many +things can happen!). When using Maneage, you can put the definition of +`shm-maneage` in `reproduce/analysis/make/initialize.mk` to be usable in +all the different Makefiles of your analysis, and you won't need the three +lines above it. **Finally, BE RESPONSIBLE:** after you are finished, be +sure to clean up any possibly remaining files (due to crashes in the +processing while you are working), otherwise your RAM may fill up very +fast. You can do it easily with a command like this on your command-line: +`rm -f /dev/shm/$(whoami)-*`. + + + +## Inputs (software tarballs and data) + +It is critically important to document the raw inputs to your project +(software tarballs and raw input data): + +- *Keep the source tarball of dependencies*: After configuration finishes, + the `.build/software/tarballs` directory will contain all the software + tarballs that were necessary for your project. You can mirror the + contents of this directory to keep a backup of all the software tarballs + used in your project (possibly as another version controlled repository) + that is also published with your project. Note that software web-pages + are not written in stone and can suddenly go offline or not be + accessible in some conditions. This backup is thus very important. If + you intend to release your project in a place like Zenodo, you can + upload/keep all the necessary tarballs (and data) there with your + project. [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746) is one + example of how the data, Gnuastro (main software used) and all major + Gnuastro's dependencies have been uploaded with the project's + source. Just note that this is only possible for [free + software](https://www.gnu.org/philosophy/free-sw.html). + +- *Keep your input data*: The input data is also critical to the project's + reproducibility, so like the above for software, make sure you have a + backup of them, or their persistent identifiers (PIDs). + + + +## Project outputs + +During your research, it is possible to checkout a specific commit and +reproduce its results. However, the processing can be time +consuming. Therefore, it is useful to also keep track of the final outputs +of your project (at minimum, the paper's PDF) in important points of +history. However, keeping a snapshot of these (most probably large volume) +outputs in the main history of the project can unreasonably bloat it. It is +thus recommended to make a separate Git repo to keep those files and keep +your project's source as small as possible. For example if your project is +called `my-exciting-project`, the name of the outputs repository can be +`my-exciting-project-output`. This enables easy sharing of the output files +with your co-authors (with necessary permissions) and not having to bloat +your email archive with extra attachments also (you can just share the link +to the online repo in your communications). After the research is +published, you can also release the outputs repository, or you can just +delete it if it is too large or un-necessary (it was just for convenience, +and fully reproducible after all). For example Maneage's output is +available for demonstration in [a +separate](http://git.maneage.org/output-raw.git/) repository. + + + +## Git and version control + +Version control is a critical component of Maneage. Here are some tips to +help in effectively using it. + +### Committing + +Every commit is a significant point in the history of your project: when +you consider one component of the job to be complete and ready to move to +the next component. This significance is very subjective: it can be a +significant change in one part of one file, or many files, or even a +temporal criteria: a milestone is reaching and you need to change +topic. Here are some tips on good habits in committing. + +- *Regular commits*: It is important (and extremely useful) to have the + history of your project under version control. So try to make commits + regularly (after any meaningful change/step/result). + +- *Commit message*: The commit message is a very important and useful + aspect of version control. To make the commit message useful for others + (or yourself, one year later), it is good to follow a consistent + style. Maneage already has a consistent formatting (described below), + which you can also follow in your project if you like. You can see many + examples by running `git log` in the `maneage` branch. If you intend to + push commits to Maneage, for the consistency of Maneage, it is necessary + to follow these guidelines. 1) No line should be more than 75 characters + (to enable easy reading of the message when you run `git log` on the + standard 80-character terminal). 2) The first line is the title of the + commit and should summarize it (so `git log --oneline` can be + useful). The title should also not end with a point (`.`, because its a + short single sentence, so a point is not necessary and only wastes + space). 3) After the title, leave an empty line and start the body of + your message (possibly containing many paragraphs). 4) Describe the + context of your commit (the problem it is trying to solve) as much as + possible, then go onto how you solved it. One suggestion is to start the + main body of your commit with "Until now ...", and continue describing + the problem in the first paragraph(s). Afterwards, start the next + paragraph with "With this commit ...". + +### Full Git history in one file + +When you are publishing your project (for example to Zenodo for long term +preservation), it is more convenient to have the whole project's Git +history into one file to save with your datasets. After all, you can't be +sure that your current Git server (for example GitLab, Github, or +Bitbucket) will be active forever. While they are good for the immediate +future, you can't rely on them for archival purposes. Fortunately keeping +your whole history in one file is easy with Git using the following +commands. To learn more about it, run `git help bundle`. + +Run the following command to "bundle" your project's history into one file +(just don't forget to change `my-project-git.bundle` to a descriptive name +of your project): - * **`software-XXXXXXX.tar.gz`**: This is effectively a copy of all the - software source code tarballs in your project's - `.build/software/tarballs`. It is necessary to upload these with - your project to avoid relying on third party servers. In the future - any one of those servers may go down and if so, your project won't - be buildable. You can generate this tarball easily with `./project - make dist-software`. +```shell +$ git bundle create my-project-git.bundle --all +``` - * All the figure (and other) output datasets of the project. Don't - rename these files, let them have the same descriptive name - mentioned above. Also recall that a link to all these files is also - in the caption of the respective figure. +You can easily upload `my-project-git.bundle` anywhere. Later, if you need +to un-bundle it, you can use the following command. - - **Upload to [arXiv](https://arxiv.org)**: or to any other pre-print - server (if you want to). Of course, you can also do this after the - initial/final submission to your desired journal. But we'll just add the - necessary points for arXiv submission here: +```shell +$ git clone my-project-git.bundle +``` - * *Necessary links in comments*: put a link to your project's Git - repository, Zenodo-DOI (this is not your paper's DOI, its the - data/resources DOI), and/or SoftwareHeritage link in the comments. +### File modification dates (meta data) - - *Update `metadata.conf`*: Once you have your final arXiv ID (formated - as: `1234.56789`) put it in `reproduce/analysis/config/metadata.conf`. +While Git does an excellent job at keeping a history of the contents of +files, it makes no effort in keeping the file meta data, and in particular +the dates of files. Therefore when you checkout to a different branch, +files that are re-written by Git will have a newer date than the other +project files. However, file dates are important in the current design of +Maneage: Make checks the dates of the prerequisite files and target files +to see if the target should be re-built. - - **Submission to a journal**: different journals accept submissions in - different formats, some accept LaTeX, some only want a PDF, or etc. It - would be good if you highlight in the cover-letter that your work is - reproducible and provide the Zenodo and Software Heritage links (if they - are public). If not, you can mention that everything is ready for such a - submission after acceptance. +This is relevant, for example, in the `analysis/` side of calculations when +a rule in a `reproduce/analysis/make/SOMETHING.mk` is dependent on +`reproduce/analysis/conf/SOMETHING.conf` (where `SOMETHING` is one of the +science pipeline steps that you have prepared). Switching back and forth +between git branches will update the timestamp on `SOMETHING.conf`, +incorrectly implying, according to the Make rules, that a rule in +SOMETHING.mk has to be run again. - * Some journals (like the "Astronomy and Astrophysics" journal) - require that only a single LaTeX file be submitted without any - commens (along with all the figures). In other words, they do not - want any sub-directories: only a single LaTeX file, one `.bbl` file - and one PDF per figure. Here is how you can do this: - ```shell - # If you don't already have 'latexpand' in your OS, install it: - $ tlmgr install latexpand +To address this problem, you can use the `touch` command to backdate the +timestamp of files that you want to be considered to be old. For example +with the command below: + +```shell +$ touch --date=2000-01-01 reproduce/analysis/conf/SOMETHING.conf +$ ls -l reproduce/analysis/conf/SOMETHING.conf # to check +``` + +You may want to use a date that is obviously (to you) fake, so that you +don't accidentally believe the date later on. - # Unpack the tarball and make a directory for a single-LaTeX source. - $ tar -xf maneaged-XXXXXXX.tar.gz - $ cd maneaged-XXXXXXX.tar.gz - $ mkdir one-latex - ## The '--empty-comments' and subsequent call to AWK (that removes - ## lines starting with '%') is because we need the lines ending in a - ## '%' for some LaTeX macros). - $ latexpand paper.tex --output=one-latex/paper-0.tex \ - --empty-comments - $ awk '!/^%/' one-latex/paper-0.tex > one-latex/paper.tex - $ rm one-latex/paper-0.tex - ## Copy the other necessary files (if you are using specific journal - ## style files, also copy them). - $ cp paper.bbl one-latex/ - $ cp tex/tikz/*.pdf one-latex/ - ## Correct the location of the images. - $ sed -i -e's|tex/tikz/|./|' one-latex/paper.tex - ``` - * You can confirm the contents of the `one-latex` directory made - above by running `pdflatex paper` inside that directory. Your - operating system's LaTeX may not have all the necessary packages, - in that case run `tlmgr install XXXXX` (where `XXXXX` is the - value of `texlive-packages` in - `reproduce/software/config/texlive-packages.conf`. Afterwards, - clean up all the temporary files with this command: - ```shell - $ rm *.log *.aux *.auxlock *.bcf *.out *.xml - ``` - * Upload the contents of this `one-latex` directory to the journal. - - **Future versions**: Both Zenodo and arXiv allow uploading new versions - after your first publication. So it is recommended to put more recent - versions of your published projects later (for example after applying - the changes suggested by the referee). In Zenodo (unlike arXiv), you - only need to publish a new version if the uploaded files have - changed. You can always update the project's metadata with no effect on - the DOI (so you don't need to upload a new version if you just want to - update the metadata). - - **After acceptance (before publication)**: Congratulations on the - acceptance! The main science content of your paper can't be changed any - more, but the paper will now go to the publication editor (for language - and style). Your approval of the final proof is necessary before the - paper is finally published. Use this period to finalize the final - metadata of your project: the journal's DOI. Some journals associate - your paper's DOI during this process. So before approving the final - proof do these steps: - * Add the Journal DOI in `reproduce/analysis/config/metadata.conf`, - and re-build your final data products, so this important metadata is - added. - * Once you get the final proof, and if everything is OK for you, - implement all the good language corrections/edits they have made - inside your own copy here and commit it into your project. This will - be the final commit of your project before publication. +# Maneaged papers already published - * Submit your final project as a new version to Zenodo (and - arXiv). The Zenodo one is most important because your plots will - link to it and you want the commit hash in the data files that - readers will get from Zenodo to be the same hash as the paper. +The list below shows some of the works that have already been published +with (earlier versions of) Maneage, and some that have been recently +submitted for peer review. Maneage is evolving rapidly, so some details +will differ between the different versions. The more recent papers will +tend to be the most useful as good working examples if you are just +starting. - * Tell the journal's publication editor to correct the hash and Zenodo - ID in your final proof confirmation (so the links point to the - correct place). Recall that on every new version upload in Zenodo, - you get a new DOI (or Zenodo ID). +In the list below, by clicking on the paper's publication year, you will be +taken to the paper's indexed page on the [SciX Digital +Library](https://scixplorer.org) (successor to the [NASA +ADS](https://ui.adsabs.harvard.edu)) which contains further bibliographic +information like title, DOI and etc, in a unified format (independent of +which journal the paper is published in). + + - Eskandarlou et + al. ([2026](https://scixplorer.org/abs/2026A%26A...709A.210E), A&A + Volume 709, id.A210, 17 pp; + arXiv:[2603.22166](https://arxiv.org/abs/2603.22166)). The + project's version controlled source is on + [Gitlab](https://gitlab.com/sepideh.esk/alba), necessary software, + outputs and backup of history are available at + [zenodo.17674805](http://doi.org/10.5281/zenodo.17674805); and the + archived git history is available at + [swh:1:dir:7c203a565ec6ef40d8419dbdf70e8639b78f4b99](https://archive.softwareheritage.org/swh:1:dir:7c203a565ec6ef40d8419dbdf70e8639b78f4b99;origin=https://gitlab.com/Sepideh.Esk/alba;visit=swh:1:snp:adc584ff3540ea3151de6ebc8462805e8818b1f4;anchor=swh:1:rev:f9d0f9cb25c80415107c962cc53d369a650e9b82). + - Eskandarlou et + al. ([2026](https://scixplorer.org/abs/2026A%26A...705A..67E), A&A, + Volume 705, id.A67, 14 pp; + arXiv:[2510.12940](https://arxiv.org/abs/2510.12940)). The project's + version controlled source is on + [Gitlab](https://gitlab.com/sepideh.esk/psf-j-plus), necessary software, + outputs and backup of history are available at + [zenodo.17348653](http://doi.org/10.5281/zenodo.17348653); and the + archived git history is available at + [swh:1:dir:fc95ed9ad173de6fa64690e3d71ab041c630f32c](https://archive.softwareheritage.org/swh:1:dir:fc95ed9ad173de6fa64690e3d71ab041c630f32c;origin=https://gitlab.com/Sepideh.Esk/psf-j-plus;visit=swh:1:snp:31bb4550902f7902a0a65d0674106271de776d3e;anchor=swh:1:rev:4860c70d6285a60300e429889e8637e98568a915). + - Saremi et al. ([2025](https://scixplorer.org/abs/2025A%26A...701A.116S), + A&A, Volume 701, id.A116, 18 pp.; + arXiv:[2508.02780](https://arxiv.org/abs/2508.02780)). The project's + version controlled source is on + [Gitlab](https://gitlab.com/nasim-projects/pipeline), necessary + software, outputs and backup of history are available at + [zenodo.16152699](https://doi.org/10.5281/zenodo.16152699); and the + archived git history is available at + [swh:1:dir:b3657cfb6053fd976695bd63c15cb99e5095648a](https://archive.softwareheritage.org/swh:1:dir:b3657cfb6053fd976695bd63c15cb99e5095648a;origin=https://gitlab.com/nasim-projects/pipeline;visit=swh:1:snp:ab7c6f0b9999f42d77154103c1bc082fa23b325c;anchor=swh:1:rev:afeb282c01983cba2a11eb4b2f25d5a40d35c164). + - Eskandarlou & Akhlaghi + ([2024](https://scixplorer.org/abs/2024RNAAS...8..168E), RNAAS, Volume + 8, Issue 6, id.168; + arXiv:[2406.14619](https://arxiv.org/abs/2406.14619)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `polar-plot` + branch). Necessary software, outputs and backup of history are available + at [zenodo.11403643](https://doi.org/10.5281/zenodo.11403643); and the + archived git history is available at + [swh:1:dir:4e09bf85f9f87336fa55920bf67e7bcf6d58bbd5](https://archive.softwareheritage.org/swh:1:dir:4e09bf85f9f87336fa55920bf67e7bcf6d58bbd5;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:557ee1a90de465659659ecc46df0c5ce29d0bb61;anchor=swh:1:rev:375e12e52080006be6a28e10980e79ef54d13d1d). + - Infante-Sainz et + al. ([2024](https://scixplorer.org/abs/2024RNAAS...8...22I), RNAAS, + Volume 8, Issue 1, id.22; + arXiv:[2401.05303](https://arxiv.org/abs/2401.05303)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `radial-profile` + branch). Necessary software, outputs and backup of history are available + at [zenodo.10124582](https://doi.org/10.5281/zenodo.10124582); and the + archived git history is available at + [swh:1:dir:d5029e066916cb64f0d95d20eb88294acc78b2b1](https://archive.softwareheritage.org/swh:1:dir:d5029e066916cb64f0d95d20eb88294acc78b2b1;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:b065324c2ef3b48bc26e8f30e48102a1abd2052f;anchor=swh:1:rev:61764447b16da44538e5ddbf7fb69937ba138e81). + - Infante-Sainz & Akhlaghi + ([2024](https://scixplorer.org/abs/2024RNAAS...8...10I), RNAAS, Volume + 8, Issue 1, id.10; + arXiv:[2401.03814](https://arxiv.org/abs/2401.03814)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `color-faint-gray` + branch). Necessary software, outputs and backup of history are available + at [zenodo.10058165](https://doi.org/10.5281/zenodo.10058165); and the + archived git history is available at + [swh:1:dir:1064a48d4bb58d6684c3df33c6633a04d4141d2d](https://archive.softwareheritage.org/swh:1:dir:1064a48d4bb58d6684c3df33c6633a04d4141d2d;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:a083ff647c571f895d1ccc9f7432fa1b9a1d03a8;anchor=swh:1:rev:ff77b619daa50b05ddd83206d979d1f8a53d040b). + - Peper, Roukema & Bolejko + ([2023](https://scixplorer.org/abs/2023MNRAS.525...91P), MNRAS, 525, 91, + DOI:10.1093/mnras/stad2246, + arXiv:[2304.00591](https://arxiv.org/abs/2304.00591)): The live version + of the controlled source is [at + Codeberg](https://codeberg.org/mpeper/lensing); the main input + dataset, a software snapshot, the software tarballs, the project outputs + and editing history are available at + [zenodo.8103985](https://zenodo.org/record/8103985); and the archived + git history is available at + [swh:1:dir:f57fd0e254bcb976c1ab34b6c9c9708e891f7914](https://archive.softwareheritage.org/swh:1:dir:f57fd0e254bcb976c1ab34b6c9c9708e891f7914;origin=https://codeberg.org/mpeper/lensing;visit=swh:1:snp:984413c49bb02ee0eec88c3ac01614808d57beb7;anchor=swh:1:rev:185c36f1b6263588526be3881078626869041882). + - Eskandarlou et + al. ([2023](https://scixplorer.org/abs/2023RNAAS...7..269E), RNAAS, + Volume 7, Issue 12, id.269; + arXiv:[2312.04263](https://arxiv.org/abs/2312.04263)). The project's + version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the `zeropoint` + branch). Necessary software, outputs and backup of history are available + at [zenodo.10256845](https://doi.org/10.5281/zenodo.10256845); and the + archived git history is available at + [swh:1:dir:8b2d1f63be96de3de03aa3e2bb68fa7fa52df56f](https://archive.softwareheritage.org/swh:1:dir:8b2d1f63be96de3de03aa3e2bb68fa7fa52df56f;origin=https://codeberg.org/gnuastro/papers;visit=swh:1:snp:e37e226bab517eef24d854467682b2fcf5d7dc32;anchor=swh:1:rev:ea682783d83707c0e1d114a5de74a100be9f545d). + - Akhlaghi ([2023](https://scixplorer.org/abs/2023RNAAS...7..211A), RNAAS), Volume 7, Issue + 10, id.211; arXiv:[2310.15006](https://arxiv.org/abs/2310.15006)). The + project's version controlled source is on + [Codeberg](https://codeberg.org/gnuastro/papers) (the + `pointing-simulate` branch). + - Borkowska & Roukema + ([2022](https://scixplorer.org/abs/2022CQGra..39u5007B), Classical and + Quantum Gravity, Volume 39, Issue 21, id.215007, 29 pp; + arXiv:[2112.14174](https://arxiv.org/abs/2112.14174)): The live version + of the controlled source is [at + Codeberg](https://codeberg.org/boud/gevcurvtest); the main input + dataset, a software snapshot, the software tarballs, the project outputs + and editing history are available at + [zenodo.5806027](https://doi.org/10.5281/zenodo.5806027); and the + archived git history is available at + [swh:1:rev:54398b720ddbac269ede30bf1e27fe27f07567f7](https://archive.softwareheritage.org/browse/revision/54398b720ddbac269ede30bf1e27fe27f07567f7). + - Peper & Roukema ([2021](https://scixplorer.org/abs/2021MNRAS.505.1223P), + MNRAS, 505, 1223; arXiv:[2010.03742](https://arxiv.org/abs/2010.03742)): + The live version of the controlled source is [at + Codeberg](https://codeberg.org/boud/elaphrocentre); the main input + dataset, a software snapshot, the software tarballs, the project outputs + and editing history are available at + [zenodo.4699702](https://zenodo.org/record/4699702); and the archived + git history is available at + [swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e](https://archive.softwareheritage.org/swh:1:rev:a029edd32d5cd41dbdac145189d9b1a08421114e). + - Roukema ([2021](https://scixplorer.org/abs/2021PeerJ...911856R), PeerJ, + Vol. 9, article id. e11856; + arXiv:[2007.11779](https://arxiv.org/abs/2007.11779)): The live version + of the controlled source is [at + Codeberg](https://codeberg.org/boud/subpoisson); the main input dataset, + a software snapshot, the software tarballs, the project outputs and + editing history are available at + [zenodo.4765705](https://zenodo.org/record/4765705); and the archived + git history is available at + [swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37](https://archive.softwareheritage.org/swh:1:rev:72242ca8eade9659031ea00394a30e0cc5cc1c37). + - Akhlaghi et al. ([2021](https://scixplorer.org/abs/2021CSE....23c..82A), + CiSE, 23(3), 82; arXiv:[2006.03018](https://arxiv.org/abs/2006.03018)): + The project's version controlled source is [on + Gitlab](https://gitlab.com/makhlaghi/maneage-paper), necessary software, + outputs and backup of history are available at + [zenodo.3872248](https://doi.org/10.5281/zenodo.3872248); and the + archived git history is available at + [swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717](https://archive.softwareheritage.org/swh:1:dir:45a9e282a86145fe9babef529c8fce52ffe8d717). + - Infante-Sainz et + al. ([2020](https://scixplorer.org/abs/2020MNRAS.491.5317I), MNRAS, 491, + 5317; arXiv:[1911.01430](https://arxiv.org/abs/1911.01430)): The version + controlled project source is available [on + GitLab](https://gitlab.com/infantesainz/sdss-extended-psfs-paper) and is + also archived on Zenodo with all the necessary software tarballs: + [zenodo.3524937](https://zenodo.org/record/3524937). + - Akhlaghi ([2019](https://scixplorer.org/abs/2019arXiv190911230A), IAU + Symposium 355; + arXiv:[1909.11230](https://arxiv.org/abs/1909.11230)). The version + controlled project source is available [on + GitLab](https://gitlab.com/makhlaghi/iau-symposium-355) and is also + archived on Zenodo with all the necessary software tarballs: + [zenodo.3408481](https://doi.org/10.5281/zenodo.3408481). + +The items below were the precursors to Maneage. In other words, the work to +have reproducible results in the items below lead to what is now +Maneage. They are therefore much less complete than what Maneage is today, +but are important from a historical perspective. + - Section 7.3 of Bacon et + al. ([2017](https://scixplorer.org/abs/2017A%26A...608A...1B), A&A 608, + A1; arXiv:[1710.03002](https://arxiv.org/abs/1710.03002)): The version + controlled project source is available [on + GitLab](https://gitlab.com/makhlaghi/muse-udf-origin-only-hst-magnitudes) + and a snapshot of the project along with all the necessary input + datasets and outputs is available in + [zenodo.1164774](https://doi.org/10.5281/zenodo.1164774). + - Section 4 of Bacon et + al. ([2017](https://scixplorer.org/abs/2017A%26A...608A...1B), A&A, 608, + A1; arXiv:[1710.03002](https://arxiv.org/abs/1710.03002)): The version + controlled project is available [on + GitLab](https://gitlab.com/makhlaghi/muse-udf-photometry-astrometry) and + a snapshot of the project along with all the necessary input datasets is + available in [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746). + - Akhlaghi & Ichikawa + ([2015](https://scixplorer.org/abs/2015ApJS..220....1A), ApJS, 220, 1; + arXiv:[1505.01664](https://arxiv.org/abs/1505.01664)): The version + controlled project is available [on + GitLab](https://gitlab.com/makhlaghi/NoiseChisel-paper). This is the + very first (and much less mature!) incarnation of Maneage: the history + of Maneage started more than two years after this paper was + published. It is a very rudimentary/initial implementation, thus it is + only included here for historical reasons. However, the project source + is complete, accurate and uploaded to arXiv along with the paper. -Tips for designing your project -=============================== -The following is a list of design points, tips, or recommendations that -have been learned after some experience with this type of project -management. Please don't hesitate to share any experience you gain after -using it with us. In this way, we can add it here (with full giving credit) -for the benefit of others. - - **Modularity**: Modularity is the key to easy and clean growth of a - project. So it is always best to break up a job into as many - sub-components as reasonable. Here are some tips to stay modular. - - - *Short recipes*: if you see the recipe of a rule becoming more than a - handful of lines which involve significant processing, it is probably - a good sign that you should break up the rule into its main - components. Try to only have one major processing step per rule. - - - *Context-based (many) Makefiles*: For maximum modularity, this design - allows easy inclusion of many Makefiles: in - `reproduce/analysis/make/*.mk` for analysis steps, and - `reproduce/software/make/*.mk` for building software. So keep the - rules for closely related parts of the processing in separate - Makefiles. - - - *Descriptive names*: Be very clear and descriptive with the naming of - the files and the variables because a few months after the - processing, it will be very hard to remember what each one was - for. Also this helps others (your collaborators or other people - reading the project source after it is published) to more easily - understand your work and find their way around. - - - *Naming convention*: As the project grows, following a single standard - or convention in naming the files is very useful. Try best to use - multiple word filenames for anything that is non-trivial (separating - the words with a `-`). For example if you have a Makefile for - creating a catalog and another two for processing it under models A - and B, you can name them like this: `catalog-create.mk`, - `catalog-model-a.mk` and `catalog-model-b.mk`. In this way, when - listing the contents of `reproduce/analysis/make` to see all the - Makefiles, those related to the catalog will all be close to each - other and thus easily found. This also helps in auto-completions by - the shell or text editors like Emacs. - - - *Source directories*: If you need to add files in other languages for - example in shell, Python, AWK or C, keep the files in the same - language in a separate directory under `reproduce/analysis`, with the - appropriate name. - - - *Configuration files*: If your research uses special programs as part - of the processing, put all their configuration files in a devoted - directory (with the program's name) within - `reproduce/software/config`. It is much cleaner and readable (thus - less buggy) to avoid mixing the configuration files, even if there is - no technical necessity. - - - - **Contents**: It is good practice to follow the following - recommendations on the contents of your files, whether they are source - code for a program, Makefiles, scripts or configuration files - (copyrights aren't necessary for the latter). - - - *Copyright*: Always start a file containing programming constructs - with a copyright statement like the ones that Maneage starts with - (for example in the top level `Makefile`). - - - *Comments*: Comments are vital for readability (by yourself in two - months, or others). Describe everything you can about why you are - doing something, how you are doing it, and what you expect the result - to be. Write the comments as if it was what you would say to describe - the variable, recipe or rule to a friend sitting beside you. When - writing the project it is very tempting to just steam ahead with - commands and codes, but be patient and write comments before the - rules or recipes. This will also allow you to think more about what - you should be doing. Also, in several months when you come back to - the code, you will appreciate the effort of writing them. Just don't - forget to also read and update the comment first if you later want to - make changes to the code (variable, recipe or rule). As a general - rule of thumb: first the comments, then the code. - - - *File title*: In general, it is good practice to start all files with - a single line description of what that particular file does. If - further information about the totality of the file is necessary, add - it after a blank line. This will help a fast inspection where you - don't care about the details, but just want to remember/see what that - file is (generally) for. This information must of course be commented - (its for a human), but this is kept separate from the general - recommendation on comments, because this is a comment for the whole - file, not each step within it. - - - - **Make programming**: Here are some experiences that we have come to - learn over the years in using Make and are useful/handy in research - contexts. - - - *Environment of each recipe*: If you need to define a special - environment (or aliases, or scripts to run) for all the recipes in - your Makefiles, you can use a Bash startup file - `reproduce/software/shell/bashrc.sh`. This file is loaded before every - Make recipe is run, just like the `.bashrc` in your home directory is - loaded every time you start a new interactive, non-login terminal. See - the comments in that file for more. - - - *Automatic variables*: These are wonderful and very useful Make - constructs that greatly shrink the text, while helping in - read-ability, robustness (less bugs in typos for example) and - generalization. For example even when a rule only has one target or - one prerequisite, always use `$@` instead of the target's name, `$<` - instead of the first prerequisite, `$^` instead of the full list of - prerequisites and etc. You can see the full list of automatic - variables - [here](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html). If - you use GNU Make, you can also see this page on your command-line: - - ```shell - $ info make "automatic variables" - ``` - - - *Debug*: Since Make doesn't follow the common top-down paradigm, it - can be a little hard to get accustomed to why you get an error or - un-expected behavior. In such cases, run Make with the `-d` - option. With this option, Make prints a full list of exactly which - prerequisites are being checked for which targets. Looking - (patiently) through this output and searching for the faulty - file/step will clearly show you any mistake you might have made in - defining the targets or prerequisites. - - - *Large files*: If you are dealing with very large files (thus having - multiple copies of them for intermediate steps is not possible), one - solution is the following strategy (Also see the next item on "Fast - access to temporary files"). Set a small plain text file as the - actual target and delete the large file when it is no longer needed - by the project (in the last rule that needs it). Below is a simple - demonstration of doing this. In it, we use Gnuastro's Arithmetic - program to add all pixels of the input image with 2 and create - `large1.fits`. We then subtract 2 from `large1.fits` to create - `large2.fits` and delete `large1.fits` in the same rule (when its no - longer needed). We can later do the same with `large2.fits` when it - is no longer needed and so on. - ``` - large1.fits.txt: input.fits - astarithmetic $< 2 + --output=$(subst .txt,,$@) - echo "done" > $@ - large2.fits.txt: large1.fits.txt - astarithmetic $(subst .txt,,$<) 2 - --output=$(subst .txt,,$@) - rm $(subst .txt,,$<) - echo "done" > $@ - ``` - A more advanced Make programmer will use Make's [call - function](https://www.gnu.org/software/make/manual/html_node/Call-Function.html) - to define a wrapper in `reproduce/analysis/make/initialize.mk`. This - wrapper will replace `$(subst .txt,,XXXXX)`. Therefore, it will be - possible to greatly simplify this repetitive statement and make the - code even more readable throughout the whole project. - - - *Fast access to temporary files*: Most Unix-like operating systems - will give you a special shared-memory device (directory): on systems - using the GNU C Library (all GNU/Linux system), it is `/dev/shm`. The - contents of this directory are actually in your RAM, not in your - persistence storage like the HDD or SSD. Reading and writing from/to - the RAM is much faster than persistent storage, so if you have enough - RAM available, it can be very beneficial for large temporary files to - be put there. You can use the `mktemp` program to give the temporary - files a randomly-set name, and use text files as targets to keep that - name (as described in the item above under "Large files") for later - deletion. For example, see the minimal working example Makefile below - (which you can actually put in a `Makefile` and run if you have an - `input.fits` in the same directory, and Gnuastro is installed). - ``` - .ONESHELL: - .SHELLFLAGS = -ec - all: mean-std.txt - shm-maneage := /dev/shm/$(shell whoami)-maneage-XXXXXXXXXX - large1.txt: input.fits - out=$$(mktemp $(shm-maneage)) - astarithmetic $< 2 + --output=$$out.fits - echo "$$out" > $@ - large2.txt: large1.txt - input=$$(cat $<) - out=$$(mktemp $(shm-maneage)) - astarithmetic $$input.fits 2 - --output=$$out.fits - rm $$input.fits $$input - echo "$$out" > $@ - mean-std.txt: large2.txt - input=$$(cat $<) - aststatistics $$input.fits --mean --std > $@ - rm $$input.fits $$input - ``` - The important point here is that the temporary name template - (`shm-maneage`) has no suffix. So you can add the suffix - corresponding to your desired format afterwards (for example - `$$out.fits`, or `$$out.txt`). But more importantly, when `mktemp` - sets the random name, it also checks if no file exists with that name - and creates a file with that exact name at that moment. So at the end - of each recipe above, you'll have two files in your `/dev/shm`, one - empty file with no suffix one with a suffix. The role of the file - without a suffix is just to ensure that the randomly set name will - not be used by other calls to `mktemp` (when running in parallel) and - it should be deleted with the file containing a suffix. This is the - reason behind the `rm $$input.fits $$input` command above: to make - sure that first the file with a suffix is deleted, then the core - random file (note that when working in parallel on powerful systems, - in the time between deleting two files of a single `rm` command, many - things can happen!). When using Maneage, you can put the definition - of `shm-maneage` in `reproduce/analysis/make/initialize.mk` to be - usable in all the different Makefiles of your analysis, and you won't - need the three lines above it. **Finally, BE RESPONSIBLE:** after you - are finished, be sure to clean up any possibly remaining files (due - to crashes in the processing while you are working), otherwise your - RAM may fill up very fast. You can do it easily with a command like - this on your command-line: `rm -f /dev/shm/$(whoami)-*`. - - - - **Software tarballs and raw inputs**: It is critically important to - document the raw inputs to your project (software tarballs and raw - input data): - - - *Keep the source tarball of dependencies*: After configuration - finishes, the `.build/software/tarballs` directory will contain all - the software tarballs that were necessary for your project. You can - mirror the contents of this directory to keep a backup of all the - software tarballs used in your project (possibly as another version - controlled repository) that is also published with your project. Note - that software web-pages are not written in stone and can suddenly go - offline or not be accessible in some conditions. This backup is thus - very important. If you intend to release your project in a place like - Zenodo, you can upload/keep all the necessary tarballs (and data) - there with your - project. [zenodo.1163746](https://doi.org/10.5281/zenodo.1163746) is - one example of how the data, Gnuastro (main software used) and all - major Gnuastro's dependencies have been uploaded with the project's - source. Just note that this is only possible for [free - software](https://www.gnu.org/philosophy/free-sw.html). - - - *Keep your input data*: The input data is also critical to the - project's reproducibility, so like the above for software, make sure - you have a backup of them, or their persistent identifiers (PIDs). - - - **Version control**: Version control is a critical component of - Maneage. Here are some tips to help in effectively using it. - - - *Regular commits*: It is important (and extremely useful) to have the - history of your project under version control. So try to make commits - regularly (after any meaningful change/step/result). - - - *Commit message*: The commit message is a very important and useful - aspect of version control. To make the commit message useful for - others (or yourself, one year later), it is good to follow a - consistent style. Maneage already has a consistent formatting - (described below), which you can also follow in your project if you - like. You can see many examples by running `git log` in the `maneage` - branch. If you intend to push commits to Maneage, for the consistency - of Maneage, it is necessary to follow these guidelines. 1) No line - should be more than 75 characters (to enable easy reading of the - message when you run `git log` on the standard 80-character - terminal). 2) The first line is the title of the commit and should - summarize it (so `git log --oneline` can be useful). The title should - also not end with a point (`.`, because its a short single sentence, - so a point is not necessary and only wastes space). 3) After the - title, leave an empty line and start the body of your message - (possibly containing many paragraphs). 4) Describe the context of - your commit (the problem it is trying to solve) as much as possible, - then go onto how you solved it. One suggestion is to start the main - body of your commit with "Until now ...", and continue describing the - problem in the first paragraph(s). Afterwards, start the next - paragraph with "With this commit ...". - - - *Project outputs*: During your research, it is possible to checkout a - specific commit and reproduce its results. However, the processing - can be time consuming. Therefore, it is useful to also keep track of - the final outputs of your project (at minimum, the paper's PDF) in - important points of history. However, keeping a snapshot of these - (most probably large volume) outputs in the main history of the - project can unreasonably bloat it. It is thus recommended to make a - separate Git repo to keep those files and keep your project's source - as small as possible. For example if your project is called - `my-exciting-project`, the name of the outputs repository can be - `my-exciting-project-output`. This enables easy sharing of the output - files with your co-authors (with necessary permissions) and not - having to bloat your email archive with extra attachments also (you - can just share the link to the online repo in your - communications). After the research is published, you can also - release the outputs repository, or you can just delete it if it is - too large or un-necessary (it was just for convenience, and fully - reproducible after all). For example Maneage's output is available - for demonstration in [a - separate](http://git.maneage.org/output-raw.git/) repository. - - - *Full Git history in one file*: When you are publishing your project - (for example to Zenodo for long term preservation), it is more - convenient to have the whole project's Git history into one file to - save with your datasets. After all, you can't be sure that your - current Git server (for example GitLab, Github, or Bitbucket) will be - active forever. While they are good for the immediate future, you - can't rely on them for archival purposes. Fortunately keeping your - whole history in one file is easy with Git using the following - commands. To learn more about it, run `git help bundle`. - - - "bundle" your project's history into one file (just don't forget to - change `my-project-git.bundle` to a descriptive name of your - project): - - ```shell - $ git bundle create my-project-git.bundle --all - ``` - - - You can easily upload `my-project-git.bundle` anywhere. Later, if - you need to un-bundle it, you can use the following command. - - ```shell - $ git clone my-project-git.bundle - ``` - - - - - -Maneage Development Workflow -============================ -Maneage users can only use it by developing it, therefore Maneage is a very -unique software: the boundary between a user and developer is very blurry -by design/philosophy. We hope that all users can contribute to the -`maneage` branch for project-agnostic bugs/features that they fix/add. In -this section, the development workflow is described: +# Maneage Development + +This section describes Maneage's own development workflow, not the workflow +of your Maneage-based projects (where you are free to select any workflow +model that you prefer). So it is only intended for those who would like to +contribute to the core Maneage project, not those who only develop their +own project. + +## Development workflow + +Maneage users can only use it by developing it, therefore Maneage is a very +unique software: the boundary between a user and developer is blurry by +design/philosophy. We hope that all users can contribute to the `maneage` +branch for project-agnostic bugs/features that they fix/add. In this +section, the development workflow is described: - At any point that you find a bug or want to share a fix, anyone is welcome to open a "Bug" or "Task" on GNU Savannah (which is the main project management interface of Maneage). In case the submitter already has a commit to merge, they can fork Maneage (similar to [this forking tutorial](https://www.gnu.org/software/gnuastro/manual/html_node/Forking-tutorial.html)) and share their commit in the Savannah post. - - The review of bugs and merge requests will happen at the earliest possible time for the maintainer. At this point, the maintainer will create a new development branch over the `maneage` branch in a @@ -1882,40 +2057,41 @@ this section, the development workflow is described: the commits in that branch will be `squash`ed into a single commit that will be merged over the `maneage` branch. Of course, every contributor will be acknowledged in the squashed commit for their contribution. - - The `squash`ing into a single commit is intentionally done because the `maneage` branch is shared between all projects that use Maneage and we do not want to introduce complexity in their history by preserving all the various development commits that were done before the merge. - - After the merge, the development branch may be deleted from the development repository: any historically relevant point that you find when making a commit should be written as comments within the source (to be preserved during the squash) and not in the commit message. - - Checklist for the maintainer: - - Make sure all contributors are properly acknowledged. - - The final commit amend (done to edit the final text of the commit message) should be like this: `git commit --amend --date=now`. This helps have a date on the commit that matches the merge (independent of which development commit was squashed first). - - - - -Future improvements -=================== +## Future improvements This is an evolving project and as time goes on, it will evolve and become more robust. Some of the most prominent issues we plan to implement in the future are listed below, please join us if you are interested. -Package management ------------------- +### Full documentation in Info + +Currently the only documentation for Maneage's development is this file! +But this format is not too convenient for a complete documentation: README +files are only meant to point readers to the proper place of the manual, +not be the whole documentation themselves! This choice was due to time +constraints. Work has already started on this in the +`[texibook](https://gitlab.com/maneage/project-dev/-/tree/texibook)` branch +that will build a professional manual in info, HTML and PDF formats using +Texinfo during the configuration phase. But this still needs to be +finished. + +### Package management It is important to have control of the environment of the project. Maneage currently builds the higher-level programs (for example GNU Bash, GNU Make, @@ -1962,46 +2138,15 @@ commonly have large overlaps in lower-level programs). -Appendix: Necessity of exact reproduction in scientific research -================================================================ - -In case [the link above](http://akhlaghi.org/reproducible-science.html) is -not accessible at the time of reading, here is a copy of the introduction -of that link, describing the necessity for a reproducible project like this -(copied on February 7th, 2018): - -The most important element of a "scientific" statement/result is the fact -that others should be able to falsify it. The Tsunami of data that has -engulfed astronomers in the last two decades, combined with faster -processors and faster internet connections has made it much more easier to -obtain a result. However, these factors have also increased the complexity -of a scientific analysis, such that it is no longer possible to describe -all the steps of an analysis in the published paper. Citing this -difficulty, many authors suffice to describing the generalities of their -analysis in their papers. - -However, It is impossible to falsify (or even study) a result if you can't -exactly reproduce it. The complexity of modern science makes it vitally -important to exactly reproduce the final result. Because even a small -deviation can be due to many different parts of an analysis. Nature is -already a black box which we are trying so hard to comprehend. Not letting -other scientists see the exact steps taken to reach a result, or not -allowing them to modify it (do experiments on it) is a self-imposed black -box, which only exacerbates our ignorance. - -Other scientists should be able to reproduce, check and experiment on the -results of anything that is to carry the "scientific" label. Any result -that is not reproducible (due to incomplete information by the author) is -not scientific: the readers have to have faith in the subjective experience -of the authors in the very important choice of configuration values and -order of operations: this is contrary to the scientific spirit. +# Copyright information +Contributors:\ +Copyright (C) 2020-2026 Raul Infante-Sainz \ +Copyright (C) 2021-2026 Boud Roukema -Copyright information ---------------------- This file is part of Maneage (https://maneage.org). This file is free software: you can redistribute it and/or modify it -- cgit v1.2.1