aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/make/atlas-single.mk
AgeCommit message (Collapse)AuthorLines
2021-01-02Copyright year updated in all source filesMohammad Akhlaghi-1/+1
Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.
2020-06-27IMPORTANT: many improvements to low-level software building phaseMohammad Akhlaghi-0/+0
POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause conflicts to your project if you have changed the software building Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and 'python.mk'). If your project has only added analysis, it shouldn't be affected. This is a large commit, involving a long series of corrections in a differnt branch which is now finally being merged into the core Maneage branch. All changes were related and came up naturally as the low-level infrastructure was improved. So separating them in the end for the final merge would have been very time consuming and we are merging them as one commit. In general, the software building Makefiles are now much more easier to read, modify and use, along with several new features that have been added. See below for the full list. - Until now, Maneage needed the host to have a 'make' implementation because Make was necessary to build Lzip. Lzip is then used to uncompress the source of our own GNU Make. However, in the minimalist/slim versions of operating systems (for example used to build Docker images) Make isn't included by default. Since Lzip was the only program before our own GNU Make was installed, we consulting Antonio Diaz Diaz (creator of Lzip) and he kindly added the necessary functionality to a new version of Lzip, which we are using now. Hence we don't need to assume a Make implementation on the host any more. With this commit, Lzip and GNU Make are built without Make, allowing everything else to be safely built with our own custom version of GNU Make and not using the host's 'make' at all. - Until recently (Commit 3d8aa5953c4) GNU Make was built in 'basic.mk'. Therefore 'basic.mk' was written in a way that it can be used with other 'make' implementations also (i.e., important shell commands starting with '&&' and ending in '\' without any comments between them!). Furthermore, to help in style uniformity, the rules in 'high-level.mk' and 'python.mk' also followed a similar structure. But due to the point above, we can now guarantee that GNU Make is used from the very first Makefile, so this hard-to-read structure has been removed in the software build recipes and they are much more readable and edit-friendly now. - Until now, the default backup servers where at some fixed URLs, on our own pages or on Gitlab. But recently we uploaded all the necessary software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is more suitable for this task (it promises longevity, has a fixed DOI, while allowing us to add new content, or new software tarball versions). With this commit, a small script has been written to extract the most recent Zenodo upload link from the Zenodo DOI and use it for downloading the software source codes. - Until now, we primarily used the webpage of each software for downloading its tarball. But this caused many problems: 1) Some of them needed Javascript before the download, 2) Some URLs had a complex dependency on the version number, 3) some servers would be randomly down for maintenance and etc. So thanks to the point above, we now use the Zenodo server as the primary download location. However, if a user wants to use a custom software that is not (yet!) in Zenodo, the download script gives priority to a custom URL that the users can give as Make variables. If that variable is defined, then the script will use that URL before going onto Zenodo. We now have a special place for such URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a good documentation themselves) are preserved here, but are commented by default. - The software source code downloading and checksum verification step has been moved into a Make function called 'import-source' (defined in the 'build-rules.mk' and loaded in all software Makefiles). Having taken all the low-level steps there, I noticed that there is no more need for having the tarball as a separate target! So with this commit, a single rule is the only place that needs to be edited/added (greatly simplifying the software building Makefiles). - Following task #15272, A new option has been added to the './project' script called '--all-highlevel'. When this option is given, the contents of 'TARGETS.conf' are ignored and all the software in Maneage are built (selected by parsing the 'versions.conf' file). This new option was added to confirm the extensive changes made in all the software building recipes and is great for development/testing purposes. - Many of the software hadn't been tested for a long time! So after using the newly added '--all-highlevel', we noticed that some need to be updated. In general, with this commit, 'libpaper' and 'pcre' were added as new software, and the versions of the following software was updated: 'boost', 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in' shell script was added in 'reproduce/software/shell/' which is installed with 'libpaper'. - Even though we intentionally add the necessary flags to add RPATH inside the built executable at compilation time, some software don't do it (different software on different operating systems!). Until now, for historical reasons this check was done in different ways for different software on GNU/Linux sytems. But now it is unified: if 'patchelf' is present we apply it. Because of this, 'patchelf' has been put as a top-level prerequisite, right after Tar and is installed before anything else. - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in 'basic.mk', it was 'glibtool'! This caused many confusions and is corrected with this commit (in 'basic.mk', it is also 'libtool'). - A new argument is added to the './project' script to allow easy loading of the project's shell and environment for fast/temporary testing of things in the same environment as the project. Before activating the project's shell, we completely remove all host environment variables to simulate the project's environment. It can be called with this command: './project shell'. A simple prompt has also been added to highlight that the user is using the Maneage shell!
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-8/+11
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-04-20Maneage instead of Template in README-hacking.md and copyright noticesMohammad Akhlaghi-11/+8
Until now, throughout Maneage we were using the old name of "Reproducible Paper Template". But we have finally decided to use Maneage, so to avoid confusion, the name has been corrected in `README-hacking.md' and also in the copyright notices. Note also that in `README-hacking.md', the main Maneage branch is now called `maneage', and the main Git remote has been changed to `https://gitlab.com/maneage/project' (this is a new GitLab Group that I have setup for all Maneage-related projects). In this repository there is only one `maneage' branch to avoid complications with the `master' branch of the projects using Maneage later.
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-1/+1
Now that its 2020, its necessary to include this year in the copyright statements.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-2/+2
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-04-15New architecture to separate software-building and analysis stepsMohammad Akhlaghi-0/+54
Until now, the software building and analysis steps of the pipeline were intertwined. However, these steps (of how to build a software, and how to use it) are logically completely independent. Therefore with this commit, the pipeline now has a new architecture (particularly in the `reproduce' directory) to emphasize this distinction: The `reproduce' directory now has the two `software' and `analysis' subdirectories and the respective parts of the previous architecture have been broken up between these two based on their function. There is also no more `src' directory. The `config' directory for software and analysis is now mixed with the language-specific directories. Also, some of the software versions were also updated after some checks with their webpages. This new architecture will allow much more focused work on each part of the pipeline (to install the software and to run them for an analysis).