From 6ef4cc854d1df46b719de5d66b45537b0aa11f92 Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Sun, 28 Jul 2019 02:32:20 +0100 Subject: Single wrapper instead of old ./configure, Makefile and ./for-group Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682. --- .file-metadata | Bin 6106 -> 6239 bytes README-hacking.md | 140 +- README.md | 12 +- configure | 1371 -------------------- for-group | 114 -- project | 291 +++++ reproduce/analysis/make/initialize.mk | 20 +- reproduce/analysis/make/paper.mk | 6 +- reproduce/analysis/make/top.mk | 12 +- reproduce/software/bash/configure.sh | 1224 +++++++++++++++++ reproduce/software/config/installation/LOCAL.mk.in | 4 +- reproduce/software/make/atlas-multiple.mk | 4 +- reproduce/software/make/atlas-single.mk | 4 +- reproduce/software/make/basic.mk | 10 +- reproduce/software/make/high-level.mk | 13 +- reproduce/software/make/python.mk | 4 +- 16 files changed, 1634 insertions(+), 1595 deletions(-) delete mode 100755 configure delete mode 100755 for-group create mode 100755 project create mode 100755 reproduce/software/bash/configure.sh diff --git a/.file-metadata b/.file-metadata index af859e9..1e4b3c5 100644 Binary files a/.file-metadata and b/.file-metadata differ diff --git a/README-hacking.md b/README-hacking.md index 375bf85..cf2e332 100644 --- a/README-hacking.md +++ b/README-hacking.md @@ -240,10 +240,10 @@ first understand its architecture so you can navigate your way in the directories and understand how to implement your research project within its framework: where to add new files and which existing files to modify for what purpose. But before reading this theoretical discussion, please -run the template (described in `README.md`: first run `./configure`, then -`.local/bin/make -j8`) without any change, just to see how it works (note -that the configure step builds all necessary software, so it can take long, -but you can read along while its working). +run the template (described in `README.md`: first run `./project +configure`, then `./project make -j8`) without any change, just to see how +it works (note that the configure step builds all necessary software, so it +can take long, but you can read along while its working). The project has two top-level directories: `reproduce` and `tex`. `reproduce` hosts all the software building and analysis @@ -253,24 +253,21 @@ a PDF using LaTeX. The `reproduce` directory has two sub-directories: `software` and `analysis`. As the name says, the former contains all the instructions to download, build and install (independent of the host operating system) the -necessary software (these are called by the `./configure` script). The -latter contains instructions on how to use those software to do your -project's analysis. - -After it finishes, `./configure` will create the following symbolic links -in the project's top source directory: 1) `Makefile` in the top directory -(which points to `reproduce/analysis/make/top.mk`). 2) `.build` which -points to the top build directory.And 3) `.local` for easy access to the -custom built software packages installation directory. The first is for -practical necessity (so you can run `make` from the top source directory), -but the latter is just for convenience (fast access to the built outputs -and software). - -Therefore, by running `.local/bin/make` we are doing the project's analysis -with its own custom version of GNU Make, not the host system's Make. The -first file that is read by Make (the template's starting point) is the -top-level `Makefile` (created by `./configure`). Therefore, we'll start -describing the template's architecture with this file. This file is +necessary software (these are called by the `./project configure` +command). The latter contains instructions on how to use those software to +do your project's analysis. + +After it finishes, `./project configure` will create the following symbolic +links in the project's top source directory: `.build` which points to the +top build directory and `.local` for easy access to the custom built +software installation directory. + +Once the project is configured for your system, `./project make` will doing +the project's analysis with its own custom version of software. The process +is managed through Make and `./project make` will start with +`reproduce/analysis/make/top.mk` (called `top.mk` from now on). + +Let's continue the template's architecture with this file. `top.mk` is relatively short and heavily commented so hopefully the descriptions in each comment will be enough to understand the general details. As you read this section, please also look at the contents of the mentioned files and @@ -286,13 +283,13 @@ strategy to deal with large/huge files). To keep the source and (intermediate) built files separate, you _must_ define a top-level build directory variable (or `$(BDIR)`) to host all the -intermediate files (you defined it during `./configure`). This directory -doesn't need to be version controlled or even synchronized, or backed-up in -other servers: its contents are all products, and can be easily re-created -any time. As you define targets for your new rules, it is thus important to -place them all under sub-directories of `$(BDIR)`. As mentioned above, you -always have fast access to this "build"-directory with the `.build` -symbolic link. +intermediate files (you defined it during `./project configure`). This +directory doesn't need to be version controlled or even synchronized, or +backed-up in other servers: its contents are all products, and can be +easily re-created any time. As you define targets for your new rules, it is +thus important to place them all under sub-directories of `$(BDIR)`. As +mentioned above, you always have fast access to this "build"-directory with +the `.build` symbolic link. In this architecture, we have two types of Makefiles that are loaded into the top `Makefile`: _configuration-Makefiles_ (only independent @@ -301,13 +298,13 @@ actually contain analysis/processing rules). The configuration-Makefiles are those that satisfy these two wildcards: `reproduce/software/config/installation/*.mk` (for building the necessary -software when you run `./configure`) and `reproduce/analysis/config/*.mk` -(for the high-level analysis, when you run `.local/bin/make`). These -Makefiles don't actually have any rules, they just have values for various -free parameters throughout the configuration or analysis. Open a few of -them to see for yourself. These Makefiles must only contain raw Make -variables (project configurations). By "raw" we mean that the Make -variables in these files must not depend on variables in any other +software when you run `./project configure`) and +`reproduce/analysis/config/*.mk` (for the high-level analysis, when you run +`./project make`). These Makefiles don't actually have any rules, they just +have values for various free parameters throughout the configuration or +analysis. Open a few of them to see for yourself. These Makefiles must only +contain raw Make variables (project configurations). By "raw" we mean that +the Make variables in these files must not depend on variables in any other configuration-Makefile. This is because we don't want to assume any order in reading them. It is also very important to *not* define any rule, or other Make construct, in these configuration-Makefiles. @@ -343,18 +340,17 @@ users of a Unix group (when working on large computer clusters). In this scenario, each user can have their own cloned project source, but share the large built files between each other. To do this, it is necessary for all built files to give full permission to group members while not allowing any -other users access to the contents. Therefore the `./configure` and Make -steps must be called with special conditions which are managed in the -`for-group` script. - -Let's see how this design is implemented. The `./configure` script's final -step is to put a `Makefile` in the top directory. This allows us to start -"making" the project. Please open and inspect it as we go along here. The -first step (un-commented line) is to import the local configuration -(answers to the questions `./configure` asked you). They are defined in the -configuration-Makefile `reproduce/software/config/installation/LOCAL.mk` -which was also built by `./configure` (based on the `LOCAL.mk.in` template -of the same directory). +other users access to the contents. Therefore the `./project configure` and +`./project make` steps must be called with special conditions which are +managed in the `--group` option. + +Let's see how this design is implemented. Please open and inspect `top.mk` +it as we go along here. The first step (un-commented line) is to import the +local configuration (your answers to the questions of `./project +configure`). They are defined in the configuration-Makefile +`reproduce/software/config/installation/LOCAL.mk` which was also built by +`./project configure` (based on the `LOCAL.mk.in` template of the same +directory). The next non-commented set of the top `Makefile` defines the ultimate target of the whole project (`paper.pdf`). But to avoid mistakes, a sanity @@ -450,10 +446,10 @@ Customization checklist ======================= Take the following steps to fully customize this template for your research -project. After finishing the list, be sure to run `./configure` and `make` -to see if everything works correctly before expanding it. If you notice -anything missing or any in-correct part (probably a change that has not -been explained here), please let us know to correct it. +project. After finishing the list, be sure to run `./project configure` and +`project make` to see if everything works correctly. If you notice anything +missing or any in-correct part (probably a change that has not been +explained here), please let us know to correct it. As described above, the concept of reproducibility (during a project) heavily relies on [version @@ -487,21 +483,21 @@ get more advanced in later stages of your work. - **Test the template**: Before making any changes, it is important to test it and see if everything works properly with the commands - below. If there is any problem in the `./configure` or `make` steps, - please contact us to fix the problem before continuing. Since the - building of dependencies in `./configure` can take long, you can take - the next few steps (editing the files) while its working (they don't - affect the configuration). After `make` is finished, open `paper.pdf` - and if it looks fine, you are ready to start customizing the template - for your project. But before that, clean all the extra template - outputs with `make clean` as shown below. + below. If there is any problem in the `./project configure` or + `./project make` steps, please contact us to fix the problem before + continuing. Since the building of dependencies in configuration can + take long, you can take the next few steps (editing the files) while + its working (they don't affect the configuration). After `./project + make` is finished, open `paper.pdf`. If it looks fine, you are ready + to start customizing the template for your project. But before that, + clean all the extra template outputs with `make clean` as shown below. ```shell - $ ./configure --host-cc # Set top directories and build dependencies (except for GCC which can take long). - $ .local/bin/make # Do the (mainly symbolic) processing and build paper + $ ./project configure --host-cc # Configure project (except for GCC which can take long). + $ ./project make # Do the (mainly symbolic) processing and build paper # Open 'paper.pdf' and see if everything is ok. - $ .local/bin/make clean # Delete high-level outputs. + $ ./project make clean # Delete high-level outputs (keep software) ``` - **Setup the remote**: You can use any [hosting @@ -636,8 +632,8 @@ get more advanced in later stages of your work. commit to be sure it works as expected). ```shell - $ .local/bin/make clean # Delete outputs ('make distclean' for everything) - $ .local/bin/make # Build the project to ensure everything is fine. + $ ./project make clean # Delete outputs ('make distclean' for everything) + $ ./project make # Build the project to ensure everything is fine. $ git add -u # Stage all the changes. $ git status # Make sure everything is fine. $ git commit # Your first commit, add a nice description. @@ -711,7 +707,7 @@ get more advanced in later stages of your work. script will crash). To do that, just modify the years in `reproduce/software/config/installation/texlive.conf`, then delete `.build/software/tarballs/install-tl-unx.tar.gz`. The next time you - run `./configure`, the new TeXLive will be installed and used. + run `./project configure`, the new TeXLive will be installed and used. - **Pre-publication: add notice on reproducibility**: Add a notice somewhere prominent in the first page within your paper, informing the @@ -967,12 +963,12 @@ for the benefit of others. [reproducible-paper-output](https://gitlab.com/makhlaghi/reproducible-paper-output) repository. - - **Inspecting status**: When you run `./configure`, several programs and - libraries start to get configured and build (in many cases, - simultaneously). To understand the building process, or for debugging a - strange situation, it is sometimes useful to know which programs are - being built at every moment. To do this, you can look into the - `.build/software/build-tmp` directory (from the top project + - **Inspecting status**: When you run `./project configure`, several + programs and libraries start to get configured and build (in many + cases, simultaneously). To understand the building process, or for + debugging a strange situation, it is sometimes useful to know which + programs are being built at every moment. To do this, you can look + into the `.build/software/build-tmp` directory (from the top project directory). This temporary directory is only present while building the software. At every moment, it contains the unpacked source tarball directories of the all the packages that are being built. After a diff --git a/README.md b/README.md index 212e178..bb46fe0 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,8 @@ a compressed tarball of the project). ```shell $ git clone XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -$ ./configure -$ .local/bin/make -j8 +$ ./project configure +$ ./project make -j8 ``` For a general introduction to reproducible science as implemented in this @@ -44,8 +44,8 @@ This project was designed to have as few dependencies as possible. 1.1: Minimal software building tools like C compiler, Make, and other tools found on any Unix-like operating system (GNU/Linux, BSD, Mac OS, and others). All necessary dependencies will be built from - source (for use only within this project) by the `./configure' - script (next step). + source (for use only within this project) by the `./project + configure' script (next step). 1.2: (OPTIONAL) Tarball of dependencies. If they are already present (in a directory given at configuration time), they will be @@ -69,7 +69,7 @@ This project was designed to have as few dependencies as possible. programs (project dependencies). So it may take a while to complete. ```shell - $ ./configure + $ ./project configure ``` 3. Run the following command (local build of the Make software) to @@ -79,7 +79,7 @@ This project was designed to have as few dependencies as possible. system by running `./.local/bin/nproc`) ```shell - $ .local/bin/make -j8 + $ ./project make -j8 ``` diff --git a/configure b/configure deleted file mode 100755 index 6dd796c..0000000 --- a/configure +++ /dev/null @@ -1,1371 +0,0 @@ -#! /bin/bash -# -# Necessary preparations/configurations for the reproducible project. -# -# Copyright (C) 2018-2019 Mohammad Akhlaghi -# -# This script is free software: you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the -# Free Software Foundation, either version 3 of the License, or (at your -# option) any later version. -# -# This script is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General -# Public License for more details. -# -# A copy of the GNU General Public License is available at -# . - - -# Script settings -# --------------- -# Stop the script if there are any errors. -set -e - - - - - -# Default option values -jobs=0 -build_dir= -input_dir= -host_cc=0 -software_dir= -existing_conf=0 -minmapsize=10000000000 -scriptname="./configure" - - - - - -# Output of --help -# ---------------- -print_help() { - # Print the output. - cat < "$1" - then - echo "#" >> "$1" - echo "# This file was created during configuration" >> "$1" - echo "# ('./configure'). Therefore, it is not under version" >> "$1" - echo "# control and any manual changes to it will be" >> "$1" - echo "# over-written if the project re-configured." >> "$1" - echo "#" >> "$1" - else - echo; echo "Can't write to $1"; echo; - exit 1 - fi -} - - - - - -# Get absolute address -# -------------------- -# -# Since the build directory will go into a symbolic link, we want it to be -# an absolute address. With this function we can make sure of that. -function absolute_dir() { - if stat "$1" 1> /dev/null; then - echo "$(cd "$(dirname "$1")" && pwd )/$(basename "$1")" - else - exit 1; - fi -} - - - - - -# Inform the user -# --------------- -# -# Print some basic information so the user gets a feeling of what is going -# on and is prepared on what will happen next. -cat < /dev/null 2>/dev/null; then - name=$(which wget) - - # By default Wget keeps the remote file's timestamp, so we'll have - # to disable it manually. - downloader="$name --no-use-server-timestamps -O"; - elif type curl > /dev/null 2>/dev/null; then - name=$(which curl) - - # - cURL doesn't keep the remote file's timestamp by default. - # - With the `-L' option, we tell cURL to follow redirects. - downloader="$name -L -o" - else - cat < /dev/null; then - bdir=$(absolute_dir $build_dir) - echo " -- Build directory: '$bdir'" - rm -rf $build_dir/$junkname - else - echo " -- Can't write in '$build_dir'" - fi - else - if mkdir $build_dir 2> /dev/null; then - bdir=$(absolute_dir $build_dir) - echo " -- Build directory set to (the newly created): '$bdir'" - else - echo " -- Can't create '$build_dir'" - fi - fi - - # Reset `build_dir' to blank, so it continues asking when the - # previous value wasn't usable. - build_dir= - done -fi - - - - - -# Input directory -# --------------- -if [ x"$input_dir" = x ]; then - indir=$optionaldir -else - indir=$input_dir -fi -wfpc2name=$(awk '!/^#/ && $1=="WFPC2IMAGE" {print $3}' $adir/INPUTS.mk) -wfpc2md5=$(awk '!/^#/ && $1=="WFPC2MD5" {print $3}' $adir/INPUTS.mk) -wfpc2size=$(awk '!/^#/ && $1=="WFPC2SIZE" {print $3}' $adir/INPUTS.mk) -wfpc2url=$(awk '!/^#/ && $1=="WFPC2URL" {print $3}' $adir/INPUTS.mk) -if [ $rewritepconfig = yes ] && [ x"$input_dir" = x ]; then - cat <> $pconf -else - # Read the values from existing configuration file. - inbdir=$(awk '$1=="BDIR" {print $3}' $pconf) - - # Read the software directory. - ddir=$(awk '$1=="DEPENDENCIES-DIR" {print $3}' $pconf) - - # The downloader command may contain multiple elements, so we'll just - # change the (in memory) first and second tokens to empty space and - # write the full line (the original file is unchanged). - downloader=$(awk '$1=="DOWNLOADER" {$1=""; $2=""; print $0}' $pconf) - - # Make sure all necessary variables have a value - err=0 - verr=0 - novalue="" - if [ x"$inbdir" = x ]; then novalue="BDIR, "; fi - if [ x"$downloader" = x ]; then novalue="$novalue"DOWNLOADER; fi - if [ x"$novalue" != x ]; then verr=1; err=1; fi - - # Make sure `bdir' is an absolute path and it exists. - berr=0 - ierr=0 - bdir=$(absolute_dir $inbdir) - - if ! [ -d $bdir ]; then if ! mkdir $bdir; then berr=1; err=1; fi; fi - if [ $err = 1 ]; then - cat <> $glconf - echo " minmapsize $minmapsize" >> $glconf - echo >> $glconf - echo "# Version of Gnuastro that must be used." >> $glconf - echo " onlyversion $gversion" >> $glconf -else - ingversion=$(awk '$1=="onlyversion" {print $NF}' $glconf) - if [ x$ingversion != x$gversion ]; then - cat <8GB) is large enough for the parallel building of the software. -# -# For the name of the directory under `/dev/shm' (for this project), we'll -# use the names of the two parent directories to the current/running -# directory, separated by a `-' instead of `/'. We'll then appended that -# with the user's name (in case multiple users may be working on similar -# project names). Maybe later, we can use something like `mktemp' to add -# random characters to this name and make it unique to every run (even for -# a single user). -tmpblddir=$sdir/build-tmp -if ! [ -d $tmpblddir ]; then - - # Set the top-level shared memory location. - if [ -d /dev/shm ]; then shmdir=/dev/shm - else shmdir="" - fi - - # If a shared memory mounted directory exists and there is enough space - # there (in RAM), build a temporary directory for this project. - needed_space=2000000 - if [ x"$shmdir" != x ]; then - available_space=$(df $shmdir | awk 'NR==2{print $4}') - if [ $available_space -gt $needed_space ]; then - dirname=$(pwd | sed -e's/\// /g' \ - | awk '{l=NF-1; printf("%s-%s",$l, $NF)}') - tbshmdir=$shmdir/"$dirname"-$(whoami) - if ! [ -d $tbshmdir ]; then mkdir $tbshmdir; fi - fi - else - tbshmdir="" - fi - - # If a shared memory directory was created set `build-tmp' to be a - # symbolic link to it. Otherwise, just build the temporary build - # directory under the project build directory. - if [ x$tbshmdir = x ]; then mkdir $tmpblddir; - else ln -s $tbshmdir $tmpblddir; - fi -fi - - - - - -# Check for C/C++ compilers -# ------------------------- -# -# To build the software, we'll need some basic tools (the compilers in -# particular) to be present. -hascc=0; -if type cc > /dev/null 2>/dev/null; then - if type c++ > /dev/null 2>/dev/null; then hascc=1; fi -else - if type gcc > /dev/null 2>/dev/null; then - if type g++ > /dev/null 2>/dev/null; then hascc=1; fi - else - if type clang > /dev/null 2>/dev/null; then - if type clang++ > /dev/null 2>/dev/null; then hascc=1; fi - fi - fi -fi -if [ $hascc = 0 ]; then - cat <" > $cprog -#echo "int main(void) {return 0;}" >> $cprog -#if [ x$CC = x ]; then CC=gcc; fi; -#if $CC $cprog -o$oprog -static &> /dev/null; then -# export static_build="yes" -#else -# export static_build="no" -#fi -#rm -f $oprog $cprog -#if [ $printnotice = yes ] && [ $static_build = "no" ]; then -# cat <" > $cprog -echo "int main(void) {return 0;}" >> $cprog -if [ x$CC = x ]; then CC=gcc; fi; -if $CC $cprog -o$oprog -Wl,-rpath-link &> /dev/null; then - export rpath_command="-Wl,-rpath-link=$instdir/lib" -else - export rpath_command="" -fi -rm -f $oprog $cprog - - - - - -# See if we need the dynamic-linker (-ldl) -# ---------------------------------------- -# -# Some programs (like Wget) need dynamic loading (using `libdl'). On -# GNU/Linux systems, we'll need the `-ldl' flag to link such programs. But -# Mac OS doesn't need any explicit linking. So we'll check here to see if -# it is present (thus necessary) or not. -oprog=$sdir/ldl-test -cprog=$sdir/ldl-test.c -cat > $cprog < -#include -int -main(void) { - void *handle=dlopen ("/lib/CEDD_LIB.so.6", RTLD_LAZY); - return 0; -} -EOF -if gcc $cprog -o$oprog &> /dev/null; then needs_ldl=no; else needs_ldl=yes; fi -rm -f $oprog $cprog - - - - - -# inform the user that the build process is starting -# ------------------------------------------------- -if [ $printnotice = yes ]; then - tsec=10 - cat < /dev/null 2>/dev/null; then - host_cc=1 - on_mac_os=yes -else - on_mac_os=no -fi - - - - - -# Build `flock' -# ------------- -# -# Flock (or file-lock) is a unique program that is necessary to serialize -# the (generally parallel) processing of make when necessary. GNU/Linux -# machines have it as part of their `util-linux' programs. But to be -# consistent in non-GNU/Linux systems, we will be using our own build. -# -# The reason that `flock' is sepecial is that we need it to serialize the -# download process of the software tarballs. -flockversion=$(awk '/flock-version/{print $3}' $depverfile) -flocktar=flock-$flockversion.tar.gz -flockurl=http://github.com/discoteq/flock/releases/download/v$flockversion/ - -# Prepare/download the tarball. -if ! [ -f $tardir/$flocktar ]; then - if [ -f $ddir/$flocktar ]; then - cp $ddir/$flocktar $tardir/$flocktar - else - if ! $downloader $tardir/$flocktar $flockurl/$flocktar; then - rm -f $tardir/$flocktar; - echo - echo "DOWNLOAD ERROR: Couldn't download the 'flock' tarball:" - echo " $flockurl" - echo - echo "You can manually place it in '$ddir' to avoid downloading." - exit 1 - fi - fi -fi - -# If the tarball is newer than the (possibly existing) program (the version -# has changed), then delete the program. -if [ -f .local/bin/flock ]; then - if [ $tardir/$flocktar -nt $ibidir/flock ]; then - rm $ibidir/flock - fi -fi - -# Build `flock' if necessary. -if ! [ -f $ibidir/flock ]; then - cd $tmpblddir - tar xf $tardir/$flocktar - cd flock-$flockversion - ./configure --prefix=$instdir - make - make install - cd $topdir - rm -rf $tmpblddir/flock-$flockversion - echo "Discoteq flock $flockversion" > $ibidir/flock -fi - - - - - -# See if GCC can be built -# ----------------------- -# -# On some GNU/Linux distros, the C compiler is broken into `multilib' (for -# 32-bit and 64-bit support, with their own headers). On these systems, -# `/usr/include/sys/cdefs.h' and `/usr/lib/libc.a' are not available by -# default. So GCC will crash with different ugly errors! The only solution -# is that user manually installs the `multilib' part as root, before -# running the configure script. -# -# Note that `sys/cdefs.h' may be available in other directories (for -# example `/usr/include/x86_64-linux-gnu/') that are automatically included -# in an installed GCC. HOWEVER during the build of GCC, all those other -# directories are ignored. So even if they exist, they are useless. -warningsleep=0 -if [ $host_cc = 0 ]; then - if ! [ -f /usr/include/sys/cdefs.h ]; then - host_cc=1 - warningsleep=1 - cat < /dev/null 2>/dev/null; then hasfc=1; fi - - if [ $hasfc = 0 ]; then - cat < /dev/null; then - if [ $jobs = 0 ]; then - numthreads=$(nproc --all); - else - numthreads=$jobs - fi -else - numthreads=1; -fi - - - - - -# Build basic software -# -------------------- -# -# When building these software we don't have our own un-packing software, -# Bash, Make, or AWK. In this step, we'll install such low-level basic -# tools, but we have to be very portable (and use minimal features in all). -make -f reproduce/software/make/basic.mk \ - rpath_command=$rpath_command \ - static_build=$static_build \ - needs_ldl=$needs_ldl \ - on_mac_os=$on_mac_os \ - numthreads=$numthreads \ - host_cc=$host_cc \ - -j$numthreads - - - - - -# All other software -# ------------------ -# -# We will be making all the dependencies before running the top-level -# Makefile. To make the job easier, we'll do it in a Makefile, not a -# script. Bash and Make were the tools we need to run Makefiles, so we had -# to build them in this script. But after this, we can rely on Makefiles. -if [ $jobs = 0 ]; then - numthreads=$($instdir/bin/nproc --all) -else - numthreads=$jobs -fi -./.local/bin/make -f reproduce/software/make/high-level.mk \ - rpath_command=$rpath_command \ - static_build=$static_build \ - numthreads=$numthreads \ - on_mac_os=$on_mac_os \ - host_cc=$host_cc \ - -j$numthreads - - - - - -# Make sure TeX Live installed successfully -# ----------------------------------------- -# -# TeX Live is managed over the internet, so if there isn't any, or it -# suddenly gets cut, it can't be built. However, when TeX Live isn't -# installed, the project can do all its processing independent of it. It -# will just stop at the stage when all the processing is complete and it is -# only necessary to build the PDF. So we don't want to stop the project's -# configuration and building if its not present. -if [ -f $itidir/texlive-ready-tlmgr ]; then - texlive_result=$(cat $itidir/texlive-ready-tlmgr) -else - texlive_result="NOT!" -fi -if [ x"$texlive_result" = x"NOT!" ]; then - cat <0 { \ - c++; \ - if(c==1) \ - { \ - if('$num'==1) printf("%s", $0); \ - else printf("%s", $0); \ - } \ - else if(c=='$num') printf(" and %s\n", $0); \ - else printf(", %s", $0) \ - }' - fi -} - -# Report the different software in separate contexts (separating Python and -# TeX packages from the C/C++ programs and libraries). -proglibs=$(prepare_name_version $verdir/proglib/*) -pymodules=$(prepare_name_version $verdir/python/*) -texpkg=$(prepare_name_version $verdir/tex/texlive) - -# Write them as one paragraph for LaTeX. -pkgver=$mtexdir/dependencies.tex -.local/bin/echo "This research was done with the following free" > $pkgver -.local/bin/echo "software programs and libraries: $proglibs." >> $pkgver -if [ x"$pymodules" != x ]; then - .local/bin/echo "Within Python, the following modules" >> $pkgver - echo "were used: $pymodules." >> $pkgver -fi -.local/bin/echo "The \LaTeX{} source of the paper was compiled" >> $pkgver -.local/bin/echo "to make the PDF using the following packages:" >> $pkgver -.local/bin/echo "$texpkg. We are very grateful to all their" >> $pkgver -.local/bin/echo "creators for freely providing this necessary" >> $pkgver -.local/bin/echo "infrastructure. This research (and many " >> $pkgver -.local/bin/echo "others) would not be possible without them." >> $pkgver - -# Prepare the BibTeX entries for the used software (if there are any). -hasentry=0 -bibfiles="$ictdir/*" -for f in $bibfiles; do if [ -f $f ]; then hasentry=1; break; fi; done; - -# Make sure we start with an empty output file. -pkgbib=$mtexdir/dependencies-bib.tex -echo "" > $pkgbib - -# Fill it in with all the BibTeX entries in this directory. We'll just -# avoid writing any comments (usually copyright notices) and also put an -# empty line after each file's contents to make the output more readable. -if [ $hasentry = 1 ]; then - for f in $bibfiles; do - awk '!/^%/{print} END{print ""}' $f >> $pkgbib - done -fi - - - - - -# Clean the temporary build directory -# --------------------------------- -# -# By the time the script reaches here the temporary software build -# directory should be empty, so just delete it. Note `tmpblddir' may be a -# symbolic link to shared memory. So, to work in any scenario, first delete -# the contents of the directory (if it has any), then delete `tmpblddir'. -.local/bin/rm -rf $tmpblddir/* -.local/bin/rm -rf $tmpblddir - - - - - -# Final step: available Makefile -# ------------------------------ -# -# We only want `make' to work after the configuration is complete. So we -# will only put in the top-level Makefile after all the steps above are -# done. -.local/bin/ln -s $topdir/reproduce/analysis/make/top.mk Makefile - - - - - -# Final notice -# ------------ -# -# The configuration is now complete, we can inform the user on the next -# step(s) to take. -if [ x$reproducible_paper_group_name = x ]; then - buildcommand=".local/bin/make -j8" -else - buildcommand="./for-group $reproducible_paper_group_name make -j8" -fi -cat < -# -# This script is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This script is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# A copy of the GNU General Public License is available at -# . - - - - - -# Script settings -# --------------- -# Stop the script if there are any errors. -set -e - - - - - -# See if any argument are given at all. -if [ "x$1" = x ]; then - echo "$0: At least two arguments are necessary:" - echo - echo " To configure: $ ./for-group group_name configure" - echo " To build: $ ./for-group group_name make" - exit 1 -fi - - - - - -# Prepare any other argument to pass onto the `./configure' or `make' -# commands. `$@' is the list of command-line tokens given to the this -# (`./for-group') script. Therefore, the first token in it is the group -# name and the second is the script name. As a result, we want anything -# after the third token. -options=$(echo "$@" | awk '{for(i=3;i<=NF;++i) printf("%s ", $i)}') - - - - - -# Make sure the given group is usable. -if sg "$1" "echo test &> /dev/null" &> /dev/null; then - if [ "x$2" = xconfigure ]; then script="./configure" - elif [ "x$2" = xmake ]; then script=".local/bin/make" - else - echo "$0: a third argument is necessary." - echo "It specifies the action: either 'configure' or 'make'" - exit 1 - fi -else - echo "$0: '$1' is not a usable group name on this system."; - echo "TIP: you can use the 'groups' command to see your groups." - exit 1 -fi - - - - - -# Define the group, and set the permission so the user and group both have -# read and write permissions. Then run the respective script. -# -# We are also exporting a special variable so `./configure' and Make can -# prepare for sanity checks and avoid re-doing the whole analysis with a -# typo (not using this script properly after configuration). -export reproducible_paper_group_name="$1" -sg "$1" "umask u+r,u+w,g+r,g+w,o-r,o-w,o-x && $script $options" - - - - - -# Group writing permissions for dependencies directory -# -# The common build process sets the writing permissions of the installed -# programs/libraries to `755'. So group members can't write over a -# file. This creates problems when another group member wants to update the -# software for example. We thus need to manually add the group writing flag -# to all installed software files. -if [ "x$2" = xconfigure ]; then - echo "Enabling group writing permission on all installed software..." - .local/bin/chmod -R g+w .local/; -fi diff --git a/project b/project new file mode 100755 index 0000000..b366e00 --- /dev/null +++ b/project @@ -0,0 +1,291 @@ +#!/bin/bash +# +# High-level script to manage the project. +# Run `./project --help' for a description of how to use it. +# +# Copyright (C) 2019 Mohammad Akhlaghi +# +# This script is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# This script is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. See . + + +# Basic settings +# -------------- +# Stop the script if there are any errors. +set -e + +# Default option values +jobs=0 # 0 is for the default for the `configure.sh' script. +group= +debug= +host_cc=0 +operation= +build_dir= +input_dir= +make_targets= +software_dir= +existing_conf=0 +scriptname="./project" +minmapsize=10000000000 + + + + + +# Parse the options +# ----------------- +# +# Separate command-line arguments from options. Then put the option value +# into the respective variable. +# +# Each option has two lines because we want to process both these formats: +# `--name=value' and `--name value'. The former (with `=') is a single +# command-line argument, so we just need to shift the counter by one. The +# latter (without `=') is two arguments, so we'll need two shifts. +# +# Note on the case strings: for every option, we need three lines: one when +# the option name and value are separate. Another when there is an equal +# between them, and finally one where the value is immediately after the +# short-format. This exact order is important. Otherwise, there will be a +# conflict between them. + +print_help() { + # Print the output. + cat < /dev/null" &> /dev/null; then + echo "$scriptname: '$group' is not a usable group name on this system."; + echo "(TIP: you can use the 'groups' command to see your groups)" + exit 1 + fi + + # Set the group option for running Make. + gopt="reproducible_paper_group_name=$group" +fi + + + + + +# Do requested operation +# ---------------------- +perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x" +configscript=./reproduce/software/bash/configure.sh +case $operation in + + # Build the project's software. + configure) + + # Variables to pass to the configuration script. + export jobs=$jobs + export host_cc=$host_cc + export build_dir=$build_dir + export input_dir=$input_dir + export scriptname=$scriptname + export minmapsize=$minmapsize + export software_dir=$software_dir + export existing_conf=$existing_conf + export reproducible_paper_group_name=$group + + # Run the configuration script + if [ x"$group" = x ]; then + $configscript + else + # Set the group and permission flags. + sg "$group" "umask $perms && $configscript" + + # Set the group writing permission for everything in the + # installed software directory. The common build process sets + # the writing permissions of the installed programs/libraries + # to `755'. So group members can't write over a file. This + # creates problems when another group member wants to update + # the software for example. We thus need to manually add the + # group writing flag to all installed software files. + echo "Enabling group writing permission on all installed software..." + .local/bin/chmod -R g+w .local/; + fi + ;; + + # Run the project + make) + + # Make sure the configure script has been completed properly + # (`configuration-done.txt' exists). + if ! [ -f .build/software/configuration-done.txt ]; then + cat < $@ diff --git a/reproduce/analysis/make/top.mk b/reproduce/analysis/make/top.mk index b753979..7d20800 100644 --- a/reproduce/analysis/make/top.mk +++ b/reproduce/analysis/make/top.mk @@ -19,7 +19,8 @@ -# Load the local configuration (created after running `./configure'). +# Load the local configuration (created after running +# `./project configure'). include reproduce/software/config/installation/LOCAL.mk @@ -49,9 +50,8 @@ include reproduce/software/config/installation/LOCAL.mk # # Controlling this requires two variables that are available at this stage: # -# - `GROUP-NAME': from `LOCAL.mk' (which was built by `./configure'). -# - `reproducible_paper_group_name': from the `./for-group' script (if it -# was used to call Make). +# - `GROUP-NAME': from `LOCAL.mk' (which was built by `./project configure'). +# - `reproducible_paper_group_name': value to the `--group' option. # # The analysis is only done when both have the same group name. Note that # when the project isn't being built for a group, both variables will be an @@ -70,10 +70,10 @@ else all: @if [ "x$(GROUP-NAME)" = x ]; then \ echo "Project is NOT configured for groups, please run"; \ - echo " $$ .local/bin/make"; \ + echo " $$ ./project make"; \ else \ echo "Project is configured for groups, please run"; \ - echo " $$ ./for-group $(GROUP-NAME) make -j8"; \ + echo " $$ ./project make --group=$(GROUP-NAME) -j8"; \ fi endif diff --git a/reproduce/software/bash/configure.sh b/reproduce/software/bash/configure.sh new file mode 100755 index 0000000..65b728e --- /dev/null +++ b/reproduce/software/bash/configure.sh @@ -0,0 +1,1224 @@ +#! /bin/bash +# +# Necessary preparations/configurations for the reproducible project. +# +# Copyright (C) 2018-2019 Mohammad Akhlaghi +# +# This script is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# This script is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# A copy of the GNU General Public License is available at +# . + + +# Script settings +# --------------- +# Stop the script if there are any errors. +set -e + + + + + +# Internal directories +# -------------------- +# +# These are defined to help make this script more readable. +topdir=$(pwd) +optionaldir="/optional/path" +adir=reproduce/analysis/config +cdir=reproduce/software/config + +sbdir=$cdir/installation + +pconf=$sbdir/LOCAL.mk +ptconf=$sbdir/LOCAL_tmp.mk +poconf=$sbdir/LOCAL_old.mk +depverfile=$cdir/installation/versions.mk +# --------- Delete for no Gnuastro --------- +glconf=$cdir/gnuastro/gnuastro-local.conf +# ------------------------------------------ + + + + + +# Notice for top of generated files +# --------------------------------- +# +# In case someone opens the files output from the configuration scripts in +# a text editor and wants to edit them, it is important to let them know +# that their changes are not going to be permenant. +function create_file_with_notice() { + if echo "# IMPORTANT: file can be RE-WRITTEN after './project configure'" > "$1" + then + echo "#" >> "$1" + echo "# This file was created during configuration" >> "$1" + echo "# ('./project configure'). Therefore, it is not under" >> "$1" + echo "# version control and any manual changes to it will be" >> "$1" + echo "# over-written if the project re-configured." >> "$1" + echo "#" >> "$1" + else + echo; echo "Can't write to $1"; echo; + exit 1 + fi +} + + + + + +# Get absolute address +# -------------------- +# +# Since the build directory will go into a symbolic link, we want it to be +# an absolute address. With this function we can make sure of that. +function absolute_dir() { + if stat "$1" 1> /dev/null; then + echo "$(cd "$(dirname "$1")" && pwd )/$(basename "$1")" + else + exit 1; + fi +} + + + + + +# Inform the user +# --------------- +# +# Print some basic information so the user gets a feeling of what is going +# on and is prepared on what will happen next. +cat < /dev/null 2>/dev/null; then + name=$(which wget) + + # By default Wget keeps the remote file's timestamp, so we'll have + # to disable it manually. + downloader="$name --no-use-server-timestamps -O"; + elif type curl > /dev/null 2>/dev/null; then + name=$(which curl) + + # - cURL doesn't keep the remote file's timestamp by default. + # - With the `-L' option, we tell cURL to follow redirects. + downloader="$name -L -o" + else + cat < /dev/null; then + bdir=$(absolute_dir $build_dir) + echo " -- Build directory: '$bdir'" + rm -rf $build_dir/$junkname + else + echo " -- Can't write in '$build_dir'" + fi + else + if mkdir $build_dir 2> /dev/null; then + bdir=$(absolute_dir $build_dir) + echo " -- Build directory set to (the newly created): '$bdir'" + else + echo " -- Can't create '$build_dir'" + fi + fi + + # Reset `build_dir' to blank, so it continues asking when the + # previous value wasn't usable. + build_dir= + done +fi + + + + + +# Input directory +# --------------- +if [ x"$input_dir" = x ]; then + indir=$optionaldir +else + indir=$input_dir +fi +wfpc2name=$(awk '!/^#/ && $1=="WFPC2IMAGE" {print $3}' $adir/INPUTS.mk) +wfpc2md5=$(awk '!/^#/ && $1=="WFPC2MD5" {print $3}' $adir/INPUTS.mk) +wfpc2size=$(awk '!/^#/ && $1=="WFPC2SIZE" {print $3}' $adir/INPUTS.mk) +wfpc2url=$(awk '!/^#/ && $1=="WFPC2URL" {print $3}' $adir/INPUTS.mk) +if [ $rewritepconfig = yes ] && [ x"$input_dir" = x ]; then + cat <> $pconf +else + # Read the values from existing configuration file. + inbdir=$(awk '$1=="BDIR" {print $3}' $pconf) + + # Read the software directory. + ddir=$(awk '$1=="DEPENDENCIES-DIR" {print $3}' $pconf) + + # The downloader command may contain multiple elements, so we'll just + # change the (in memory) first and second tokens to empty space and + # write the full line (the original file is unchanged). + downloader=$(awk '$1=="DOWNLOADER" {$1=""; $2=""; print $0}' $pconf) + + # Make sure all necessary variables have a value + err=0 + verr=0 + novalue="" + if [ x"$inbdir" = x ]; then novalue="BDIR, "; fi + if [ x"$downloader" = x ]; then novalue="$novalue"DOWNLOADER; fi + if [ x"$novalue" != x ]; then verr=1; err=1; fi + + # Make sure `bdir' is an absolute path and it exists. + berr=0 + ierr=0 + bdir=$(absolute_dir $inbdir) + + if ! [ -d $bdir ]; then if ! mkdir $bdir; then berr=1; err=1; fi; fi + if [ $err = 1 ]; then + cat <> $glconf + echo " minmapsize $minmapsize" >> $glconf + echo >> $glconf + echo "# Version of Gnuastro that must be used." >> $glconf + echo " onlyversion $gversion" >> $glconf +else + ingversion=$(awk '$1=="onlyversion" {print $NF}' $glconf) + if [ x$ingversion != x$gversion ]; then + cat <8GB) is large enough for the parallel building of the software. +# +# For the name of the directory under `/dev/shm' (for this project), we'll +# use the names of the two parent directories to the current/running +# directory, separated by a `-' instead of `/'. We'll then appended that +# with the user's name (in case multiple users may be working on similar +# project names). Maybe later, we can use something like `mktemp' to add +# random characters to this name and make it unique to every run (even for +# a single user). +tmpblddir=$sdir/build-tmp +rm -rf $tmpblddir/* $tmpblddir # If its a link, we need to empty its + # contents first, then itself. + +# Set the top-level shared memory location. +if [ -d /dev/shm ]; then shmdir=/dev/shm +else shmdir="" +fi + +# If a shared memory mounted directory exists and there is enough space +# there (in RAM), build a temporary directory for this project. +needed_space=2000000 +if [ x"$shmdir" != x ]; then + available_space=$(df $shmdir | awk 'NR==2{print $4}') + if [ $available_space -gt $needed_space ]; then + dirname=$(pwd | sed -e's/\// /g' \ + | awk '{l=NF-1; printf("%s-%s",$l, $NF)}') + tbshmdir=$shmdir/"$dirname"-$(whoami) + if ! [ -d $tbshmdir ]; then mkdir $tbshmdir; fi + fi +else + tbshmdir="" +fi + +# If a shared memory directory was created set `build-tmp' to be a +# symbolic link to it. Otherwise, just build the temporary build +# directory under the project build directory. +if [ x$tbshmdir = x ]; then mkdir $tmpblddir; +else ln -s $tbshmdir $tmpblddir; +fi + + + + + +# Check for C/C++ compilers +# ------------------------- +# +# To build the software, we'll need some basic tools (the compilers in +# particular) to be present. +hascc=0; +if type cc > /dev/null 2>/dev/null; then + if type c++ > /dev/null 2>/dev/null; then hascc=1; fi +else + if type gcc > /dev/null 2>/dev/null; then + if type g++ > /dev/null 2>/dev/null; then hascc=1; fi + else + if type clang > /dev/null 2>/dev/null; then + if type clang++ > /dev/null 2>/dev/null; then hascc=1; fi + fi + fi +fi +if [ $hascc = 0 ]; then + cat <" > $cprog +#echo "int main(void) {return 0;}" >> $cprog +#if [ x$CC = x ]; then CC=gcc; fi; +#if $CC $cprog -o$oprog -static &> /dev/null; then +# export static_build="yes" +#else +# export static_build="no" +#fi +#rm -f $oprog $cprog +#if [ $printnotice = yes ] && [ $static_build = "no" ]; then +# cat <" > $cprog +echo "int main(void) {return 0;}" >> $cprog +if [ x$CC = x ]; then CC=gcc; fi; +if $CC $cprog -o$oprog -Wl,-rpath-link &> /dev/null; then + export rpath_command="-Wl,-rpath-link=$instdir/lib" +else + export rpath_command="" +fi +rm -f $oprog $cprog + + + + + +# See if we need the dynamic-linker (-ldl) +# ---------------------------------------- +# +# Some programs (like Wget) need dynamic loading (using `libdl'). On +# GNU/Linux systems, we'll need the `-ldl' flag to link such programs. But +# Mac OS doesn't need any explicit linking. So we'll check here to see if +# it is present (thus necessary) or not. +oprog=$sdir/ldl-test +cprog=$sdir/ldl-test.c +cat > $cprog < +#include +int +main(void) { + void *handle=dlopen ("/lib/CEDD_LIB.so.6", RTLD_LAZY); + return 0; +} +EOF +if gcc $cprog -o$oprog &> /dev/null; then needs_ldl=no; else needs_ldl=yes; fi +rm -f $oprog $cprog + + + + + +# inform the user that the build process is starting +# ------------------------------------------------- +if [ $printnotice = yes ]; then + tsec=10 + cat < /dev/null 2>/dev/null; then + host_cc=1 + on_mac_os=yes +else + on_mac_os=no +fi + + + + + +# Build `flock' +# ------------- +# +# Flock (or file-lock) is a unique program that is necessary to serialize +# the (generally parallel) processing of make when necessary. GNU/Linux +# machines have it as part of their `util-linux' programs. But to be +# consistent in non-GNU/Linux systems, we will be using our own build. +# +# The reason that `flock' is sepecial is that we need it to serialize the +# download process of the software tarballs. +flockversion=$(awk '/flock-version/{print $3}' $depverfile) +flocktar=flock-$flockversion.tar.gz +flockurl=http://github.com/discoteq/flock/releases/download/v$flockversion/ + +# Prepare/download the tarball. +if ! [ -f $tardir/$flocktar ]; then + if [ -f $ddir/$flocktar ]; then + cp $ddir/$flocktar $tardir/$flocktar + else + if ! $downloader $tardir/$flocktar $flockurl/$flocktar; then + rm -f $tardir/$flocktar; + echo + echo "DOWNLOAD ERROR: Couldn't download the 'flock' tarball:" + echo " $flockurl" + echo + echo "You can manually place it in '$ddir' to avoid downloading." + exit 1 + fi + fi +fi + +# If the tarball is newer than the (possibly existing) program (the version +# has changed), then delete the program. +if [ -f .local/bin/flock ]; then + if [ $tardir/$flocktar -nt $ibidir/flock ]; then + rm $ibidir/flock + fi +fi + +# Build `flock' if necessary. +if ! [ -f $ibidir/flock ]; then + cd $tmpblddir + tar xf $tardir/$flocktar + cd flock-$flockversion + ./configure --prefix=$instdir + make + make install + cd $topdir + rm -rf $tmpblddir/flock-$flockversion + echo "Discoteq flock $flockversion" > $ibidir/flock +fi + + + + + +# See if GCC can be built +# ----------------------- +# +# On some GNU/Linux distros, the C compiler is broken into `multilib' (for +# 32-bit and 64-bit support, with their own headers). On these systems, +# `/usr/include/sys/cdefs.h' and `/usr/lib/libc.a' are not available by +# default. So GCC will crash with different ugly errors! The only solution +# is that user manually installs the `multilib' part as root, before +# running the configure script. +# +# Note that `sys/cdefs.h' may be available in other directories (for +# example `/usr/include/x86_64-linux-gnu/') that are automatically included +# in an installed GCC. HOWEVER during the build of GCC, all those other +# directories are ignored. So even if they exist, they are useless. +warningsleep=0 +if [ $host_cc = 0 ]; then + if ! [ -f /usr/include/sys/cdefs.h ]; then + host_cc=1 + warningsleep=1 + cat < /dev/null 2>/dev/null; then hasfc=1; fi + + if [ $hasfc = 0 ]; then + cat < /dev/null; then + if [ $jobs = 0 ]; then + numthreads=$(nproc --all); + else + numthreads=$jobs + fi +else + numthreads=1; +fi + + + + + +# Build basic software +# -------------------- +# +# When building these software we don't have our own un-packing software, +# Bash, Make, or AWK. In this step, we'll install such low-level basic +# tools, but we have to be very portable (and use minimal features in all). +make -f reproduce/software/make/basic.mk \ + rpath_command=$rpath_command \ + static_build=$static_build \ + needs_ldl=$needs_ldl \ + on_mac_os=$on_mac_os \ + numthreads=$numthreads \ + host_cc=$host_cc \ + -j$numthreads + + + + + +# All other software +# ------------------ +# +# We will be making all the dependencies before running the top-level +# Makefile. To make the job easier, we'll do it in a Makefile, not a +# script. Bash and Make were the tools we need to run Makefiles, so we had +# to build them in this script. But after this, we can rely on Makefiles. +if [ $jobs = 0 ]; then + numthreads=$($instdir/bin/nproc --all) +else + numthreads=$jobs +fi +.local/bin/env -i HOME=$bdir \ + .local/bin/make -f reproduce/software/make/high-level.mk \ + rpath_command=$rpath_command \ + static_build=$static_build \ + numthreads=$numthreads \ + on_mac_os=$on_mac_os \ + host_cc=$host_cc \ + -j$numthreads + + + + + +# Make sure TeX Live installed successfully +# ----------------------------------------- +# +# TeX Live is managed over the internet, so if there isn't any, or it +# suddenly gets cut, it can't be built. However, when TeX Live isn't +# installed, the project can do all its processing independent of it. It +# will just stop at the stage when all the processing is complete and it is +# only necessary to build the PDF. So we don't want to stop the project's +# configuration and building if its not present. +if [ -f $itidir/texlive-ready-tlmgr ]; then + texlive_result=$(cat $itidir/texlive-ready-tlmgr) +else + texlive_result="NOT!" +fi +if [ x"$texlive_result" = x"NOT!" ]; then + cat <0 { \ + c++; \ + if(c==1) \ + { \ + if('$num'==1) printf("%s", $0); \ + else printf("%s", $0); \ + } \ + else if(c=='$num') printf(" and %s\n", $0); \ + else printf(", %s", $0) \ + }' + fi +} + +# Report the different software in separate contexts (separating Python and +# TeX packages from the C/C++ programs and libraries). +proglibs=$(prepare_name_version $verdir/proglib/*) +pymodules=$(prepare_name_version $verdir/python/*) +texpkg=$(prepare_name_version $verdir/tex/texlive) + +# Write them as one paragraph for LaTeX. +pkgver=$mtexdir/dependencies.tex +.local/bin/echo "This research was done with the following free" > $pkgver +.local/bin/echo "software programs and libraries: $proglibs." >> $pkgver +if [ x"$pymodules" != x ]; then + .local/bin/echo "Within Python, the following modules" >> $pkgver + echo "were used: $pymodules." >> $pkgver +fi +.local/bin/echo "The \LaTeX{} source of the paper was compiled" >> $pkgver +.local/bin/echo "to make the PDF using the following packages:" >> $pkgver +.local/bin/echo "$texpkg. We are very grateful to all their" >> $pkgver +.local/bin/echo "creators for freely providing this necessary" >> $pkgver +.local/bin/echo "infrastructure. This research (and many " >> $pkgver +.local/bin/echo "others) would not be possible without them." >> $pkgver + +# Prepare the BibTeX entries for the used software (if there are any). +hasentry=0 +bibfiles="$ictdir/*" +for f in $bibfiles; do if [ -f $f ]; then hasentry=1; break; fi; done; + +# Make sure we start with an empty output file. +pkgbib=$mtexdir/dependencies-bib.tex +echo "" > $pkgbib + +# Fill it in with all the BibTeX entries in this directory. We'll just +# avoid writing any comments (usually copyright notices) and also put an +# empty line after each file's contents to make the output more readable. +if [ $hasentry = 1 ]; then + for f in $bibfiles; do + awk '!/^%/{print} END{print ""}' $f >> $pkgbib + done +fi + + + + + +# Clean the temporary build directory +# --------------------------------- +# +# By the time the script reaches here the temporary software build +# directory should be empty, so just delete it. Note `tmpblddir' may be a +# symbolic link to shared memory. So, to work in any scenario, first delete +# the contents of the directory (if it has any), then delete `tmpblddir'. +.local/bin/rm -rf $tmpblddir/* $tmpblddir + + + + + +# Register successful completion +# ------------------------------ +echo `.local/bin/date` > $finaltarget + + + + + + +# Final notice +# ------------ +# +# The configuration is now complete, we can inform the user on the next +# step(s) to take. +if [ x$reproducible_paper_group_name = x ]; then + buildcommand="./project make -j8" +else + buildcommand="./project make --group=$reproducible_paper_group_name -j8" +fi +cat < # diff --git a/reproduce/software/make/atlas-multiple.mk b/reproduce/software/make/atlas-multiple.mk index fef25c7..f5efb35 100755 --- a/reproduce/software/make/atlas-multiple.mk +++ b/reproduce/software/make/atlas-multiple.mk @@ -3,8 +3,8 @@ # ------------------------------------------------------------------------ # !!!!! IMPORTANT NOTES !!!!! # -# This Makefile will be run during the initial `./configure' script. It is -# not included into the reproduction pipe after that. +# This Makefile will be run during the initial `./project configure' +# script. It is not included into the reproduction pipe after that. # # ------------------------------------------------------------------------ # diff --git a/reproduce/software/make/atlas-single.mk b/reproduce/software/make/atlas-single.mk index dde2926..c4621ce 100755 --- a/reproduce/software/make/atlas-single.mk +++ b/reproduce/software/make/atlas-single.mk @@ -3,8 +3,8 @@ # ------------------------------------------------------------------------ # !!!!! IMPORTANT NOTES !!!!! # -# This Makefile will be run during the initial `./configure' script. It is -# not included into the reproduction pipe after that. +# This Makefile will be run during the initial `./project configure' +# script. It is not included into the reproduction pipe after that. # # ------------------------------------------------------------------------ # diff --git a/reproduce/software/make/basic.mk b/reproduce/software/make/basic.mk index bab37ed..e4ab55a 100644 --- a/reproduce/software/make/basic.mk +++ b/reproduce/software/make/basic.mk @@ -4,8 +4,8 @@ # ------------------------------------------------------------------------ # !!!!! IMPORTANT NOTES !!!!! # -# This Makefile will be run by the initial `./configure' script. It is not -# included into the project after that. +# This Makefile will be run by the initial `./project configure' script. It +# is not included into the project after that. # # This Makefile builds very low-level and basic tools like GNU Tar, GNU # Bash, GNU Make, GCC and etc. Therefore this is the only Makefile in the @@ -58,7 +58,11 @@ export PKG_CONFIG_LIBDIR := $(ildir)/pkgconfig export CPPFLAGS := -I$(idir)/include $(CPPFLAGS) export LD_LIBRARY_PATH := $(ildir):$(LD_LIBRARY_PATH) export LDFLAGS := $(rpath_command) -L$(ildir) $(LDFLAGS) -export DYLD_LIBRARY_PATH := $(ildir):$(DYLD_LIBRARY_PATH) + +# RPATH is automatically written in macOS, so `DYLD_LIBRARY_PATH' is +# ultimately redundant. But on some systems, even having a single value +# causes crashs (see bug #56682). So we'll just give it no value at all. +export DYLD_LIBRARY_PATH := # Define the top-level basic programs (that don't depend on any other). top-level-programs = low-level-links gcc diff --git a/reproduce/software/make/high-level.mk b/reproduce/software/make/high-level.mk index dee23ca..b9a70de 100644 --- a/reproduce/software/make/high-level.mk +++ b/reproduce/software/make/high-level.mk @@ -3,8 +3,8 @@ # ------------------------------------------------------------------------ # !!!!! IMPORTANT NOTES !!!!! # -# This Makefile will be run by the initial `./configure' script. It is not -# included into the reproduction pipe after that. +# This Makefile will be run by the initial `./project configure' script. It +# is not included into the reproduction pipe after that. # # ------------------------------------------------------------------------ # @@ -76,7 +76,11 @@ export LD_RUN_PATH := $(ildir):$(il64dir) export PKG_CONFIG_PATH := $(ildir)/pkgconfig export LD_LIBRARY_PATH := $(ildir):$(il64dir) export PKG_CONFIG_LIBDIR := $(ildir)/pkgconfig -export DYLD_LIBRARY_PATH := $(ildir):$(il64dir) + +# RPATH is automatically written in macOS, so `DYLD_LIBRARY_PATH' is +# ultimately redundant. But on some systems, even having a single value +# causes crashs (see bug #56682). So we'll just give it no value at all. +export DYLD_LIBRARY_PATH := # Building flags: # @@ -90,6 +94,9 @@ export CXXFLAGS := -liconv endif + + + # We want the download to happen on a single thread. So we need to define a # lock, and call a special script we have written for this job. These are # placed here because we want them both in the `high-level.mk' and diff --git a/reproduce/software/make/python.mk b/reproduce/software/make/python.mk index 22b668f..ed1c87d 100644 --- a/reproduce/software/make/python.mk +++ b/reproduce/software/make/python.mk @@ -3,8 +3,8 @@ # ------------------------------------------------------------------------ # !!!!! IMPORTANT NOTES !!!!! # -# This Makefile will be run by the initial `./configure' script. It is not -# included into the reproduction pipe after that. +# This Makefile will be run by the initial `./project configure' script. It +# is not included into the reproduction pipe after that. # # ------------------------------------------------------------------------ # -- cgit v1.2.1