aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/shell/configure.sh
diff options
context:
space:
mode:
authorMohammad Akhlaghi <mohammad@akhlaghi.org>2021-10-04 02:51:45 +0200
committerMohammad Akhlaghi <mohammad@akhlaghi.org>2022-01-21 01:15:24 +0100
commit8463df97c6f26ec4d22cd5828bb0574fd5e450d2 (patch)
treedbaa2e7c2fc44856eb98555b79c6814f210a6c17 /reproduce/software/shell/configure.sh
parent775fc036e0091f05ff56e41b855bc416b9ed36c8 (diff)
IMPORTANT: Updates to almost all software
This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2
Diffstat (limited to 'reproduce/software/shell/configure.sh')
-rwxr-xr-xreproduce/software/shell/configure.sh134
1 files changed, 92 insertions, 42 deletions
diff --git a/reproduce/software/shell/configure.sh b/reproduce/software/shell/configure.sh
index 0f7278f..07381e6 100755
--- a/reproduce/software/shell/configure.sh
+++ b/reproduce/software/shell/configure.sh
@@ -2,8 +2,9 @@
#
# Necessary preparations/configurations for the reproducible project.
#
-# Copyright (C) 2018-2021 Mohammad Akhlaghi <mohammad@akhlaghi.org>
-# Copyright (C) 2021 Raul Infante-Sainz <infantesainz@gmail.com>
+# Copyright (C) 2018-2022 Mohammad Akhlaghi <mohammad@akhlaghi.org>
+# Copyright (C) 2021-2022 Raul Infante-Sainz <infantesainz@gmail.com>
+# Copyright (C) 2022 Pedram Ashofteh Ardakani <pedramardakani@pm.me>
#
# This script is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -112,7 +113,7 @@ absolute_dir ()
# Check file permission handling (POSIX-compatibility)
# ----------------------------------------------------
#
-# Check if a `given' directory handles permissions as expected.
+# Check if a 'given' directory handles permissions as expected.
#
# This is to prevent a known bug in the NTFS filesystem that prevents
# proper installation of Perl, and probably some other packages. This
@@ -120,15 +121,15 @@ absolute_dir ()
# file, and examines whether the given directory handles the file
# permissions as expected.
#
-# Returns `0' if everything is fine, and `255' otherwise. Choosing `0' is
-# to mimic the `$ echo $?' behavior, while choosing `255' is to prevent
+# Returns '0' if everything is fine, and '255' otherwise. Choosing '0' is
+# to mimic the '$ echo $?' behavior, while choosing '255' is to prevent
# misunderstanding 0 and 1 as true and false.
#
# ===== CAUTION! ===== #
#
-# Since there is a `set -e' before running this function, the whole script
-# stops and exits IF the `check_permission' (or any other function) returns
-# anything OTHER than `0'! So, only use this function as a test. Here's a
+# Since there is a 'set -e' before running this function, the whole script
+# stops and exits IF the 'check_permission' (or any other function) returns
+# anything OTHER than '0'! So, only use this function as a test. Here's a
# minimal example:
#
# if $(check_permission $some_directory) ; then
@@ -136,7 +137,7 @@ absolute_dir ()
# fi ;
check_permission ()
{
- # Make a `junk' file, activate its executable flag and record its
+ # Make a 'junk' file, activate its executable flag and record its
# permissions generally.
local junkfile="$1"/check_permission_tmp_file
rm -f "$junkfile"
@@ -309,7 +310,9 @@ fi
# system. Here, it is checked that this is the case, and if not, warn the user
# about not having Xcode already installed.
if [ x$on_mac_os = xyes ]; then
- xcode=$(which xcodebuild)
+
+ # 'which' isn't in POSIX, so we are using 'command -v' instead.
+ xcode=$(command -v xcodebuild)
if [ x$xcode != x ]; then
xcode_version=$(xcodebuild -version | grep Xcode)
echo " "
@@ -475,8 +478,8 @@ fi
# See if we need the dynamic-linker (-ldl)
# ----------------------------------------
#
-# Some programs (like Wget) need dynamic loading (using `libdl'). On
-# GNU/Linux systems, we'll need the `-ldl' flag to link such programs. But
+# Some programs (like Wget) need dynamic loading (using 'libdl'). On
+# GNU/Linux systems, we'll need the '-ldl' flag to link such programs. But
# Mac OS doesn't need any explicit linking. So we'll check here to see if
# it is present (thus necessary) or not.
cat > $testsource <<EOF
@@ -508,7 +511,7 @@ fi
# programs will go and find their necessary libraries on the host system.
#
# Another good advantage of shared libraries is that we can actually use
-# the shared library tool of the system (`ldd' with GNU C Library) and see
+# the shared library tool of the system ('ldd' with GNU C Library) and see
# exactly where each linked library comes from. But in static building,
# unless you follow the build closely, its not easy to see if the source of
# the library came from the system or our build.
@@ -545,7 +548,7 @@ fi
#
# On some systems (in particular Debian-based OSs), the static C library
# and necessary headers in a non-standard place, and we can't build GCC. So
-# we need to find them first. The `sys/cdefs.h' header is also in a
+# we need to find them first. The 'sys/cdefs.h' header is also in a
# similarly different location.
sys_cpath=""
sys_library_path=""
@@ -576,7 +579,7 @@ fi
# See if a link-able static C library exists
# ------------------------------------------
#
-# A static C library and the `sys/cdefs.h' header are necessary for
+# A static C library and the 'sys/cdefs.h' header are necessary for
# building GCC.
if [ x"$host_cc" = x0 ]; then
echo; echo; echo "Checking if static C library is available...";
@@ -752,9 +755,9 @@ EOF
# What to do with possibly existing configuration file
# ----------------------------------------------------
#
-# `LOCAL.conf' is the top-most local configuration for the project. If it
+# 'LOCAL.conf' is the top-most local configuration for the project. If it
# already exists when this script is run, we'll make a copy of it as backup
-# (for example the user might have ran `./project configure' by mistake).
+# (for example the user might have ran './project configure' by mistake).
printnotice=yes
rewritepconfig=yes
if [ -f $pconf ]; then
@@ -807,16 +810,18 @@ fi
# need to check the host's available tool for downloading at this step.
if [ $rewritepconfig = yes ]; then
if type wget > /dev/null 2>/dev/null; then
- name=$(which wget)
+
+ # 'which' isn't in POSIX, so we are using 'command -v' instead.
+ name=$(command -v wget)
# By default Wget keeps the remote file's timestamp, so we'll have
# to disable it manually.
downloader="$name --no-use-server-timestamps -O";
elif type curl > /dev/null 2>/dev/null; then
- name=$(which curl)
+ name=$(command -v curl)
# - cURL doesn't keep the remote file's timestamp by default.
- # - With the `-L' option, we tell cURL to follow redirects.
+ # - With the '-L' option, we tell cURL to follow redirects.
downloader="$name -L -o"
else
cat <<EOF
@@ -913,7 +918,7 @@ EOF
# If it was newly created, it will be empty, so delete it.
if ! [ "$(ls -A $bdir)" ]; then rm --dir "$bdir"; fi
- # Inform the user that this is not acceptable and reset `bdir'.
+ # Inform the user that this is not acceptable and reset 'bdir'.
bdir=
echo " ** The build-directory cannot be under the source-directory."
fi
@@ -960,7 +965,7 @@ EOF
fi
# If the build directory was good, the loop will stop, if not,
- # reset `build_dir' to blank, so it continues asking for another
+ # reset 'build_dir' to blank, so it continues asking for another
# directory and let the user know that they must select a new
# directory.
if [ x"$bdir" = x ]; then
@@ -1118,7 +1123,7 @@ else
if [ x"$downloader" = x ]; then novalue="$novalue"DOWNLOADER; fi
if [ x"$novalue" != x ]; then verr=1; err=1; fi
- # Make sure `bdir' is an absolute path and it exists.
+ # Make sure 'bdir' is an absolute path and it exists.
berr=0
ierr=0
bdir="$(absolute_dir "$inbdir")"
@@ -1202,6 +1207,10 @@ if ! [ -d "$ibidir" ]; then mkdir "$ibidir"; fi
ipydir="$verdir"/python
if ! [ -d "$ipydir" ]; then mkdir "$ipydir"; fi
+# R module versions and citation.
+ircrandir="$verdir"/r-cran
+if ! [ -d "$ircrandir" ]; then mkdir "$ircrandir"; fi
+
# Used software BibTeX entries.
ictdir="$verdir"/cite
if ! [ -d "$ictdir" ]; then mkdir "$ictdir"; fi
@@ -1210,6 +1219,15 @@ if ! [ -d "$ictdir" ]; then mkdir "$ictdir"; fi
itidir="$verdir"/tex
if ! [ -d "$itidir" ]; then mkdir "$itidir"; fi
+# Some software install their libraries in '$(idir)/lib64'. But all other
+# libraries are in '$(idir)/lib'. Since Maneage's build is only for a
+# single architecture, we can set the '$(idir)/lib64' as a symbolic link to
+# '$(idir)/lib' so all the libraries are always available in the same
+# place.
+instlibdir="$instdir"/lib
+if ! [ -d "$instlibdir" ]; then mkdir "$instlibdir"; fi
+ln -fs "$instlibdir" "$instdir"/lib64
+
@@ -1312,38 +1330,69 @@ if [ x"$shmdir" != x ]; then
# Set the Maneage-specific directory within the shared
# memory. We'll use the names of the two parent directories to the
- # current/running directory, separated by a `-' instead of
- # `/'. We'll then appended that with the user's name (in case
+ # current/running directory, separated by a '-' instead of
+ # '/'. We'll then appended that with the user's name (in case
# multiple users may be working on similar project names).
#
- # Maybe later, we can use something like `mktemp' to add random
+ # Maybe later, we can use something like 'mktemp' to add random
# characters to this name and make it unique to every run (even for
# a single user).
dirname=$(pwd | sed -e's/\// /g' \
| awk '{l=NF-1; printf("%s-%s", $l, $NF)}')
tbshmdir="$shmdir"/"$dirname"-$(whoami)
- if ! [ -d "$tbshmdir" ]; then mkdir "$tbshmdir"; fi
+
+ # Try to make the directory if it does not yet exist. A failed
+ # directory creation will be tested for a few lines later, when
+ # testing for the existence and executability of a test file.
+ if ! [ -d "$tbshmdir" ]; then (mkdir "$tbshmdir" || true); fi
# Some systems may protect '/dev/shm' against the right to execute
# programs by ordinary users. We thus need to check that the device
# allows execution within this directory by this user.
shmexecfile="$tbshmdir"/shm-execution-check.sh
rm -f $shmexecfile # We also don't want any existing flags.
- cat > "$shmexecfile" <<EOF
+
+ # Create the file to be executed, but do not fail fatally if it
+ # cannot be created. We will check a few lines later if the file
+ # really exists.
+ (cat > "$shmexecfile" <<EOF || true)
#!/bin/sh
-printf "This file successfully executed.\n"
+echo "This file successfully executed."
EOF
- # Make the file executable and see if it runs. If not, set
- # 'tbshmdir' to an empty string so it is not used in later steps.
- # In any case, delete the temporary file afterwards.
- chmod u+x "$shmexecfile"
- if ! "$shmexecfile" &> /dev/null; then tbshmdir=""; fi
- rm "$shmexecfile"
+
+ # If the file was successfully created, then make the file
+ # executable and see if it runs. If not, set 'tbshmdir' to an empty
+ # string so it is not used in later steps. In any case, delete the
+ # temporary file afterwards.
+ #
+ # We aren't adding '&> /dev/null' after the execution command
+ # because it can produce false failures randomly on some systems.
+ if [ -e "$shmexecfile" ]; then
+
+ # Add the executable flag.
+ chmod +x "$shmexecfile"
+
+ # The following line tries to execute the file.
+ if "$shmexecfile"; then
+ # Successful execution. The colon is a "no-op" (no
+ # operation) shell command.
+ :
+ else
+ tbshmdir=""
+ fi
+ rm "$shmexecfile"
+ else
+ tbshmdir=""
+ fi
fi
else
tbshmdir=""
fi
+
+
+
+
# If a shared memory directory was created, set the software building
# directory to be a symbolic link to it. Otherwise, just build the
# temporary build directory under the project's build directory.
@@ -1426,7 +1475,7 @@ fi
# See if the linker accepts -Wl,-rpath-link
# -----------------------------------------
#
-# `-rpath-link' is used to write the information of the linked shared
+# '-rpath-link' is used to write the information of the linked shared
# library into the shared object (library or program). But some versions of
# LLVM's linker don't accept it an can cause problems.
#
@@ -1459,13 +1508,13 @@ rm -rf $compilertestdir
-# Paths needed by the host compiler (only for `basic.mk')
+# Paths needed by the host compiler (only for 'basic.mk')
# -------------------------------------------------------
#
# At the end of the basic build, we need to build GCC. But GCC will build
# in multiple phases, making its own simple compiler in order to build
# itself completely. The intermediate/simple compiler doesn't recognize
-# some system specific locations like `/usr/lib/ARCHITECTURE' that some
+# some system specific locations like '/usr/lib/ARCHITECTURE' that some
# operating systems use. We thus need to tell the intermediate compiler
# where its necessary libraries and headers are.
if [ x"$sys_library_path" != x ]; then
@@ -1502,7 +1551,7 @@ fi
# which will download the DOI-resolved webpage, and extract the Zenodo-URL
# of the most recent version from there (using the 'coreutils' tarball as
# an example, the directory part of the URL for all the other software are
-# the same). This is not done if the option `--debug' is used.
+# the same). This is not done if the option '--debug' is used.
zenodourl=""
user_backup_urls=""
zenodocheck=.build/software/zenodo-check.html
@@ -1535,6 +1584,7 @@ user_backup_urls="$user_backup_urls $zenodourl"
# (compression program), GNU Make (that 'basic.mk' is written in), Dash
# (minimal Bash-like shell) and Flock (to lock files and enable serial
# download).
+export on_mac_os
./reproduce/software/shell/pre-make-build.sh \
"$bdir" "$ddir" "$downloader" "$user_backup_urls"
@@ -1672,7 +1722,7 @@ fi
# software.
prepare_name_version ()
{
- # First see if the (possible) `*' in the input arguments corresponds to
+ # First see if the (possible) '*' in the input arguments corresponds to
# anything. Note that some of the given directories may be empty (no
# software installed).
hasfiles=0
@@ -1773,9 +1823,9 @@ hw_class_fixed="$(echo $hw_class | sed -e 's/_/\\_/')"
# ---------------------------------
#
# By the time the script reaches here the temporary software build
-# directory should be empty, so just delete it. Note `tmpblddir' may be a
+# directory should be empty, so just delete it. Note 'tmpblddir' may be a
# symbolic link to shared memory. So, to work in any scenario, first delete
-# the contents of the directory (if it has any), then delete `tmpblddir'.
+# the contents of the directory (if it has any), then delete 'tmpblddir'.
.local/bin/rm -rf $tmpblddir/* $tmpblddir