From ef9fe47c5d5ecc195a1e4bf059dab3b709852ee7 Mon Sep 17 00:00:00 2001 From: Boud Roukema Date: Tue, 18 May 2021 14:53:13 +0200 Subject: Configuration: improved warning when TeX Live couldn't be installed Once a year, the texlive update system becomes incompatible with the version from the previous year. Since a texlive install failure is considered non-fatal by 'high-level.mk', so until now, the user could miss the printed message and mistakenly believe that the configure is valid. This commit explicitly adds a 10-second delay that should be enough for a user who does the 'configure --existing-conf' step alone to notice that there is a TeX Live problem. It also adds the explicit instruction of how to allow an update from an earlier year's texlive installer to the warning message (by deleting '.build/software/tarballs/install-tl-unx.tar.gz'). I had to rediscover this a few times for old Maneage installs. Also, a few lines in 'reproduce/software/shell/configure.sh' were indented with a TAB (that is not recommended because TAB is displayed with different widths on different browsers). So while doing this commit, those TABs were also converted to a space. --- reproduce/software/shell/configure.sh | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) (limited to 'reproduce/software/shell') diff --git a/reproduce/software/shell/configure.sh b/reproduce/software/shell/configure.sh index e15a8e9..12690f7 100755 --- a/reproduce/software/shell/configure.sh +++ b/reproduce/software/shell/configure.sh @@ -158,7 +158,7 @@ check_permission () return 1 else # Setting permission SUCCESSFUL - return 0 + return 0 fi } @@ -938,10 +938,10 @@ EOF fi fi - # If everything is still fine so far, see if we're able to - # manipulate file permissions in the directory's filesystem and if - # so, see if there is atleast 5GB free space. - if ! [ x"$bdir" = x ]; then + # If everything is still fine so far, see if we're able to + # manipulate file permissions in the directory's filesystem and if + # so, see if there is atleast 5GB free space. + if ! [ x"$bdir" = x ]; then if ! $(check_permission "$bdir"); then # Unable to handle permissions well bdir= @@ -1610,17 +1610,21 @@ is not used at all during the analysis. Therefore, if you don't need the final PDF, and just want to do the analysis, you can safely ignore this warning and continue. -If you later have internet access and would like to add TeX live to your -project, please delete the respective files, then re-run configure as shown -below. +If you later have internet access and would like to add TeX Live to your +project, then please delete the following two files: rm .local/version-info/tex/texlive-ready-tlmgr + rm .build/software/tarballs/install-tl-unx.tar.gz + +and re-run configure: + ./project configure -e !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! EOF + sleep 10 # increase the chance that an interactive user reads this message fi -- cgit v1.2.1 From 016d938cec703a6d0062377ae8001c8bc7532073 Mon Sep 17 00:00:00 2001 From: Boud Roukema Date: Thu, 24 Jun 2021 21:19:38 +0200 Subject: Configuration: New check to see if /dev/shm allows execution On systems that allow it (like GNU/Linux systems), Maneage will build the necessary software in shared memory (a directory that is actually in the RAM, not on an SSD/HDD, on GNU/Linux systems, it is '/dev/shm'). This allows Maneage to operate faster and not harm the HDD/SSD with all the temporary writing of many small files. Until now, we would only check that this directory exists and that it has enough space. However, some systems also set the 'noexec' flag on shared memory for security reasons [1]. This causes Maneage to crash upon building of the software in later phases. With this commit, at the very start of the configuration step, and after all other shared-memory checks are done, a dummy executable script file is created there and its execution is tested. If it doesn't work, shared memory will not be used at all. In the process, the steps dealing with the software building directory in the configure script have been brought in one place and comments were added to further clarify every step. This commit was initially done by Boud Roukema and later edited by Mohammad Akhlaghi. [1] https://web.archive.org/web/20210624192819/https://serverfault.com/questions/72356/how-useful-is-mounting-tmp-noexec --- reproduce/software/shell/configure.sh | 88 ++++++++++++++++++++++++----------- 1 file changed, 61 insertions(+), 27 deletions(-) (limited to 'reproduce/software/shell') diff --git a/reproduce/software/shell/configure.sh b/reproduce/software/shell/configure.sh index 12690f7..0f7278f 100755 --- a/reproduce/software/shell/configure.sh +++ b/reproduce/software/shell/configure.sh @@ -1210,22 +1210,6 @@ if ! [ -d "$ictdir" ]; then mkdir "$ictdir"; fi itidir="$verdir"/tex if ! [ -d "$itidir" ]; then mkdir "$itidir"; fi -# Temporary software un-packing/build directory: if the host has the -# standard `/dev/shm' mounting-point, we'll do it in shared memory (on the -# RAM), to avoid harming/over-using the HDDs/SSDs. The RAM of most systems -# today (>8GB) is large enough for the parallel building of the software. -# -# For the name of the directory under `/dev/shm' (for this project), we'll -# use the names of the two parent directories to the current/running -# directory, separated by a `-' instead of `/'. We'll then appended that -# with the user's name (in case multiple users may be working on similar -# project names). Maybe later, we can use something like `mktemp' to add -# random characters to this name and make it unique to every run (even for -# a single user). -tmpblddir="$sdir"/build-tmp -rm -rf "$tmpblddir"/* "$tmpblddir" # If its a link, we need to empty its - # contents first, then itself. - @@ -1297,29 +1281,75 @@ rm -f .gnuastro -# Set the top-level shared memory location. + + +# Software building directory (possibly in RAM) +# --------------------------------------------- +# +# Building the software for the project will need the creation of many +# small temporary files that will ultimately be deleted. To avoid harming +# HDDs/SSDs and improve speed, it is therefore better to build them in the +# RAM when possible. The RAM of most systems today (>8GB) is large enough +# for the parallel building of the software. + +# Set the top-level shared memory location. Currently there is only one +# standard location (for GNU/Linux OSs), so doing this check here and the +# main job below may seem redundant. However, it is written separately from +# the main code below because later, we expect to add more possible +# mounting locations (for other OSs). if [ -d /dev/shm ]; then shmdir=/dev/shm else shmdir="" fi -# If a shared memory mounted directory exists and there is enough space -# there (in RAM), build a temporary directory for this project. -needed_space=2000000 +# If a shared memory mounted directory exists and has the necessary +# conditions, set that directory to build software. if [ x"$shmdir" != x ]; then + + # Make sure it has enough space. + needed_space=2000000 available_space=$(df "$shmdir" | awk 'NR==2{print $4}') if [ $available_space -gt $needed_space ]; then + + # Set the Maneage-specific directory within the shared + # memory. We'll use the names of the two parent directories to the + # current/running directory, separated by a `-' instead of + # `/'. We'll then appended that with the user's name (in case + # multiple users may be working on similar project names). + # + # Maybe later, we can use something like `mktemp' to add random + # characters to this name and make it unique to every run (even for + # a single user). dirname=$(pwd | sed -e's/\// /g' \ - | awk '{l=NF-1; printf("%s-%s",$l, $NF)}') + | awk '{l=NF-1; printf("%s-%s", $l, $NF)}') tbshmdir="$shmdir"/"$dirname"-$(whoami) if ! [ -d "$tbshmdir" ]; then mkdir "$tbshmdir"; fi + + # Some systems may protect '/dev/shm' against the right to execute + # programs by ordinary users. We thus need to check that the device + # allows execution within this directory by this user. + shmexecfile="$tbshmdir"/shm-execution-check.sh + rm -f $shmexecfile # We also don't want any existing flags. + cat > "$shmexecfile" < /dev/null; then tbshmdir=""; fi + rm "$shmexecfile" fi else tbshmdir="" fi -# If a shared memory directory was created set `build-tmp' to be a -# symbolic link to it. Otherwise, just build the temporary build -# directory under the project build directory. +# If a shared memory directory was created, set the software building +# directory to be a symbolic link to it. Otherwise, just build the +# temporary build directory under the project's build directory. +tmpblddir="$sdir"/build-tmp +rm -rf "$tmpblddir"/* "$tmpblddir" # If it is a link, we need to empty + # its contents first, then itself. if [ x"$tbshmdir" = x ]; then mkdir "$tmpblddir"; else ln -s "$tbshmdir" "$tmpblddir"; fi @@ -1330,6 +1360,9 @@ fi # Inform the user that the build process is starting # ------------------------------------------------- +# +# Everything is ready, let the user know that the building is going to +# start. if [ $printnotice = yes ]; then tsec=10 cat < Date: Mon, 4 Oct 2021 02:51:45 +0200 Subject: IMPORTANT: Updates to almost all software This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2 --- reproduce/software/shell/bashrc.sh | 6 +- reproduce/software/shell/configure.sh | 134 ++++++++++++++++++++--------- reproduce/software/shell/git-post-checkout | 10 +-- reproduce/software/shell/git-pre-commit | 10 +-- reproduce/software/shell/pre-make-build.sh | 30 +++++-- reproduce/software/shell/run-parts.in | 4 +- 6 files changed, 132 insertions(+), 62 deletions(-) (limited to 'reproduce/software/shell') diff --git a/reproduce/software/shell/bashrc.sh b/reproduce/software/shell/bashrc.sh index 23845d6..6bb871b 100755 --- a/reproduce/software/shell/bashrc.sh +++ b/reproduce/software/shell/bashrc.sh @@ -3,10 +3,10 @@ # To have better control over the environment of each analysis step (Make # recipe), besides having environment variables (directly included from # Make), it may also be useful to have a Bash startup file (this file). All -# of the Makefiles set this file as the `BASH_ENV' environment variable, so +# of the Makefiles set this file as the 'BASH_ENV' environment variable, so # it is loaded into all the Make recipes within the project. # -# The special `PROJECT_STATUS' environment variable is defined in every +# The special 'PROJECT_STATUS' environment variable is defined in every # top-level Makefile of the project. It defines the the state of the Make # that is calling this script. It can have three values: # @@ -29,7 +29,7 @@ # versions. # # -# Copyright (C) 2019-2021 Mohammad Akhlaghi +# Copyright (C) 2019-2022 Mohammad Akhlaghi # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/reproduce/software/shell/configure.sh b/reproduce/software/shell/configure.sh index 0f7278f..07381e6 100755 --- a/reproduce/software/shell/configure.sh +++ b/reproduce/software/shell/configure.sh @@ -2,8 +2,9 @@ # # Necessary preparations/configurations for the reproducible project. # -# Copyright (C) 2018-2021 Mohammad Akhlaghi -# Copyright (C) 2021 Raul Infante-Sainz +# Copyright (C) 2018-2022 Mohammad Akhlaghi +# Copyright (C) 2021-2022 Raul Infante-Sainz +# Copyright (C) 2022 Pedram Ashofteh Ardakani # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -112,7 +113,7 @@ absolute_dir () # Check file permission handling (POSIX-compatibility) # ---------------------------------------------------- # -# Check if a `given' directory handles permissions as expected. +# Check if a 'given' directory handles permissions as expected. # # This is to prevent a known bug in the NTFS filesystem that prevents # proper installation of Perl, and probably some other packages. This @@ -120,15 +121,15 @@ absolute_dir () # file, and examines whether the given directory handles the file # permissions as expected. # -# Returns `0' if everything is fine, and `255' otherwise. Choosing `0' is -# to mimic the `$ echo $?' behavior, while choosing `255' is to prevent +# Returns '0' if everything is fine, and '255' otherwise. Choosing '0' is +# to mimic the '$ echo $?' behavior, while choosing '255' is to prevent # misunderstanding 0 and 1 as true and false. # # ===== CAUTION! ===== # # -# Since there is a `set -e' before running this function, the whole script -# stops and exits IF the `check_permission' (or any other function) returns -# anything OTHER than `0'! So, only use this function as a test. Here's a +# Since there is a 'set -e' before running this function, the whole script +# stops and exits IF the 'check_permission' (or any other function) returns +# anything OTHER than '0'! So, only use this function as a test. Here's a # minimal example: # # if $(check_permission $some_directory) ; then @@ -136,7 +137,7 @@ absolute_dir () # fi ; check_permission () { - # Make a `junk' file, activate its executable flag and record its + # Make a 'junk' file, activate its executable flag and record its # permissions generally. local junkfile="$1"/check_permission_tmp_file rm -f "$junkfile" @@ -309,7 +310,9 @@ fi # system. Here, it is checked that this is the case, and if not, warn the user # about not having Xcode already installed. if [ x$on_mac_os = xyes ]; then - xcode=$(which xcodebuild) + + # 'which' isn't in POSIX, so we are using 'command -v' instead. + xcode=$(command -v xcodebuild) if [ x$xcode != x ]; then xcode_version=$(xcodebuild -version | grep Xcode) echo " " @@ -475,8 +478,8 @@ fi # See if we need the dynamic-linker (-ldl) # ---------------------------------------- # -# Some programs (like Wget) need dynamic loading (using `libdl'). On -# GNU/Linux systems, we'll need the `-ldl' flag to link such programs. But +# Some programs (like Wget) need dynamic loading (using 'libdl'). On +# GNU/Linux systems, we'll need the '-ldl' flag to link such programs. But # Mac OS doesn't need any explicit linking. So we'll check here to see if # it is present (thus necessary) or not. cat > $testsource < /dev/null 2>/dev/null; then - name=$(which wget) + + # 'which' isn't in POSIX, so we are using 'command -v' instead. + name=$(command -v wget) # By default Wget keeps the remote file's timestamp, so we'll have # to disable it manually. downloader="$name --no-use-server-timestamps -O"; elif type curl > /dev/null 2>/dev/null; then - name=$(which curl) + name=$(command -v curl) # - cURL doesn't keep the remote file's timestamp by default. - # - With the `-L' option, we tell cURL to follow redirects. + # - With the '-L' option, we tell cURL to follow redirects. downloader="$name -L -o" else cat < "$shmexecfile" < "$shmexecfile" < /dev/null; then tbshmdir=""; fi - rm "$shmexecfile" + + # If the file was successfully created, then make the file + # executable and see if it runs. If not, set 'tbshmdir' to an empty + # string so it is not used in later steps. In any case, delete the + # temporary file afterwards. + # + # We aren't adding '&> /dev/null' after the execution command + # because it can produce false failures randomly on some systems. + if [ -e "$shmexecfile" ]; then + + # Add the executable flag. + chmod +x "$shmexecfile" + + # The following line tries to execute the file. + if "$shmexecfile"; then + # Successful execution. The colon is a "no-op" (no + # operation) shell command. + : + else + tbshmdir="" + fi + rm "$shmexecfile" + else + tbshmdir="" + fi fi else tbshmdir="" fi + + + + # If a shared memory directory was created, set the software building # directory to be a symbolic link to it. Otherwise, just build the # temporary build directory under the project's build directory. @@ -1426,7 +1475,7 @@ fi # See if the linker accepts -Wl,-rpath-link # ----------------------------------------- # -# `-rpath-link' is used to write the information of the linked shared +# '-rpath-link' is used to write the information of the linked shared # library into the shared object (library or program). But some versions of # LLVM's linker don't accept it an can cause problems. # @@ -1459,13 +1508,13 @@ rm -rf $compilertestdir -# Paths needed by the host compiler (only for `basic.mk') +# Paths needed by the host compiler (only for 'basic.mk') # ------------------------------------------------------- # # At the end of the basic build, we need to build GCC. But GCC will build # in multiple phases, making its own simple compiler in order to build # itself completely. The intermediate/simple compiler doesn't recognize -# some system specific locations like `/usr/lib/ARCHITECTURE' that some +# some system specific locations like '/usr/lib/ARCHITECTURE' that some # operating systems use. We thus need to tell the intermediate compiler # where its necessary libraries and headers are. if [ x"$sys_library_path" != x ]; then @@ -1502,7 +1551,7 @@ fi # which will download the DOI-resolved webpage, and extract the Zenodo-URL # of the most recent version from there (using the 'coreutils' tarball as # an example, the directory part of the URL for all the other software are -# the same). This is not done if the option `--debug' is used. +# the same). This is not done if the option '--debug' is used. zenodourl="" user_backup_urls="" zenodocheck=.build/software/zenodo-check.html @@ -1535,6 +1584,7 @@ user_backup_urls="$user_backup_urls $zenodourl" # (compression program), GNU Make (that 'basic.mk' is written in), Dash # (minimal Bash-like shell) and Flock (to lock files and enable serial # download). +export on_mac_os ./reproduce/software/shell/pre-make-build.sh \ "$bdir" "$ddir" "$downloader" "$user_backup_urls" @@ -1672,7 +1722,7 @@ fi # software. prepare_name_version () { - # First see if the (possible) `*' in the input arguments corresponds to + # First see if the (possible) '*' in the input arguments corresponds to # anything. Note that some of the given directories may be empty (no # software installed). hasfiles=0 @@ -1773,9 +1823,9 @@ hw_class_fixed="$(echo $hw_class | sed -e 's/_/\\_/')" # --------------------------------- # # By the time the script reaches here the temporary software build -# directory should be empty, so just delete it. Note `tmpblddir' may be a +# directory should be empty, so just delete it. Note 'tmpblddir' may be a # symbolic link to shared memory. So, to work in any scenario, first delete -# the contents of the directory (if it has any), then delete `tmpblddir'. +# the contents of the directory (if it has any), then delete 'tmpblddir'. .local/bin/rm -rf $tmpblddir/* $tmpblddir diff --git a/reproduce/software/shell/git-post-checkout b/reproduce/software/shell/git-post-checkout index 7a90108..d49504d 100755 --- a/reproduce/software/shell/git-post-checkout +++ b/reproduce/software/shell/git-post-checkout @@ -1,14 +1,14 @@ #!@BINDIR@/bash # # The example hook script to store the metadata information of version -# controlled files (with each commit) using the `metastore' program. +# controlled files (with each commit) using the 'metastore' program. # # Copyright (C) 2016 Przemyslaw Pawelczyk -# Copyright (C) 2018-2021 Mohammad Akhlaghi +# Copyright (C) 2018-2022 Mohammad Akhlaghi # -# This script is taken from the `examples/hooks/pre-commit' file of the -# `metastore' package (installed within the project, with an MIT license -# for copyright). We have just changed the name of the `MSFILE' and also +# This script is taken from the 'examples/hooks/pre-commit' file of the +# 'metastore' package (installed within the project, with an MIT license +# for copyright). We have just changed the name of the 'MSFILE' and also # set special characters for the installation location of meta-store so our # own installation is found by Git. # diff --git a/reproduce/software/shell/git-pre-commit b/reproduce/software/shell/git-pre-commit index 85d3474..10ad710 100755 --- a/reproduce/software/shell/git-pre-commit +++ b/reproduce/software/shell/git-pre-commit @@ -1,10 +1,10 @@ #!@BINDIR@/bash # # The example hook script to store the metadata information of version -# controlled files (with each commit) using the `metastore' program. +# controlled files (with each commit) using the 'metastore' program. # # Copyright (C) 2016 Przemyslaw Pawelczyk -# Copyright (C) 2018-2021 Mohammad Akhlaghi +# Copyright (C) 2018-2022 Mohammad Akhlaghi # # WARNING: # @@ -17,9 +17,9 @@ # git reset HEAD -- .metadata # git checkout HEAD -- .metadata # -# This script is taken from the `examples/hooks/pre-commit' file of the -# `metastore' package (installed within the project, with an MIT license -# for copyright). Here, the name of the `MSFILE' and also set special +# This script is taken from the 'examples/hooks/pre-commit' file of the +# 'metastore' package (installed within the project, with an MIT license +# for copyright). Here, the name of the 'MSFILE' and also set special # characters for the installation location of meta-store so our own # installation is found by Git. # diff --git a/reproduce/software/shell/pre-make-build.sh b/reproduce/software/shell/pre-make-build.sh index a033963..e7de93d 100755 --- a/reproduce/software/shell/pre-make-build.sh +++ b/reproduce/software/shell/pre-make-build.sh @@ -2,7 +2,7 @@ # # Very basic tools necessary to start Maneage's default building. # -# Copyright (C) 2020-2021 Mohammad Akhlaghi +# Copyright (C) 2020-2022 Mohammad Akhlaghi # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -185,6 +185,19 @@ build_program() { # All others accept the configure script. ./configure --prefix="$instdir" $configoptions + # In Flock 0.4.0 there is a crash that can be fixed by simply + # replacing '%1u' with '%ld' on GNU/Linux and '%d' on macOS. This + # has been reported to flock maintainers: + # https://github.com/discoteq/flock/issues/33 + if [ x$progname = xflock ]; then + case $on_mac_os in + yes) sed -e's/\%1u/\%d/' src/flock.c > src/flock-new.c;; + no) sed -e's/\%1u/\%ld/' src/flock.c > src/flock-new.c;; + *) echo "pre-make-build.sh: '$on_mac_os' unrecognized value for on_mac_os";; + esac + mv src/flock-new.c src/flock.c + fi + # To build GNU Make, we don't want to assume the existance of a # Make program, so we use its 'build.sh' script and its own built # 'make' program to install itself. @@ -192,7 +205,7 @@ build_program() { /bin/sh build.sh ./make install else - make + make V=1 make install fi fi @@ -235,13 +248,20 @@ build_program # '--disable-dependency-tracking' configure-time option is necessary so # Make doesn't check for an existing 'make' implementation (recall that we # aren't assuming any 'make' on the host). +# +# If GNU Guile is already present on the host system, Make will try to link +# with it, and this will cause dependency problems later. So we have +# distabled Guile. If a project needs the Guile extensions of Make, we need +# to add a build rule for Guile in Maneage, with a special Guile-enabled +# Make that has a different executable name (using the '--program-prefix=' +# configure option) from the "default" make (which is this one!). progname="make" progname_tex="GNU Make" url=$(awk '/^'$progname'-url/{print $3}' $urlfile) version=$(awk '/^'$progname'-version/{print $3}' $versionsfile) tarball=$progname-$version.tar.lz download_tarball -build_program --disable-dependency-tracking +build_program "--disable-dependency-tracking --without-guile" @@ -274,11 +294,11 @@ fi # ----- # # Flock (or file-lock) is necessary to serialize operations when -# necessary. GNU/Linux machines have it as part of their `util-linux' +# necessary. GNU/Linux machines have it as part of their 'util-linux' # programs. But to be consistent in non-GNU/Linux systems, we will be using # our own build. # -# The reason that `flock' is built here is that generally the building of +# The reason that 'flock' is built here is that generally the building of # software is done in parallel, but we need it to serialize the download # process of the software tarballs to avoid network complications when too # many simultaneous download commands are called. diff --git a/reproduce/software/shell/run-parts.in b/reproduce/software/shell/run-parts.in index 7e649b1..a6db9e0 100755 --- a/reproduce/software/shell/run-parts.in +++ b/reproduce/software/shell/run-parts.in @@ -10,8 +10,8 @@ # However, it didn't have a copyright statement. So one is being added # here. # -# Copyright (C) 2021 Authors mentioned above. -# Copyright (C) 2020-2021 Mohammad Akhlaghi +# Copyright (C) 2022 Authors mentioned above. +# Copyright (C) 2020-2022 Mohammad Akhlaghi # # This script is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by -- cgit v1.2.1 From 597d1df2aa8131ef39fb32ba339798209e4ba313 Mon Sep 17 00:00:00 2001 From: Pedram Ashofteh Ardakani Date: Mon, 18 Apr 2022 00:16:26 +0430 Subject: Updated Git, Coreutils and Emacs, new script to prepare tarballs Until now, one had to follow the instructions from [1] to prepare a standard software tarball before merging with the low-level tarballs-software repository [2]. The script only worked for '.tar.gz' suffix and was only available as a comment on Savannah (in [1]). With this commit, the script has been imported into Maneage as 'reproduce/software/shell/tarball-prepare.sh' to simplify future software updates. It work with all supported '.tar.*' suffixes (of the upstream tarball repository) and will convert the tarballs to Maneage's standard format. Also, this script has a minimal argument parser and can skip the tarballs that are already unpacked, allowing faster tests. This script was used to update the versions of: Coreutiles 9.0 --> 9.1 Git 2.34 --> 2.36 Emacs 27.2 --> 28.1 The main motive behind this update was Git which announced a vulnerability issue [3] and suggested an update to the latest version as soon as possible. More detail is described in this github blog [4], but in summary, it was a security issue on multi-user systems that has been found and fixed by Git developers. Since Maneage is often installed on such shared systems, it was important to make this update. GNU Coreutils and GNU Emacs were also updated because they are also commonly used. The following improvements have also done with this commit: - .gitignore: ignore emacs auto-save files (that end with a '#') - README-hacking.md: In the checklist for updating the Maneage branch, the no-longer-necessary '--decorate' option of Git was removed from the command to check the general branch history. [1] https://savannah.nongnu.org/task/?15699 [2] https://git.maneage.org/tarballs-software.git/ [3] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/ [4] https://github.blog/2022-04-12-git-security-vulnerability-announced/ --- reproduce/software/shell/tarball-prepare.sh | 181 ++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100755 reproduce/software/shell/tarball-prepare.sh (limited to 'reproduce/software/shell') diff --git a/reproduce/software/shell/tarball-prepare.sh b/reproduce/software/shell/tarball-prepare.sh new file mode 100755 index 0000000..ccc9318 --- /dev/null +++ b/reproduce/software/shell/tarball-prepare.sh @@ -0,0 +1,181 @@ +#!/bin/bash + +# Script to convert all files (tarballs in any format; just recognized +# by 'tar') within an 'odir' to a unified '.tar.lz' format. +# +# The inputs are assumed to be formatted with 'NAME_VERSION', and only for +# the names, we are currently assuming '.tar.*' (for the 'sed' +# command). Please modify/generalize accordingly. +# +# It will unpack the source in a certain empty directory with the +# 'tmpunpack' suffix, and rename the top directory to the requested format +# of NAME-VERSION also. So irrespective of the name of the top original +# tarball directory, the resulting tarball's top directory will have a name +# formatting of NAME-VERSION. +# +# Discussion: https://savannah.nongnu.org/task/?15699 +# +# Copyright (C) 2022 Mohammad Akhlaghi +# Copyright (C) 2022 Pedram Ashofteh Ardakani +# Released under GNU GPLv3+ + +# Abort the script in case of an error. +set -e + + + + + +# Default arguments +odir= +idir= +quiet= +basedir=$PWD + + +# The --help output +print_help() { + cat <