diff options
| author | Mohammad Akhlaghi <mohammad@akhlaghi.org> | 2021-10-04 02:51:45 +0200 |
|---|---|---|
| committer | Mohammad Akhlaghi <mohammad@akhlaghi.org> | 2022-01-21 01:15:24 +0100 |
| commit | 8463df97c6f26ec4d22cd5828bb0574fd5e450d2 (patch) | |
| tree | dbaa2e7c2fc44856eb98555b79c6814f210a6c17 /project | |
| parent | 775fc036e0091f05ff56e41b855bc416b9ed36c8 (diff) | |
IMPORTANT: Updates to almost all software
This commit primarily affects the configuration step of Maneage'd projects,
and in particular, updated versions of the many of the software (see
P.S.). So it shouldn't affect your high-level analysis other than the
version bumps of the software you use (and the software's possibly
improve/changed behavior).
The following software (and thus their dependencies) couldn't be updated as
described below:
- Cryptography: isn't building because it depends on a new
setuptools-rust package that has problems
(https://savannah.nongnu.org/bugs/index.php?61731), so it has been
commented in 'versions.conf'.
- SecretStorage: because it depends on Cryptography.
- Keyring: because it depends on SecretStorage.
- Astroquery: because it depends on Keyring.
This is a "squashed" commit after rebasing a development branch of 60
commits corresponding to a roughly two-month time interval. The following
people contributed to this branch.
- Boudewijn Roukema added all the R software infrastructure and the R
packages, as well as greatly helping in fixing many bugs during the
update.
- Raul Infante-Sainz helped in testing and debugging the build.
- Pedram Ashofteh Ardakani found and fixed a bug.
- Zahra Sharbaf helped in testing and found several bugs.
Below a description of the most noteworthy points is given.
- Software tarballs: all updated software now have a unified format
tarball (ustar; if not possible, pax) and unified compression (Lzip) in
Maneage's software repository in Zenodo
(https://doi.org/10.5281/zenodo.3883409). For more on this See
https://savannah.nongnu.org/task/?15699 . This won't affect any extra
software you would like to add; you can use any format recognized by
GNU Tar, and all common compression algorithms. This new requirement is
only for software that get merged to the core Maneage branch.
- Metastore (and thus libbsd and libmd) moved to highlevel: Metastore
(and the packages it depends on) is a high-level product that is only
relevant during the project development (like Emacs!): when the user
wants the file meta data (like dates) to be unchanged after checking
out branches. So it should be considered a high-level software, not
basic. Metastore also usually causes many more headaches and error
messages, so personally, I have stopped using it! Instead I simply
merge my branches in a separate clone, then pull the merge commit: in
this way, the files of my project aren't re-written during the checkout
phase and therefore their dates are untouched (which can conflict with
Make's dates on configuration files).
- The un-official cloned version of Flex (2.6.4-91 until this commit) was
causing problems in the building of Netpbm, so with this commit, it has
been moved back to version 2.6.4.
- Netpbm's official page had version 10.73.38 as the latest stable
tarball that was just released in late 2021. But I couldn't find our
previously-used version 10.86.99 anywhere (to see when it was released
and why we used it! Its at last more than one year old!). So the
official stable version is being used now.
- Improved instructions in 'README.md' for building software environment
in a Docker container (while having project source and output data
products on the local system; including the usage of the host's
'/dev/shm' to speed up temporary operations).
- Until now, the convention in Maneage was to put eight SPACE characters
before the comment lines within recipes. This was done because by
default GNU Emacs (also many other editors) show a TAB as eight
characters. However, in other text editors, online browsers, or even
the Git diff, a TAB can correspond to a different number of
characters. In such cases, the Maneage recipes wouldn't look too
interesting (the comments and the recipe commands would show a
different indentation!).
With this commit, all the comment lines in the Makefiles within the
core Maneage branch have a hash ('#') as their first character and a
TAB as the second. This allows the comment lines in recipes to have the
same indentation as code; making the code much more easier to read in a
general scenario including a 'git diff' (editor agnostic!).
P.S. List of updated software with their old and new versions
- Software with no version update are not mentioned.
- The old version of newly added software are shown with '--'.
Name (Basic) Old version New version
------------ ----------- -----------
Bzip2 1.0.6 1.0.8
CURL 7.71.1 7.79.1
Dash 0.5.10.2 0.5.11.5
File 5.39 5.41
Flock 0.2.3 0.4.0
GNU Bash 5.0.18 5.1.8
GNU Binutils 2.35 2.37
GNU Coreutils 8.32 9.0
GNU GCC 10.2.0 11.2.0
GNU M4 1.4.18 1.4.19
GNU Readline 8.0 8.1.1
GNU Tar 1.32 1.34
GNU Texinfo 6.7 6.8
GNU diffutils 3.7 3.8
GNU findutils 4.7.0 4.8.0
GNU gmp 6.2.0 6.2.1
GNU grep 3.4 3.7
GNU gzip 1.10 1.11
GNU libunistring 0.9.10 1.0
GNU mpc 1.1.0 1.2.1
GNU mpfr 4.0.2 4.1.0
GNU nano 5.2 6.0
GNU ncurses 6.2 6.3
GNU wget 1.20.3 1.21.2
Git 2.28.0 2.34.0
Less 563 590
Libxml2 2.9.9 2.9.12
Lzip 1.22-rc2 1.22
OpenSLL 1.1.1a 3.0.0
Patchelf 0.10 0.13
Perl 5.32.0 5.34.0
Podlators -- 4.14
Name (Highlevel) Old version New version
---------------- ----------- -----------
Apachelog4cxx 0.10.0-603 0.12.1
Astrometry.net 0.80 0.85
Boost 1.73.0 1.77.0
CFITSIO 3.48 4.0.0
Cmake 3.18.1 3.21.4
Eigen 3.3.7 3.4.0
Expat 2.2.9 2.4.1
FFTW 3.3.8 3.3.10
Flex 2.6.4-91 2.6.4
Fontconfig 2.13.1 2.13.94
Freetype 2.10.2 2.11.0
GNU Astronomy Utilities 0.12 0.16.1-e0f1
GNU Autoconf 2.69.200-babc 2.71
GNU Automake 1.16.2 1.16.5
GNU Bison 3.7 3.8.2
GNU Emacs 27.1 27.2
GNU GDB 9.2 11.1
GNU GSL 2.6 2.7
GNU Help2man 1.47.11 1.48.5
Ghostscript 9.52 9.55.0
ICU -- 70.1
ImageMagick 7.0.8-67 7.1.0-13
Libbsd 0.10.0 0.11.3
Libffi 3.2.1 3.4.2
Libgit2 1.0.1 1.3.0
Libidn 1.36 1.38
Libjpeg 9b 9d
Libmd -- 1.0.4
Libtiff 4.0.10 4.3.0
Libx11 1.6.9 1.7.2
Libxt 1.2.0 1.2.1
Netpbm 10.86.99 10.73.38
OpenBLAS 0.3.10 0.3.18
OpenMPI 4.0.4 4.1.1
Pixman 0.38.0 0.40.0
Python 3.8.5 3.10.0
R 4.0.2 4.1.2
SWIG 3.0.12 4.0.2
Util-linux 2.35 2.37.2
Util-macros 1.19.2 1.19.3
Valgrind 3.15.0 3.18.1
WCSLIB 7.3 7.7
Xcb-proto 1.14 1.14.1
Xorgproto 2020.1 2021.5
Name (Python) Old version New version
------------- ----------- -----------
Astropy 4.0 5.0
Beautifulsoup4 4.7.1 4.10.0
Beniget -- 0.4.1
Cffi 1.12.2 1.15.0
Cryptography 2.6.1 36.0.1
Cycler 0.10.0 0.11.0+}
Cython 0.29.21 0.29.24
Esutil 0.6.4 0.6.9
Extension-helpers -- 0.1
Galsim 2.2.1 2.3.3
Gast -- 0.5.3
Jinja2 -- 3.0.3
MPI4py 3.0.3 3.1.3
Markupsafe -- 2.0.1
Numpy 1.19.1 1.21.3
Packaging -- 21.3
Pillow -- 8.4.0
Ply -- 3.11
Pyerfa -- 2.0.0.1
Pyparsing 2.3.1 3.0.4
Pythran -- 0.11.0
Scipy 1.5.2 1.7.3
Setuptools 41.6.0 58.3.0
Six 1.12.0 1.16.0
Uncertainties 3.1.2 3.1.6
Wheel -- 0.37.0
Name (R) Old version New version
-------- ----------- -----------
Cli -- 2.5.0
Colorspace -- 2.0-1
Cowplot -- 1.1.1
Crayon -- 1.4.1
Digest -- 0.6.27
Ellipsis -- 0.3.2
Fansi -- 0.5.0
Farver -- 2.1.0
Ggplot2 -- 3.3.4
Glue -- 1.4.2
GridExtra -- 2.3
Gtable -- 0.3.0
Isoband -- 0.2.4
Labeling -- 0.4.2
Lifecycle -- 1.0.0
Magrittr -- 2.0.1
MASS -- 7.3-54
Mgcv -- 1.8-36
Munsell -- 0.5.0
Pillar -- 1.6.1
R-Pkgconfig -- 2.0.3
R6 -- 2.5.0
RColorBrewer -- 1.1-2
Rlang -- 0.4.11
Scales -- 1.1.1
Tibble -- 3.1.2
Utf8 -- 1.2.1
Vctrs -- 0.3.8
ViridisLite -- 0.4.0
Withr -- 2.4.2
Diffstat (limited to 'project')
| -rwxr-xr-x | project | 59 |
1 files changed, 33 insertions, 26 deletions
@@ -1,10 +1,10 @@ #!/bin/sh # # High-level script to manage the project. -# Run `./project --help' for a description of how to use it. +# Run './project --help' for a description of how to use it. # -# Copyright (C) 2019-2021 Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Copyright (C) 2021 Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2019-2022 Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Copyright (C) 2021-2022 Raul Infante-Sainz <infantesainz@gmail.com> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -30,7 +30,7 @@ set -e # Default option values -jobs=0 # 0 is for the default for the `configure.sh' script. +jobs=0 # 0 is for the default for the 'configure.sh' script. group= debug= host_cc=0 @@ -59,9 +59,9 @@ scriptname="./project" # into the respective variable. # # Each option has two lines because we want to process both these formats: -# `--name=value' and `--name value'. The former (with `=') is a single +# '--name=value' and '--name value'. The former (with '=') is a single # command-line argument, so we just need to shift the counter by one. The -# latter (without `=') is two arguments, so we'll need two shifts. +# latter (without '=') is two arguments, so we'll need two shifts. # # Note on the case strings: for every option, we need three lines: one when # the option name and value are separate. Another when there is an equal @@ -172,7 +172,6 @@ do make) func_operation_set $1; shift;; shell) func_operation_set $1; shift;; - # Configure options: -b|--build-dir) build_dir="$2"; check_v "$1" "$build_dir"; shift;shift;; -b=*|--build-dir=*) build_dir="${1#*=}"; check_v "$1" "$build_dir"; shift;; @@ -207,14 +206,21 @@ do # Make options # ------------ # - # Note that Make's `debug' can take values, but when called without any - # value, it is like giving it a value of `a'): + # Note that Make's 'debug' can take values, but when called without any + # value, it is like giving it a value of 'a'): --highlight-new) highlightnew=1; shift;; --highlight-new=*) on_off_option_error --highlight-new;; --highlight-notes) highlightnotes=1; shift;; --highlight-notes=*) on_off_option_error --highlight-notes;; - -d|--debug) if [ x"$2" = x ]; then debug=a; shift; - else debug="$2"; check_v debug "$debug"; shift;shift; fi;; + -d|--debug) if [ x$operation = x ]; then + echo "Please set the operation before calling '--debug'"; exit 1 + elif [ x$operation = xconfigure ]; then debug=a; shift; + elif [ x$operation = xmake ]; then + if [ x"$2" = x ]; then echo "In make-mode, '--debug' needs a value"; exit 1 + else debug="$2"; check_v debug "$debug"; shift;shift; fi + else + echo "Operation '$operation' not recognized, please use 'configure' or 'make'" + fi;; -d=*|--debug=*) debug="${1#*=}"; check_v debug "$debug"; shift;; -d*) debug=$(echo "$1" | sed -e's/-d//'); check_v debug "$debug"; shift;; -p|--prepare-redo) prepare_redo=1; shift;; @@ -235,8 +241,8 @@ done # Check configuration status # -------------------------- if ! [ x$check_config = x ]; then - # Find the color option to pass to `ls'. Note that `--color' (for GNU - # Coreutils `ls') should be checked first because it also has `-G', but + # Find the color option to pass to 'ls'. Note that '--color' (for GNU + # Coreutils 'ls') should be checked first because it also has '-G', but # for something else. if ls --color 2> /dev/null > /dev/null; then coloropt="--color=auto" elif ls -G 2> /dev/null > /dev/null; then coloropt="-G" @@ -282,7 +288,7 @@ EOF check=$(ls .local/version-info/python/) if ! [ "x$check" = x ]; then printresults=1 - ln -s .local/version-info/python/* $checkdir/ + ln -s "$(pwd)"/.local/version-info/python/* $checkdir/ fi check=$(ls .local/version-info/proglib/) if ! [ "x$check" = x ]; then @@ -296,7 +302,7 @@ EOF # Then sort all the links based on the most recent dates of the # files they link to (with '-L'). - ls -Llt $checkdir \ + ls -Llt $checkdir \ | awk '/^-/ && c++<5 {printf "[at %s] %s\n", $(NF-1), $NF}' fi else @@ -372,10 +378,11 @@ EOF # Run operations in controlled environment # ---------------------------------------- controlled_env() { + # Get the full address of the build directory: bdir=`.local/bin/realpath .build` - # Remove all existing environment variables (with `env -i') and only + # Remove all existing environment variables (with 'env -i') and only # use some pre-defined environment variables, then build the project. envmake=".local/bin/env -i HOME=$bdir sys_rm=$(which rm) $gopt" envmake="$envmake highlightnew=$highlightnew" @@ -383,7 +390,7 @@ controlled_env() { envmake="$envmake --no-builtin-rules --no-builtin-variables -f $1" if ! [ x"$debug" = x ]; then envmake="$envmake --debug=$debug"; fi - # Set the number of jobs. Note that for the `configure.sh' script the + # Set the number of jobs. Note that for the 'configure.sh' script the # default value has to be 0, so the default is the maximum number of # threads. But here, the default value is 1. if ! [ x"$jobs" = x0 ]; then envmake="$envmake -j$jobs"; fi @@ -415,7 +422,7 @@ case $operation in # # In some scenarios (for example when using a tarball from arXiv), # it may happen that the host server has removed the executable - # flags of all the files. In `README.md' we instruct the readers on + # flags of all the files. In 'README.md' we instruct the readers on # setting the executable flag of this script. But we don't want the # user to have to worry about any other file that needs an # executable flag. @@ -456,7 +463,7 @@ case $operation in # Set the group writing permission for everything in the # installed software directory. The common build process sets # the writing permissions of the installed programs/libraries - # to `755'. So group members can't write over a file. This + # to '755'. So group members can't write over a file. This # creates problems when another group member wants to update # the software for example. We thus need to manually add the # group writing flag to all installed software files. @@ -473,17 +480,17 @@ case $operation in make) # Make sure the configure script has been completed properly - # (`configuration-done.txt' exists). + # ('configuration-done.txt' exists). if ! [ -f .build/software/configuration-done.txt ]; then configuration_necessary fi # Run data preparation phase (optionally build Makefiles with - # special values for optimizing the main `top-make.mk'). But note + # special values for optimizing the main 'top-make.mk'). But note # that data preparation is only done automatically the first time - # the project is built (when `.build/software/preparation-done.mk' + # the project is built (when '.build/software/preparation-done.mk' # doesn't yet exist). After that, if the user wants to re-do the - # preparation they have to use the `--prepare-redo' option. + # preparation they have to use the '--prepare-redo' option. if ! [ -f .build/software/preparation-done.mk ] \ || [ x"$prepare_redo" = x1 ]; then controlled_env reproduce/analysis/make/top-prepare.mk @@ -497,7 +504,7 @@ case $operation in shell) # Make sure the configure script has been completed properly - # (`configuration-done.txt' exists). + # ('configuration-done.txt' exists). if ! [ -f .build/software/configuration-done.txt ]; then configuration_necessary fi @@ -519,7 +526,7 @@ case $operation in OMPI_MCA_plm_rsh_agent=/bin/false \ PYTHONPATH="$instdir"/lib/python/site-packages \ PYTHONPATH3="$instdir"/lib/python/site-packages \ - PS1="[\[\033[32m\](maneage)\[\033[00m\] \u@\h \W]$ " \ + PS1="[\[\033[01;35m\]maneage@\h \W\[\033[32m\]\[\033[00m\]]$ " \ "$instdir"/bin/bash ;; @@ -535,5 +542,5 @@ Please run with '--help' for more information. EOF exit 1 - ;; + ;; esac |