Until this commit, the AAS journals branch was based on Maneage commit 8161194 (dated 2022-05-22). However, maneage has been updated recently so it is necessary to also update this branch. With this commit, the merge has been done (and tested).
SUMMARY: no change is necessary for your pipeline after upgrading to this
commit.
Until this commit, the following noteworthy portability problems existed
that have been fixed as described in each item.
- In commit 8908587 (titled "IMPORTANT: 73 software upgrades and added
offline mode"; committed on 2025-02-11); the versions of all (except
those that were not updated upstream) basic software as well as the
high-level software necessary for Gnuastro were updated. This included
Python. However, the update of Python could cause crashes in the
Python-related software that are also in Maneage but not tested for
that commit.
-- With this commit, Python has been revered back to 3.10.6 (from
3.13.2) and it has been taken to a dedicated part of 'versions.conf'
(reminding the reader that the version of Python should only be
updated by the Python maintainer: to ensure it doesn't conflict with
Python-based builds).
- Following the update of XZ Utils in 8908587, the custom build recipe
that was necessary is no longer necessary. This was reported by Boud
Roukema (see https://savannah.nongnu.org/bugs/index.php?62700).
-- With this commit, XZ Utils uses our basic high-level build scripts.
- CMake could use out-of-Maneage libraries for those that we already had
in Maneage on some systems. This was reported by Boud Roukema (see
https://savannah.nongnu.org/bugs/?63043)
-- With this commit, CMake is built with '--no-system-libs'; so it
builds and statically links to all its necessary libraries
internally (not confusing between Maneage and the host).
- Building Binutils 2.39 was likely to fail on older GCC versions due to
special features (such as the 'gprofng' feature when building with GCC
6.2.0). This was reported by Boud Roukema in
https://savannah.nongnu.org/bugs/index.php?63242.
-- With this commit, the 'gprofng' feature of Binutils has been
disabled by default to allow reproducibility on older systems. If
you need it, remove the '--enable-gprofng=no' option in the build
recipe of Binutils.
- The Maneage tarball repository is now also kept in 'gitlab.cefca.es'.
-- With this commit, a new server URL has been added for this.
- The list of TeXLive packages included some packages that get installed
in the basic scheme that we do for the initial setup of TeXLive.
-- With this commit, the extra packages have been removed.
Summary: This is a software update to make Maneage more portable and up to
date. It does not involve any Maneage infrastructure changes. You should
just re-build your project to make sure the updated software haven't
removed/changed any of their features that you were using.
Until now, the software in Maneage had not been updated for almost 2.5
years (since August 2022) and were starting to get dated! Also, when no
internet is available (for example in a HPC), the Zenodo check would always
cause a crash and needed to be commented manually.
With this commit, all the basic software that had been updated as well as
the dependencies of Gnuastro (which we commit to maintaining) have been
updated after a check of their webpage (see the full list below). To fix
the offline issue, a new '--offline' option has been added to the project
script. The other following changes were made:
- tarball-prepare: has become more resilient (for dealing with file name
with spaces or Lzip, which should not be compressed).
- Readme: improve documentation when used in Docker.
Below you can see the list of basic software that have been updated. The
only basic software that have not been updated upstream since the previous
Maneage software update are 'bison', 'flock', 'm4', 'pkgconfig', 'unzip'
and 'zip'.
bash 5.2-rc2 5.2.37
binutils 2.39 2.43.1
coreutils 9.1 9.6
curl 7.84.0 8.11.1
dash 0.5.11-057cd65 0.5.12
diffutils 3.8 3.10
file 5.42 5.46
findutils 4.9.0 4.10.0
gawk 5.1.1 5.3.1
gcc 12.1.0 14.2.0
gettext 0.21 0.23.1
git 2.37.1 2.48.1
gmp 6.2.1 6.3.0
grep 3.7 3.11
gzip 1.12 1.13
help2man 1.49.2 1.49.3
isl 0.24 0.27
less 590 668
libiconv 1.17 1.18
libtool 2.4.7 2.5.4
libunistring 1.0 1.3
libxml2 2.9.12 2.13.5
lzip 1.23 1.25
make 4.3 4.4.1
mpc 1.2.1 1.3.1
mpfr 4.1.0 4.2.1
nano 6.4 8.3
ncurses 6.3 6.5
openssl 3.0.5 3.4.0
perl 5.36.0 5.40.1
podlators 4.14 6.0.2
readline 8.2-rc2 8.2.13
sed 4.8 4.9
tar 1.34 1.35
texinfo 6.8 7.2
wget 1.21.2 1.25.0
which 2.21 2.23
xz 5.2.5 5.6.3
zlib 1.2.11 1.3.1
The high-level software that have been updated.
autoconf 2.71 2.72
automake 1.16.5 1.17
cfitsio 4.1.0 4.5.0
cmake 3.24.0 3.31.5
expat 2.4.1 2.6.4
fontconfig 2.14.0 2.16.0
freetype 2.11.0 2.13.3
ghostscript 9.56.1 10.04.0
gnuastro 0.18 0.23
gsl 2.7 2.8
libffi 3.4.2 3.4.7
libgit2 1.3.0 1.9.0
libice 1.0.10 1.1.2
libidn 1.38 1.42
libjpeg 9e 9f
libpaper 1.1.28 1.1.29
libpng 1.6.37 1.6.46
libpthread-stubs 0.4 0.5
libsm 1.2.3 1.2.5
libtiff 4.4.0 4.7.0
libxau 1.0.9 1.0.12
libxcb 1.15 1.17
libxdmcp 1.1.3 1.1.5
libxext 1.3.4 1.3.6
libxt 1.2.1 1.3.1
python 3.10.6 3.13.12
util-macros 1.19.3 1.20.2
wcslib 7.11 8.4
xcb-proto 1.15 1.17
xorgproto 2022.1 2024.1
xtrans 1.4.0 1.5.2
SUMMARY: no change necessary in your project, this commit only involves
changes in how already-existing software are built. Some handy options have
also been added to the top-level project script and the copyright years
have been updated.
Until now, if the host's '/bin/sh' had conflicts with the Maneage
environment, the configuration of Maneage would crash as soon as we entered
the building of high-level software. The full scenario is described in the
comments of the newly added 'reproduce/software/shell/prep-source.sh'. This
is most relevant when building older Maneage'd project in newer
environments.
With this commit, the following changes were made to avoid the problem
above:
- Maneage edits the source code of all installed software to replace
'/bin/sh' with Maneage's own shell before the programs are
built. Through this, we were able to solve the problem described
above.
- The portable '#!/usr/bin/env sh' shebangs are now used at the start of
the scripts that run during configure time so it uses the first
available shell that it finds in its PATH (the system's before Dash is
built), then Dash, and after Dash is built, Bash.
- For TeXLive, since we don't install it from source, it was necessary to
add the libraries necessary for the local '/bin/sh' in LD_LIBRARY_PATH.
Some high-level options have been added to the './project' script to
simplify certain operations:
--keep-going: do not stop upon the first crash, but keep going on to
build targets until all build-able targets have been built. This is
very useful for debugging large pipelines and allows you to isolate the
problematic part of your project.
--highlight-all: equivalent to calling both '--highlight-new' &
'--highlight-notes'.
Until now, the @-sign with the email of 'tex/src/references.tex' would conflict with BibTeX (that is necessary with AAS journals) and cause a crash in the build. With this commit, to fix the problem the '@' sign has been replaced by '::at::'.
Until now, the core Maneage branch was very generic and customizing for each separate journal required some time to prepare the LaTeX style. With this commit, a first attempt at customization of Maneage for the LaTeX styles of the AAS journals (AASTeX) has been implemented.
SUMMARY: no change is necessary in your project, unless you use the Fortran features of WCSLIB in your project. Until now, there were two compilation failures on recent macOS computers with an M1 CPU: Less would crash because it couldn't find the relevant PCRE (perl-compatible regular expression) libraries and WCSLIB would crash because the LLVM compiler's Fortran features could not be built. With this commit, both issues have been fixed by disabling the relevant feature. Extensive comments have been placed in both places in case your project needs these features, so please see the comments in the relevant part of 'reproduce/software/make/basic.mk' for Less and 'reproduce/software/make/high-level.mk' for WCSLIB. In fact the previous solution (where we would not have Fortran features in WCSLIB on macOS systems was problematic and non-reproducibile (the features of WCSLIB depended on the operating system!). Another minor change was that for macOS, we now directly use the version-string of WCSLIB to fix the internal linking issue there. As a result, WCSLIB is no longer a "Version-dependent build" software (in 'reproduce/software/config/versions.conf'). Recall that these are software that when changing the version, it is also necessary to inspect their build recipe. These two issues and their fix were discovered and fixed with the help of James Robinson.
SUMMARY: just house-cleaning, no need to do anything major in your branch. Just update the copyright years in files that you have added. Until now, the latest copyright years of the whole Maneage source code was 2022! As of this commit, we have already moved to 2023 for 5 months! Furthermore, there were a few other minor issues that needed correction: - The URL to download input datasets wasn't quoted in 'initialize.mk' or the download script! As a result, when the input URL had characters that are meaningful to the shell (like '&'), the download command would not work. - The only program that had 'make check' in the 'basic.mk' programs was MPFR. At that stage, we still haven't built our own compiler at this stage, this is not accurate. - The 'pyerfa' and 'extension-helpers' packages in Python need 'setuptools_scm' on some systems. But until now, it was not in the list of their prerequisites. With this commit, all the issues above have been corrected.
SUMMARY: Nothing special is necessary for your existing projects. This
commit just addds two new features (read the commit description for more):
1. To provide a user and password to servers that need authentication
before they allow downloading of proprietary data,
2. To use the FITS Standard's DATASUM for file verification (for cases
where the file is not static on the server, and is generated upon
receiving your download request).
Until now, Maneage didn't have any infrastructure for databases that
require authentication (through a user or password, when calling
'wget'). Furthermore, when the downloaded file is automatically generated
by the server upon request, the server usually adds metadata (like file
date, or query number and etc) in the header. Therefore the simple SHA256
checksum of the file would differ on every download! This made it very hard
to verify if the data (not headers) are unchanged.
With this commit, both these problems have been addressed:
- Server authentication: the 'reproduce/software/config/LOCAL.conf' now
contains three new variables for this purpose. With them, you can give
your username and password, along with the authentication method of the
server. The comments on top of these three variables give a full
description of their usage.
- Verifying only the data in a file (ignoring the headers): The
'reproduce/analysis/config/INPUTS.conf' now accepts two new optional
variables for each input file using the FITS standard's DATASUM
convention: 'INPUT-%-fitsdatasum' and 'INPUT-%-fitshdu'. If the SHA256
isn't specified for a file, Maneage will use these to verify the
file. With the latter, you specify the HDU of the data you want to
verify and with the former you give the DATASUM value for that HDU. As
the name suggests, this is only valid for FITS files. If we find other
formats that support a similar behavior, we can add this feature for
those formats also. This is also thoroughly discussed in the comments of
'reproduce/analysis/config/INPUTS.conf'.
This commit was done with the help of Pedram Ashofte Ardakani, Sepideh
Eskandarlou and Mohammadreza Khellat.
SUMMARY: This is a software update to make Maneage more portable and up to
date. It does not involve any Maneage infrastructure changes. You should
just re-build your project to make sure the updated software haven't
removed/changed any of their features that you were using. In particular,
for Astrometry.net users, please see the respective note in P.S.2 below.
Until now, there have been many updates in the software that are built
within Maneage. The last software upadte was almost one year ago.
With this commit, the software in the P.S.1 have been updated. A
description of notable changes in the software environment is given in
P.S.2. This software environment has been tested on an Arch GNU/Linux,
Debian, CentOS-7 and macOS.
This commit is the merging of 24 individual commits by Raul Infante-Sainz
(who put a lot of energy on porting the software below for macOS, and
updating citations), Boudewijn Roukema (who helped with memory checking for
GCC, and testing on Debian and CentOS), Sepideh Eskandarlou (who tested the
environment) and myself.
Besides the updates in the core software, the followimg improvements have
also been implemented in this commit:
- When you run './project shell':
- A welcome message is printed that will remind the caller that they
have entered a new environment, it will print the location of 'HOME'
and the location of the shell startup file.
- The 'reproduce/software/shell/bashrc.sh' is loaded as a startup
file. This allows you to customize your interactive Maneage shell. A
default step has already been placed there that will put the git
branch name (in green) within the shell prompt (which was
purple). This greatly helps when dealing with directories under Git
version control. These settings won't bother with Maneage's default
operations: through environment variables we make sure that these
'./project shell' features will not slow-down the calls to the shell
within the non-interactive Make calls.
- The host's 'COLORTERM' is passed to the Maneage environment. It is
used by some programs that can have color outputs on the terminal.
- Updates to citations:
- Numpy and Scipy (as requested on their pages):
https://numpy.org/citing-numpy and https://scipy.org/citing-scipy
- Gnuastro: Added https://arxiv.org/abs/1909.11230 which describes major
updates to Gnuastro after 10 releases.
- When a software's paper is indexed in the SAO/NASA Astrophysics Data
System (ADS), Maneage now use the BibTeX entries provided by ADS. This
helps to give a unified format to most software, and more information
(like ADS+arXiv hyperlinks in the BibLaTeX compilation of the default
bibliography).
- We were able to build this version of Maneage on a Debian system from
2010 (+12 years ago!). Only three downgrades were necessary in the
"basic" software (not affecting the high-level science software!). A
description of the necessary downgrades for such old systems has been
added in 'README.md'.
P.S.1 List of updated software:
Basic software:
cURL 7.79.1 --> 7.84.0
Dash 0.5.11.5 --> 0.5.11-057cd65
File 5.41 --> 5.42
GNU AWK 5.1.0 --> 5.1.1
GNU Bash 5.1.8 --> 5.2-rc2
GNU Binutils 2.37 --> 2.39
GNU Compiler Collection (GCC) 11.2.1 --> 12.1.0
GNU Findutils 4.8.0 --> 4.9.0
GNU Gzip 1.11 --> 1.12
GNU Help2man 1.48.5 --> 1.49.2
GNU Integer Set Library (ISL) 0.18 --> 0.24
GNU Libtool 2.4.6 --> 2.4.7
GNU Nano 6.0 --> 6.4
GNU Readline 8.1.1 --> 8.2-rc2
GNU libiconv 0.16 --> 0.17
Git 2.36.0 --> 2.37.1
OpenSSL 3.0.0 --> 3.0.5
PatchELF 0.13 --> 0.15.0
Perl 5.34.0 --> 5.36.0
High-level software:
Astrometry.net 0.89 --> 0.91
CFITSIO 4.0.0 --> 4.1.0
CMake 3.21.4 --> 3.24.0
GNU Astronomy Utilities (Gnuastro) 0.16.1 --> 0.18
GPL Ghostscript 9.55.0 --> 9.56.1
HDF5 1.10.5 --> 1.13.1
Libjpeg 9d --> 9e
Libtiff 4.3.0 --> 4.4.0
OpenBLAS 0.3.18 --> 0.3.21
PLplot n/a --> 5.15.0
Python 3.10.0 --> 3.10.6
SCAMP 2.6.7 --> 2.10.0
SWarp 2.38.0 --> 2.41.5
Util-Linux 2.37.2 --> 2.38.1
Vim 8.2 --> 9.0
WCSLIB 7.7 --> 7.11
X.org packages (used by graphical software like Ghostscript and LaTeX):
Fontconfig 2.13.94 --> 2.14.0
LibX11 1.7.2 --> 1.8
LibXCB 1.14 --> 1.15
XCB-proto 1.14.1 --> 1.15
Xorg-proto 2021.5 --> 2022.1
Python modules:
Astropy 5.0 --> 5.1
GalSim 2.3.3 --> 2.3.5
P.S.2: Notable points regarding the software environment:
- Two new links from the host's low-level tools are now included in
Maneage's build environment:
- On GNU/Linux systems, the host's 'ldd' is linked inside the custom
environment. This belongs to the GNU C Library (which is not yet
installed in Maneage). But helps in checking the linking status of the
binaries on GNU/Linux systems.
- On macOS: the 'codesign' binary is included, which is used by GNU
Emacs on macOS to sign the built executable.
- GNU Bison has been moved in basic software (necessary for GNU Binutils).
- The Zip and Unzip programs have been moved as high-level software that
have to be manually requested when necessary. This is because they are
not used by any of the basic software anymore. They were just installed
as dependencies of GNU Tar to be close the other compression
programs. Also, in the past we would use the original tarballs, and some
(for example Numpy) were distributed in Zip format. However, by default,
we now use a custom Lzip tarball and don't need Zip or Unzip. This was
suggested by Zahra Sharbaf and Raul Infante-Sainz.
- Some minor edits in 'reproduce/software/shell/tarball-prepare.sh'. In
particular the 'awk' command was effectively just replacing a '_' with
'-', so it just uses a simple SED expression instead.
- Fixed bug 62700 (https://savannah.nongnu.org/bugs/index.php?62700) by
compiling 'xz' with a patched version of the xz source file
'src/liblzma/liblzma.map'.
- Astrometry.net doesn't depend on NetPBM any more. NetPBM (and its
dependencies) were causing many crashes on macOS and it also a very
strange build system that is hard to maintain. Astrometry.net uses it to
take images as input. However, it isn't necessary when you provide
Astrometry.net with a catalog. Therefore, Raul added some instructions
on how to run astrometry from your own custom X-Y catalog. These
instruction can be seen on top of the build rule of Astrometry.net in
'reproduce/software/make/high-level.mk'.
- h5py has been removed as a dependency of Astropy. It is an optional
dependency to write tables into HDF5 format. But since we couldn't get
it to build on macOS it has been removed. None of the current Maneage
users/developers also use this feature of Astropy!
- PLplot is added a new software, but not a default pre-requisite of SCAMP
(which can use it to generate figures), because there were many build
problems on macOS. Instructions have been added on top of SCAMP on how
to add PLplot as a dependency.
- With the aim of being able to install Plplot on macOS, we have wrote
several lines to fix header problems. However, we didn't succeed. In any
case we are leaving these lines in case they are useful in the future.
- The '-Wno-nullability-completeness' compiler flag (which is primarily
necessary for macOS) is now only added for macOS systems. It was causing
many warnings of un-recognized option in GNU/Linux systems.
- The 'mkswap' program of Util-Linux has been disabled because it caused
crashes on older kernels. Generally, its not necessary for a Maneage
project because it needs root permissions to run!
- LibXT (of the x.org software) has been added as a dependency of Cairo.
- ImageMagick and Lzip were using the host's C++ standard library! But on
GNU/Linux we build our own C++ Standard Library with GCC, so with this
commit, they properly link with Maneage's C++ standard library.
- ImageMagick on macOS couldn't properly link with Maneage's Ghostscript
library! This has been fixed using macOS's install_name_tool.
- Necessary RAM to build GCC on GNU/Linux systems changed to ~8GB, see
https://savannah.nongnu.org/task/?16244#comment12
- Pythran is no longer as prerequisite of Scipy. Until now, Pythran was a
prerequisite of Scipy. But we noticed that it is optional and was
causing problems on macOS.
- The URLs of some of the software have been updated in
'reproduce/software/config/urls.conf'. By default, these are all
commented, but they can be useful when searching for new versions or
when a project needs custom software that is not (yet) in Maneage.