| Age | Commit message (Collapse) | Author | Lines |
|---|
|
SUMMARY: This is a software update to make Maneage more portable and up to
date. It does not involve any Maneage infrastructure changes. You should
just re-build your project to make sure the updated software haven't
removed/changed any of their features that you were using. In particular,
for Astrometry.net users, please see the respective note in P.S.2 below.
Until now, there have been many updates in the software that are built
within Maneage. The last software upadte was almost one year ago.
With this commit, the software in the P.S.1 have been updated. A
description of notable changes in the software environment is given in
P.S.2. This software environment has been tested on an Arch GNU/Linux,
Debian, CentOS-7 and macOS.
This commit is the merging of 24 individual commits by Raul Infante-Sainz
(who put a lot of energy on porting the software below for macOS, and
updating citations), Boudewijn Roukema (who helped with memory checking for
GCC, and testing on Debian and CentOS), Sepideh Eskandarlou (who tested the
environment) and myself.
Besides the updates in the core software, the followimg improvements have
also been implemented in this commit:
- When you run './project shell':
- A welcome message is printed that will remind the caller that they
have entered a new environment, it will print the location of 'HOME'
and the location of the shell startup file.
- The 'reproduce/software/shell/bashrc.sh' is loaded as a startup
file. This allows you to customize your interactive Maneage shell. A
default step has already been placed there that will put the git
branch name (in green) within the shell prompt (which was
purple). This greatly helps when dealing with directories under Git
version control. These settings won't bother with Maneage's default
operations: through environment variables we make sure that these
'./project shell' features will not slow-down the calls to the shell
within the non-interactive Make calls.
- The host's 'COLORTERM' is passed to the Maneage environment. It is
used by some programs that can have color outputs on the terminal.
- Updates to citations:
- Numpy and Scipy (as requested on their pages):
https://numpy.org/citing-numpy and https://scipy.org/citing-scipy
- Gnuastro: Added https://arxiv.org/abs/1909.11230 which describes major
updates to Gnuastro after 10 releases.
- When a software's paper is indexed in the SAO/NASA Astrophysics Data
System (ADS), Maneage now use the BibTeX entries provided by ADS. This
helps to give a unified format to most software, and more information
(like ADS+arXiv hyperlinks in the BibLaTeX compilation of the default
bibliography).
- We were able to build this version of Maneage on a Debian system from
2010 (+12 years ago!). Only three downgrades were necessary in the
"basic" software (not affecting the high-level science software!). A
description of the necessary downgrades for such old systems has been
added in 'README.md'.
P.S.1 List of updated software:
Basic software:
cURL 7.79.1 --> 7.84.0
Dash 0.5.11.5 --> 0.5.11-057cd65
File 5.41 --> 5.42
GNU AWK 5.1.0 --> 5.1.1
GNU Bash 5.1.8 --> 5.2-rc2
GNU Binutils 2.37 --> 2.39
GNU Compiler Collection (GCC) 11.2.1 --> 12.1.0
GNU Findutils 4.8.0 --> 4.9.0
GNU Gzip 1.11 --> 1.12
GNU Help2man 1.48.5 --> 1.49.2
GNU Integer Set Library (ISL) 0.18 --> 0.24
GNU Libtool 2.4.6 --> 2.4.7
GNU Nano 6.0 --> 6.4
GNU Readline 8.1.1 --> 8.2-rc2
GNU libiconv 0.16 --> 0.17
Git 2.36.0 --> 2.37.1
OpenSSL 3.0.0 --> 3.0.5
PatchELF 0.13 --> 0.15.0
Perl 5.34.0 --> 5.36.0
High-level software:
Astrometry.net 0.89 --> 0.91
CFITSIO 4.0.0 --> 4.1.0
CMake 3.21.4 --> 3.24.0
GNU Astronomy Utilities (Gnuastro) 0.16.1 --> 0.18
GPL Ghostscript 9.55.0 --> 9.56.1
HDF5 1.10.5 --> 1.13.1
Libjpeg 9d --> 9e
Libtiff 4.3.0 --> 4.4.0
OpenBLAS 0.3.18 --> 0.3.21
PLplot n/a --> 5.15.0
Python 3.10.0 --> 3.10.6
SCAMP 2.6.7 --> 2.10.0
SWarp 2.38.0 --> 2.41.5
Util-Linux 2.37.2 --> 2.38.1
Vim 8.2 --> 9.0
WCSLIB 7.7 --> 7.11
X.org packages (used by graphical software like Ghostscript and LaTeX):
Fontconfig 2.13.94 --> 2.14.0
LibX11 1.7.2 --> 1.8
LibXCB 1.14 --> 1.15
XCB-proto 1.14.1 --> 1.15
Xorg-proto 2021.5 --> 2022.1
Python modules:
Astropy 5.0 --> 5.1
GalSim 2.3.3 --> 2.3.5
P.S.2: Notable points regarding the software environment:
- Two new links from the host's low-level tools are now included in
Maneage's build environment:
- On GNU/Linux systems, the host's 'ldd' is linked inside the custom
environment. This belongs to the GNU C Library (which is not yet
installed in Maneage). But helps in checking the linking status of the
binaries on GNU/Linux systems.
- On macOS: the 'codesign' binary is included, which is used by GNU
Emacs on macOS to sign the built executable.
- GNU Bison has been moved in basic software (necessary for GNU Binutils).
- The Zip and Unzip programs have been moved as high-level software that
have to be manually requested when necessary. This is because they are
not used by any of the basic software anymore. They were just installed
as dependencies of GNU Tar to be close the other compression
programs. Also, in the past we would use the original tarballs, and some
(for example Numpy) were distributed in Zip format. However, by default,
we now use a custom Lzip tarball and don't need Zip or Unzip. This was
suggested by Zahra Sharbaf and Raul Infante-Sainz.
- Some minor edits in 'reproduce/software/shell/tarball-prepare.sh'. In
particular the 'awk' command was effectively just replacing a '_' with
'-', so it just uses a simple SED expression instead.
- Fixed bug 62700 (https://savannah.nongnu.org/bugs/index.php?62700) by
compiling 'xz' with a patched version of the xz source file
'src/liblzma/liblzma.map'.
- Astrometry.net doesn't depend on NetPBM any more. NetPBM (and its
dependencies) were causing many crashes on macOS and it also a very
strange build system that is hard to maintain. Astrometry.net uses it to
take images as input. However, it isn't necessary when you provide
Astrometry.net with a catalog. Therefore, Raul added some instructions
on how to run astrometry from your own custom X-Y catalog. These
instruction can be seen on top of the build rule of Astrometry.net in
'reproduce/software/make/high-level.mk'.
- h5py has been removed as a dependency of Astropy. It is an optional
dependency to write tables into HDF5 format. But since we couldn't get
it to build on macOS it has been removed. None of the current Maneage
users/developers also use this feature of Astropy!
- PLplot is added a new software, but not a default pre-requisite of SCAMP
(which can use it to generate figures), because there were many build
problems on macOS. Instructions have been added on top of SCAMP on how
to add PLplot as a dependency.
- With the aim of being able to install Plplot on macOS, we have wrote
several lines to fix header problems. However, we didn't succeed. In any
case we are leaving these lines in case they are useful in the future.
- The '-Wno-nullability-completeness' compiler flag (which is primarily
necessary for macOS) is now only added for macOS systems. It was causing
many warnings of un-recognized option in GNU/Linux systems.
- The 'mkswap' program of Util-Linux has been disabled because it caused
crashes on older kernels. Generally, its not necessary for a Maneage
project because it needs root permissions to run!
- LibXT (of the x.org software) has been added as a dependency of Cairo.
- ImageMagick and Lzip were using the host's C++ standard library! But on
GNU/Linux we build our own C++ Standard Library with GCC, so with this
commit, they properly link with Maneage's C++ standard library.
- ImageMagick on macOS couldn't properly link with Maneage's Ghostscript
library! This has been fixed using macOS's install_name_tool.
- Necessary RAM to build GCC on GNU/Linux systems changed to ~8GB, see
https://savannah.nongnu.org/task/?16244#comment12
- Pythran is no longer as prerequisite of Scipy. Until now, Pythran was a
prerequisite of Scipy. But we noticed that it is optional and was
causing problems on macOS.
- The URLs of some of the software have been updated in
'reproduce/software/config/urls.conf'. By default, these are all
commented, but they can be useful when searching for new versions or
when a project needs custom software that is not (yet) in Maneage.
|
|
This commit primarily affects the configuration step of Maneage'd projects,
and in particular, updated versions of the many of the software (see
P.S.). So it shouldn't affect your high-level analysis other than the
version bumps of the software you use (and the software's possibly
improve/changed behavior).
The following software (and thus their dependencies) couldn't be updated as
described below:
- Cryptography: isn't building because it depends on a new
setuptools-rust package that has problems
(https://savannah.nongnu.org/bugs/index.php?61731), so it has been
commented in 'versions.conf'.
- SecretStorage: because it depends on Cryptography.
- Keyring: because it depends on SecretStorage.
- Astroquery: because it depends on Keyring.
This is a "squashed" commit after rebasing a development branch of 60
commits corresponding to a roughly two-month time interval. The following
people contributed to this branch.
- Boudewijn Roukema added all the R software infrastructure and the R
packages, as well as greatly helping in fixing many bugs during the
update.
- Raul Infante-Sainz helped in testing and debugging the build.
- Pedram Ashofteh Ardakani found and fixed a bug.
- Zahra Sharbaf helped in testing and found several bugs.
Below a description of the most noteworthy points is given.
- Software tarballs: all updated software now have a unified format
tarball (ustar; if not possible, pax) and unified compression (Lzip) in
Maneage's software repository in Zenodo
(https://doi.org/10.5281/zenodo.3883409). For more on this See
https://savannah.nongnu.org/task/?15699 . This won't affect any extra
software you would like to add; you can use any format recognized by
GNU Tar, and all common compression algorithms. This new requirement is
only for software that get merged to the core Maneage branch.
- Metastore (and thus libbsd and libmd) moved to highlevel: Metastore
(and the packages it depends on) is a high-level product that is only
relevant during the project development (like Emacs!): when the user
wants the file meta data (like dates) to be unchanged after checking
out branches. So it should be considered a high-level software, not
basic. Metastore also usually causes many more headaches and error
messages, so personally, I have stopped using it! Instead I simply
merge my branches in a separate clone, then pull the merge commit: in
this way, the files of my project aren't re-written during the checkout
phase and therefore their dates are untouched (which can conflict with
Make's dates on configuration files).
- The un-official cloned version of Flex (2.6.4-91 until this commit) was
causing problems in the building of Netpbm, so with this commit, it has
been moved back to version 2.6.4.
- Netpbm's official page had version 10.73.38 as the latest stable
tarball that was just released in late 2021. But I couldn't find our
previously-used version 10.86.99 anywhere (to see when it was released
and why we used it! Its at last more than one year old!). So the
official stable version is being used now.
- Improved instructions in 'README.md' for building software environment
in a Docker container (while having project source and output data
products on the local system; including the usage of the host's
'/dev/shm' to speed up temporary operations).
- Until now, the convention in Maneage was to put eight SPACE characters
before the comment lines within recipes. This was done because by
default GNU Emacs (also many other editors) show a TAB as eight
characters. However, in other text editors, online browsers, or even
the Git diff, a TAB can correspond to a different number of
characters. In such cases, the Maneage recipes wouldn't look too
interesting (the comments and the recipe commands would show a
different indentation!).
With this commit, all the comment lines in the Makefiles within the
core Maneage branch have a hash ('#') as their first character and a
TAB as the second. This allows the comment lines in recipes to have the
same indentation as code; making the code much more easier to read in a
general scenario including a 'git diff' (editor agnostic!).
P.S. List of updated software with their old and new versions
- Software with no version update are not mentioned.
- The old version of newly added software are shown with '--'.
Name (Basic) Old version New version
------------ ----------- -----------
Bzip2 1.0.6 1.0.8
CURL 7.71.1 7.79.1
Dash 0.5.10.2 0.5.11.5
File 5.39 5.41
Flock 0.2.3 0.4.0
GNU Bash 5.0.18 5.1.8
GNU Binutils 2.35 2.37
GNU Coreutils 8.32 9.0
GNU GCC 10.2.0 11.2.0
GNU M4 1.4.18 1.4.19
GNU Readline 8.0 8.1.1
GNU Tar 1.32 1.34
GNU Texinfo 6.7 6.8
GNU diffutils 3.7 3.8
GNU findutils 4.7.0 4.8.0
GNU gmp 6.2.0 6.2.1
GNU grep 3.4 3.7
GNU gzip 1.10 1.11
GNU libunistring 0.9.10 1.0
GNU mpc 1.1.0 1.2.1
GNU mpfr 4.0.2 4.1.0
GNU nano 5.2 6.0
GNU ncurses 6.2 6.3
GNU wget 1.20.3 1.21.2
Git 2.28.0 2.34.0
Less 563 590
Libxml2 2.9.9 2.9.12
Lzip 1.22-rc2 1.22
OpenSLL 1.1.1a 3.0.0
Patchelf 0.10 0.13
Perl 5.32.0 5.34.0
Podlators -- 4.14
Name (Highlevel) Old version New version
---------------- ----------- -----------
Apachelog4cxx 0.10.0-603 0.12.1
Astrometry.net 0.80 0.85
Boost 1.73.0 1.77.0
CFITSIO 3.48 4.0.0
Cmake 3.18.1 3.21.4
Eigen 3.3.7 3.4.0
Expat 2.2.9 2.4.1
FFTW 3.3.8 3.3.10
Flex 2.6.4-91 2.6.4
Fontconfig 2.13.1 2.13.94
Freetype 2.10.2 2.11.0
GNU Astronomy Utilities 0.12 0.16.1-e0f1
GNU Autoconf 2.69.200-babc 2.71
GNU Automake 1.16.2 1.16.5
GNU Bison 3.7 3.8.2
GNU Emacs 27.1 27.2
GNU GDB 9.2 11.1
GNU GSL 2.6 2.7
GNU Help2man 1.47.11 1.48.5
Ghostscript 9.52 9.55.0
ICU -- 70.1
ImageMagick 7.0.8-67 7.1.0-13
Libbsd 0.10.0 0.11.3
Libffi 3.2.1 3.4.2
Libgit2 1.0.1 1.3.0
Libidn 1.36 1.38
Libjpeg 9b 9d
Libmd -- 1.0.4
Libtiff 4.0.10 4.3.0
Libx11 1.6.9 1.7.2
Libxt 1.2.0 1.2.1
Netpbm 10.86.99 10.73.38
OpenBLAS 0.3.10 0.3.18
OpenMPI 4.0.4 4.1.1
Pixman 0.38.0 0.40.0
Python 3.8.5 3.10.0
R 4.0.2 4.1.2
SWIG 3.0.12 4.0.2
Util-linux 2.35 2.37.2
Util-macros 1.19.2 1.19.3
Valgrind 3.15.0 3.18.1
WCSLIB 7.3 7.7
Xcb-proto 1.14 1.14.1
Xorgproto 2020.1 2021.5
Name (Python) Old version New version
------------- ----------- -----------
Astropy 4.0 5.0
Beautifulsoup4 4.7.1 4.10.0
Beniget -- 0.4.1
Cffi 1.12.2 1.15.0
Cryptography 2.6.1 36.0.1
Cycler 0.10.0 0.11.0+}
Cython 0.29.21 0.29.24
Esutil 0.6.4 0.6.9
Extension-helpers -- 0.1
Galsim 2.2.1 2.3.3
Gast -- 0.5.3
Jinja2 -- 3.0.3
MPI4py 3.0.3 3.1.3
Markupsafe -- 2.0.1
Numpy 1.19.1 1.21.3
Packaging -- 21.3
Pillow -- 8.4.0
Ply -- 3.11
Pyerfa -- 2.0.0.1
Pyparsing 2.3.1 3.0.4
Pythran -- 0.11.0
Scipy 1.5.2 1.7.3
Setuptools 41.6.0 58.3.0
Six 1.12.0 1.16.0
Uncertainties 3.1.2 3.1.6
Wheel -- 0.37.0
Name (R) Old version New version
-------- ----------- -----------
Cli -- 2.5.0
Colorspace -- 2.0-1
Cowplot -- 1.1.1
Crayon -- 1.4.1
Digest -- 0.6.27
Ellipsis -- 0.3.2
Fansi -- 0.5.0
Farver -- 2.1.0
Ggplot2 -- 3.3.4
Glue -- 1.4.2
GridExtra -- 2.3
Gtable -- 0.3.0
Isoband -- 0.2.4
Labeling -- 0.4.2
Lifecycle -- 1.0.0
Magrittr -- 2.0.1
MASS -- 7.3-54
Mgcv -- 1.8-36
Munsell -- 0.5.0
Pillar -- 1.6.1
R-Pkgconfig -- 2.0.3
R6 -- 2.5.0
RColorBrewer -- 1.1-2
Rlang -- 0.4.11
Scales -- 1.1.1
Tibble -- 3.1.2
Utf8 -- 1.2.1
Vctrs -- 0.3.8
ViridisLite -- 0.4.0
Withr -- 2.4.2
|
|
Having entered 2021, it was necessary to update the copyright years at the
top of the source files. We recommend that you do this for all your
project-specific source files also.
|
|
One of the main reasons to building Maneage is to properly
acknowledge/attribute the authors of software in research. So we have
adopted a standard of never referring to the GNU-based operating systems
running the Linux kernel simply as "Linux", we avoid terms like "Open
Sourse" and use Free Software instead (in the same spirit).
With this commit, a few instances of the cases above have been corrected,
they had slipped through our fingers when we initially imported them into
the project. In the special case of the "Journal for Open Source Software",
we simply replaced it with its abbreviation (JOSS). This was done because
in effect we were generally using journal name abbreviations in almost all
the citations already. To avoid any inconsistancies, the names of the three
other journals that weren't abbreviated are also abbreviated.
|
|
Until now, two of the software BibTeX sources (Matplolib and Sympy) had an
"abstract" entry that was long, not similar to the rest, and not relevant
in this context, so they are removed with this commit.
|
|
In time, some of the copyright license description had been mistakenly
shortened to two paragraphs instead of the original three that is
recommended in the GPL. With this commit, they are corrected to be exactly
in the same three paragraph format suggested by GPL.
The following files also didn't have a copyright notice, so one was added
for them:
reproduce/software/make/README.md
reproduce/software/bibtex/healpix.tex
reproduce/analysis/config/delete-me-num.conf
reproduce/analysis/config/verify-outputs.conf
|
|
MissFITS is package for manipulating FITS files.
I added it as my first commit to the project for educational
purposes.
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
Python's `lmfit' module and all its major dependencies (`asteval',
`corner', `emcee' and `uncertainties') have been included in the template.
While doing this I noticed that if the tarballs are the last prerequisite
of each software building rule, then when building in parallel, the
template will immediately start building packages as soon as the first one
is downloaded. Not like the current way that it will attempt to download
several, then start building. For now, this has been implemented in the
Python build rules for all the modules and we'll later do the same for the
other programs and libraries. This also motivated a simplification of the
`pybuild' function: it now internally looks into the prerequisites and
selects the tarball from the prerequisite that is in the tarballs
directory.
This isn't a problem for the build, but I just don't understand why Python
can't recognize the version of `emcee', Python reads the version of `emcee'
as `0.0.0'! But it doesn't cause any crash in the build, so for now its
fine.
|
|
The tarball of HEALPix includes multiple languages and doesn't include the
ready-to-run GNU Build System by default, we actually have to build the
`./configure' script for the C/C++ libraries. So it was necessary to also
include GNU Autoconf and GNU Automake as prerequisites of HEALPix.
However, the official GNU Autoconf tarball (dating from 2012) doesn't build
on modern systems, so I just cloned it from its source and bootstrapped it
and built its modern tarball which we are using here.
|
|
The following software are added with this commit: eigency, esutil, flake8,
future, galsim, lsstdesccoord, pybind11 and pyflakes.
|
|
Until now, some software were using the depreciated ADS URL
(`http://adsabs.harvard.edu....') and some were using the modern one
(`http://ui.adsabs.harvard.edu').
With this commit, to be consistent they all link to the modern URL.
|
|
Until now the SExtractor BibTeX entry has the first-name and family name in
the same field, while the family-name should be in curly braces. Also, it
was missing an ADS URL. So the default ADS generated BibTeX entry for
SExtractor is now used instead of the old one.
|
|
While reviewing Prasenjit's commits, I noticed that we had forgot to add
the citation for TIDES, also to make things clear, the program/library
build rules are now sorted alphabetically.
Finally, I noticed that after building the TiKZ PDF figures, it is crashing
(like on Prasenjit's computer). After looking around, I noticed its because
we were setting the of the `TEXINPUTS' environment variable to be the
installed TeX Live directory (which was ultimately redundant because by
default TeX will look into where it was installed). The important thing is
just that we remove any possible value the host system has, not to set new
directories.
|
|
Until this commit, `imfit' paper was not included into the
acknowledgements.
With this commit, a bib tex file for `imfit' has been added. So, now it
is properly cited into the final paper.
|
|
The BibTeX data for `sip_tpv' didn't have a copyright notice, so one was
added for it.
|
|
Until now, we were not citing the paper of `sip_tpv' package.
With this commit, we have been fix this issue.
|
|
With this commit, we add `sympy' Python package into the template. This
is a package to do symbolic mathematics.
The motivation is that it is a prerequisite of `sip_tpv' Python package,
which is useful to convert SIP distorsion coefficients into PV
coefficients (in the context of astronomical images). However, the
availability of `sympy' in the template will be useful for anyone
interested in this package.
|
|
With this commit, we add SCAMP into the project. This program is part of
the Astromatic software and the main purpose is to make the fine
astrometry and the distorsion correction astronomical images.
In principle, SCAMP needs ATLAS library to be able to work (that is what
the documentation says). However, we had some problems with ATLAS
libraries on Mac OS system, and due to that, we set the option
`--enable-openblas' in the configure step. By doing that, we are be able
to install SCAMP with no problems.
One dependency of SCAMP is `cdsclient', it has been also added with this
commit. The `cdsclient' package is a set of C and shell routines which can
be built on Unix stations or PCs running Linux, which once compiled
allow to query some databases located at CDS or on mirrors over the
network.
|
|
With this commit, we add SExtractor to be installed into the project. It
is a widely used program to detect object and build catalogues from
astronomical images.
We had some problems when installing it because it could not link with
some ATLAS libraries. But, since we have OpenBLAS installed, we can use
it to override the problem with ATLAS.
|
|
Until now, there were erros in the citation of Astrometry-net and Scamp
papers.
With this commit, we fix these problems. The Swarp bibtex has also been
modify to follow the stetic of the citation style we have right now in
the project.
We also added the `dependency-bib.tex' as a prerequisite of `paper.bbl'.
|
|
SWarp resamples and co-adds together FITS images using any arbitrary
astrometric projection defined in the WCS standard. It is a very useful
program for astronomy and that is why we added it to the pipeline.
With this commit, we also cite the paper for Astrometry-net that until
now was missing.
|
|
With this commit, we include Astrometry-net software and one remaining
dependency of it (Swig). In principle it should work as far as we tested
all of this software in GNU/Linux, but we need to test also in Mac OS.
As a result, this commit is just for setting all the rules and
dependencies, but more tests are necessaries to ensure it works
properly, specially in Mac OS systems.
|
|
Until now, the software building and analysis steps of the pipeline were
intertwined. However, these steps (of how to build a software, and how to
use it) are logically completely independent.
Therefore with this commit, the pipeline now has a new architecture
(particularly in the `reproduce' directory) to emphasize this distinction:
The `reproduce' directory now has the two `software' and `analysis'
subdirectories and the respective parts of the previous architecture have
been broken up between these two based on their function. There is also no
more `src' directory. The `config' directory for software and analysis is
now mixed with the language-specific directories.
Also, some of the software versions were also updated after some checks
with their webpages.
This new architecture will allow much more focused work on each part of the
pipeline (to install the software and to run them for an analysis).
|
