| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, there was no check on the integrity of the contents of the
downloaded/copied software tarballs, we only relied on the tarball
name. This could be bad for reproducibility and security, for example on
one server the name of a tarball may be the same but with different
content.
With this commit, the SHA512 checksums of all the software are stored in
the newly created `checksums.mk' (similar to how the versions are stored in
the `versions.mk'). The resulting variable is then defined for each
software and after downloading/copying the file we check to see if the new
tarball has the same checksum as the stored value. If it doesn't the script
will crash with an error, informing the user of the problem.
The only limitation now is a bootstrapping problem: if the host system
doesn't already an `sha512sum' executable, we will not do any checksum
verification until we install our `sha512sum' (as part of GNU
Coreutils). All the tarballs downloaded after GNU Coreutils are built will
have their checksums validated. By default almost all GNU/Linux systems
will have a usable `sha512sum' (its part of GNU Coreutils after all for a
long time: from the Coreutils Changelog file atleast since 2013).
This completes task #15347.
|
|
Until now, like all other software, PatchELF would install with dynamic
linking. However, PatchELF links with `libstdc++' so on one system, I
noticed that PatchELF gives a segmentation fault and corrupts `libstdc++'
while correcting its RPATH (after installing GCC). The solution is to build
PatchELF statically.
With this commit, we force PatchELF to be built statically (it only
installs on GNU/Linux systems anyway, so there is no problem with static
linking on macOS). This solved the problem on that system.
While looking at its documentation, I also noticed that a new version of
PatchELF has been released after almost three years, so it has been updated
in the template also.
This fixes bug #56673.
|
|
Some bugs have been fixed in the new version of WCSLIB, so it has been
updated in the template.
|
|
More than two releases and bug fixes have been made to libgit2. So we are
now using a more recent version in the template.
|
|
While testing on a system with no Texinfo, we noticed that M4 depends on
Texinfo. To fix this problem, with this commit, it is now included in the
pipeline.
While doing a clean build, a few minor issues were also found and corrected
in the other rules.
|
|
TIDES is an ODE integrator with multiple-precision arithmetic.
|
|
Prasenjit Saha reported that while building on a macOS, Git complained
about not finding `libiconv' functions. Therefore with this commit,
`libiconv' is also built as a dependency of Git.
Also, since the build of Binutils (and the subsequent GCC) doesn't depend
on the exact versions of the other basic software, they are set as
existance-only prerequsites (ignoring their timestamps). This is done to
avoid re-building when it is not needed.
|
|
The recent work by Raul to import Imfit is now merged into the master
branch. Only a few minor conflicts came up due to the update with CFITSIO
(that Raul also done in parallel on this branch).
|
|
Until now, TeXlive would only attempt to download/update LaTeX packages,
but after some time, it will be necessary for it to update itself. When
this happens, it complains about running it with `update --self'.
To fix this problem, before installing/updating the LaTeX packages, we will
first update tlmgr in any case.
Also, we weren't removing the TeXlive package version file before appending
the values to it. So it was just repeating the packages every time it was
updated. With this commit, it is being removed on every new build.
Finally, the version of Git was updated to its most recent version.
|
|
`cfitsio' was recently updated and it cames up with a problem when
trying to build shared libraries (on Mac OS system laptop). This is
because the `configure' script for building `cfitsio' include by default
`rpath'.
With this commit, this issue is fixed by modifying the installation of
`cfitsio'. Using `sed', each ocurrence of `@rpath' in `configure' is
replaced by `$(ildir)'. Once it has been done, the installation of
`cfitsio' goes as normal.
In this commit, the version of the `cfitsio' has been also updated to
the most recent one.
|
|
Until now, CFITSIO had several peculiar features which needed special
attention: 1) Its tarball didn't have a `.' in the version name and would
sometimes have extra `0's. 2) When unpacked it was just `cfitsio'. 3) It
hard coded the directory to search for cURL library. All of these forced us
to do a lot of extra steps for CFITSIO.
So I got in touch with them and fortunately they were very open to
correcting these issues and with version 3.47 all such issues have been
addressed in the tarball. Thanks to these changes, with this commit,
CFITSIO's build rule is now very similar to many other programs.
I also noticed that it has an optional `--with-bz2' feature to enable
unpacking `.fits.bz2' files. So this feature is also added.
|
|
With this commit, `imfit' has been included into the project. As described
in the official webpage (http://www.mpe.mpg.de/~erwin/code/imfit/):
Imfit is a program for fitting astronomical images -- especially images
of galaxies, though it can in principle be used for fitting other
sources.
This program uses the `scons' program to build itself (not the traditional
GNU `configure' and `make' system). I have tested it by using an example
that can be found in the `examples' directory of the decompressed
tarball. It works fine without any error on my Mac OS system.
|
|
With this commit, SCons has been included into the project. As in the
official webpage of this program says:
SCons is an Open Source software construction tool—that is, a
next-generation build tool. Think of SCons as an improved,
cross-platform substitute for the classic Make utility with integrated
functionality similar to autoconf/automake and compiler caches such as
ccache. In short, SCons is an easier, more reliable and faster way to
build software.
The motivation for including this software into the project is because
there are some software that use this system for the installation. In
particular, the necessity of installing it comes from Imfit. This is an
astronomical software that uses this system so that is why it has been
included into the project. In principle it is simple, and I have tested
it on Mac OS sytem without any problem.
|
|
With this commit, PyYAML Python package has been added into the project.
It is widely used in the Python community and the goal is to have human
readable configuration files. As in the web page
(https://pypi.org/project/PyYAML/) says:
YAML is a data serialization format designed for human readability and
interaction with scripting languages. PyYAML is a YAML parser and
emitter for Python.
|
|
With this commit, LibYAML library has been added. This library is a
prerequisite of the Python package PyYAML, which is widely used in the
Python community when configuration scripts are needed.
|
|
With this commit, ImageMagick software has been added into the project.
This software is useful to deal and treat images from the command line.
Since it is widely used and a lot of other programs rely on it, it is
worth to have it into the project.
|
|
Until now the software names and versions in
`reproduce/software/config/installation/versions.mk' were placed such that
all the versions were placed under each other. This had a visual effect,
but since many of the software have shorter names, it could sometimes cause
visual confusion on which number belongs to which software.
Since we recently removed a similar situation with the backslashes, with
this commit, there is only a single space between the variable name, `='
and the version number.
|
|
GCC 9.1 was just released and includes many new optimization features that
can benefit projects in this template also.
Also, the build directory of GCC has been moved inside the unpacked
directory (similar to all the other programs). As a result, while GCC is
being built, the `.build/software/build-tmp' directory will only have one
directory for GCC. Until now, there was one `gcc-build' directory and one
unpacked tarball.
|
|
Until this commit, we were using `libpng' version 1.4.22 (out dated).
The reason is that with `libpng` versions > 1.5, there was some
problems with `netpbm'. But we have update `netpbm' version and now
there is no conflict betweem them.
With this commit, we increase the version of `libpng' to have the most
recent one.
|
|
Until now, we installed `netpbm' by setting the default paths to
libraries and headers. That worked fine for GNU/Linux systems, however,
in Mac OS systems we have to explicity say to `netpbm' questions what
are the paths to the libraries and headers. We found this problem
because we tried to run `solve-field' to obtain the astrometry of some
images, and then it goes to use `pnmfile', a program that failed because
it was not properly linked with `libnetpmb'.
With this commit, we fix this problem by answering properly the
questions in the configure step of `netpbm' and setting explicity where
we want to place the libraries and headers. With this commit, we also
update `netpbm' version. To do that, we have to download the last
version, for which it doesn't exist a tarball. So, we built this tarball
and put it in the Mohammad's webpage.
|
|
There wasn't any conflict in this merge.
|
|
With this commit, we add `sip_tpv' Python package into the template.
This is a small package to convert SIP distorsion coefficients into PV
distorsion coefficients, and the other way around.
This package is useful in a astronomical context, specially when `swarp'
is going to be used in order to resample images. The reason is that
`swarp' only can understand PV distorsion coefficients.
|
|
With this commit, we add `mpmath' Python package into the template. This
package is a prerequisite of `sympy', a package to do symbolic
mathematics.
The motivation of adding this package is because it is a dependency of
`sympy', which is more widely used into the Python science community.
|
|
Until now we were using the official tarball of GNU Bash. However, Bash is
distributed using patches, not a public version controlled history. So to
implement newer features of Bash, its necessary to apply those patches and
make the tarball ourselves.
With this commit, we have done just that: we used the 7 patches that have
been released since version 5.0.0 and made a tarball to use for this
template. The instructions on how to make the patched tarball are also
given above the Bash build rule.
|
|
Until now, to specify which high-level software you want the project to
contain, it was necessary to go into the `high-level.mk' Makefile that is
complicated and can create bugs.
With this commit, a new `reproduce/software/config/installation/TARGETS.mk'
file has been created that is easily/cleanly in charge of documenting the
final high-level software that must be built for the project.
Also, until now, FFTW was set as a dependency of Numpy while we couldn't
actually get Numpy to use it! It was just there for future reference and to
justify its build rule. But now that many software won't be built and there
is no problem with having rules even though a project might not use them,
it has been removed.
|
|
With this commit, we add SCAMP into the project. This program is part of
the Astromatic software and the main purpose is to make the fine
astrometry and the distorsion correction astronomical images.
In principle, SCAMP needs ATLAS library to be able to work (that is what
the documentation says). However, we had some problems with ATLAS
libraries on Mac OS system, and due to that, we set the option
`--enable-openblas' in the configure step. By doing that, we are be able
to install SCAMP with no problems.
One dependency of SCAMP is `cdsclient', it has been also added with this
commit. The `cdsclient' package is a set of C and shell routines which can
be built on Unix stations or PCs running Linux, which once compiled
allow to query some databases located at CDS or on mirrors over the
network.
|
|
With this commit, we add SExtractor to be installed into the project. It
is a widely used program to detect object and build catalogues from
astronomical images.
We had some problems when installing it because it could not link with
some ATLAS libraries. But, since we have OpenBLAS installed, we can use
it to override the problem with ATLAS.
|
|
SWarp resamples and co-adds together FITS images using any arbitrary
astrometric projection defined in the WCS standard. It is a very useful
program for astronomy and that is why we added it to the pipeline.
With this commit, we also cite the paper for Astrometry-net that until
now was missing.
|
|
With this commit, we include Astrometry-net software and one remaining
dependency of it (Swig). In principle it should work as far as we tested
all of this software in GNU/Linux, but we need to test also in Mac OS.
As a result, this commit is just for setting all the rules and
dependencies, but more tests are necessaries to ensure it works
properly, specially in Mac OS systems.
|
|
With this commit, the installation of `netpbm' has been included. This
software has a crazy dialogue installation that we override by giving to
it the answers (they differs on GNU/Linux and Mac OS systems). `libxml2'
also has been built (a remaining dependency of `netpbm').
In this commit, `libpng' version has been downgraded because it is
required by `netpbm'.
The motivation of installing `netpbm' is that it is a dependency of
Astrometry-net software.
|
|
In this commit we add Cairo library. The motivation of including this
library is because it is a dependency of Astrometry-net (which is a
common software used in astronomy projects). In principle it will not
be build by default by de pipeline, but we will keep the rule just in
case it is needed.
Pixman is another library which is a prerequisite of Cairo, it is also
built with this commit.
|
|
Until now, the software building and analysis steps of the pipeline were
intertwined. However, these steps (of how to build a software, and how to
use it) are logically completely independent.
Therefore with this commit, the pipeline now has a new architecture
(particularly in the `reproduce' directory) to emphasize this distinction:
The `reproduce' directory now has the two `software' and `analysis'
subdirectories and the respective parts of the previous architecture have
been broken up between these two based on their function. There is also no
more `src' directory. The `config' directory for software and analysis is
now mixed with the language-specific directories.
Also, some of the software versions were also updated after some checks
with their webpages.
This new architecture will allow much more focused work on each part of the
pipeline (to install the software and to run them for an analysis).
|
