aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software
AgeCommit message (Collapse)AuthorLines
2019-08-01Typo in previous commit correctedMohammad Akhlaghi-1/+0
After making the previous commit, I noticed an extra line (redundantly defining a wrong `BASH_ENV') that should have been deleted. It has been corrected.
2019-08-01Bash startup script for every recipeMohammad Akhlaghi-0/+54
Until now the only way to define the environment of the Make recipes was through the exported Make variables (mostly in `initialize.mk' for the analysis steps for example). However, there is only so much you can do with environment variables! In some situations you want slightly more complicated environment control, like setting an alias or running of scripts (things that are commonly done in the `~/.bashrc' file of users to configure their interactive, non-login shells). With this commit, a `reproduce/software/bash/bashrc.sh' has been defined for this job (which is currently empty!). Every major Make step of the project adds this file as the `BASH_ENV' environment variable, so the shell that is created to execute a recipe first executes this file, then the recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment variable that enables users to limit their envirnoment setup based on the condition it is being setup (in particular in the early phase of `basic.mk', where the user can't make any assumption about the programs and has to write a portable shell script).
2019-07-31GCC: libstdc++ has a link to libiconvMohammad Akhlaghi-0/+1
After adding the libiconv library to the template, the C++ library uses three of its functions (`libiconv', `libiconv_open' and `libiconv_close'). However, it doesn't explicity link with it inside its shared library! I tried by exporting `LIBS=-liconv' before the GCC configure script but it crashed as soon as it went on to the second GCC building stage (because this environment variable was no longer present there). I also tried adding the C++ configure option of `--enable-cstdio' to the GCC configure options (so it doesn't use iconv according to the manual), but it made no change. With this commit, as a brute-force solution, `patchelf --add-needed' is run on the installed `libstdc++.so', so `libiconv.so' is explicitly included inside the `libstdc++' shared library. This bug was found by Roberto Baena Galle while trying to load Matplotlib (which needed the C++ library). This fixes bug #56702.
2019-07-29Checking software tarball checksums before building softwareMohammad Akhlaghi-245/+472
Until now, there was no check on the integrity of the contents of the downloaded/copied software tarballs, we only relied on the tarball name. This could be bad for reproducibility and security, for example on one server the name of a tarball may be the same but with different content. With this commit, the SHA512 checksums of all the software are stored in the newly created `checksums.mk' (similar to how the versions are stored in the `versions.mk'). The resulting variable is then defined for each software and after downloading/copying the file we check to see if the new tarball has the same checksum as the stored value. If it doesn't the script will crash with an error, informing the user of the problem. The only limitation now is a bootstrapping problem: if the host system doesn't already an `sha512sum' executable, we will not do any checksum verification until we install our `sha512sum' (as part of GNU Coreutils). All the tarballs downloaded after GNU Coreutils are built will have their checksums validated. By default almost all GNU/Linux systems will have a usable `sha512sum' (its part of GNU Coreutils after all for a long time: from the Coreutils Changelog file atleast since 2013). This completes task #15347.
2019-07-29Tip added at initial configuration notice on how to see progressMohammad Akhlaghi-0/+10
The configuration step (building all the ncessary software) can take some time. It is natual for the user to want to see how the build is going (which software is being built at every moment). So far, we have only put a "Inspecting status" section in `README-hacking.md' that describes a solution, but some early users may not have read it yet. With this commit a short tip was added in the initial installation notice to inform the user of this very useful command.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-14/+1249
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-07-27DYLD_LIBRARY_PATH also fixed for macOS systemsMohammad Akhlaghi-16/+18
Until now we were only setting the `LD_LIBRARY_PATH' environment variable for GNU/Linux systems. But macOS systems use the `DYLD_LIBRARY_PATH'. With this commit, for better control over the environment, we are also fixing `DYLD_LIBRARY_PATH' in all the places that we are setting the general environment variables.
2019-07-26PatchELF built statically and updatedMohammad Akhlaghi-2/+8
Until now, like all other software, PatchELF would install with dynamic linking. However, PatchELF links with `libstdc++' so on one system, I noticed that PatchELF gives a segmentation fault and corrupts `libstdc++' while correcting its RPATH (after installing GCC). The solution is to build PatchELF statically. With this commit, we force PatchELF to be built statically (it only installs on GNU/Linux systems anyway, so there is no problem with static linking on macOS). This solved the problem on that system. While looking at its documentation, I also noticed that a new version of PatchELF has been released after almost three years, so it has been updated in the template also. This fixes bug #56673.
2019-07-25Updated version of WCSLIB to 6.3, previously 6.2Mohammad Akhlaghi-3/+3
Some bugs have been fixed in the new version of WCSLIB, so it has been updated in the template.
2019-07-24libgit2 updated to version 0.28.2, from 0.26.0Mohammad Akhlaghi-3/+3
More than two releases and bug fixes have been made to libgit2. So we are now using a more recent version in the template.
2019-07-15Minor corrections in configure and high-levelMohammad Akhlaghi-4/+5
Configure script: when `texlive-ready-tlmgr' is not created, it is similar to not having installed TeXLive. A check was added so in this scenario the `./configure' script doesn't crash. high-level.mk: `cairo' and `pixman' are now installed in parallel and with `V=1' (so the full compilation and linking command is printed).
2019-07-08Corrected typo in the installation of libgit2Mohammad Akhlaghi-1/+1
In one of the last few commits, the commands in the recipe of libgit2 was merged with `&&' so it stops if anything fails. But I had forgot to add a `;' at the end of the `install_name_tool' command. This is corrected with this commit.
2019-07-08Rpath in libstdc++ and passing host_cc to high-levelMohammad Akhlaghi-1/+1
Until this commit, the addion of `-liconv' to `CXXFLAGS' in `high-level.mk' dependend on `host_cc', but I had forgot that `host_cc' isn't defined for `high-level.mk'. It is now defined for this Makefile also in the configure script. Also, the Standard C++ library depends on `libgcc_s.so.1', so after building GCC, it was necessary to add Rpath to it.
2019-07-07Minor corrections in high-level programsMohammad Akhlaghi-23/+24
While doing a clean build, several issues were found in the pipeline and corrected. The main issue was that with the recent installation of `libiconv', the GCC standard C++ library depends on `libiconv', so we need to explicity add an `-liconv' to any C++ compilation. The other corrected points are: - The C++ compiler is now explicitly defined in `CXX'. - libgit2 and WCSLIB's recipes weren't using `&&' between statements, so if there was an error, it would still build the target! - The CMake bootstrapping script now builds much faster in parallel.
2019-07-07Texinfo installed as a dependency of M4Mohammad Akhlaghi-7/+19
While testing on a system with no Texinfo, we noticed that M4 depends on Texinfo. To fix this problem, with this commit, it is now included in the pipeline. While doing a clean build, a few minor issues were also found and corrected in the other rules.
2019-07-07FFTW single-precision library builtMohammad Akhlaghi-8/+7
Until now, we were using the `--enable-single' configure-time option of FFTW to build its single-precision library. The reason was that according to the output of `./configure --help', this was equivalent to `--enable-float'. However, the single precision library wasn't being built. As a result, on systems that already had it, SExtractor would use the system's library and on other systems, it would simply not pass the configure step. With this commit, we are now using `--enable-float' which fixes this problem and installs the `libfftw3f*' libraries (showing that it is not equal to `--enable-single'!). Also, some optimization flags were added to hopefully make it faster. This issue was found thanks to Zahra Sharbaf and Hamed Altafi. This fixes bug #56588.
2019-07-07Using fixed TeX Live repositoryMohammad Akhlaghi-2/+14
Until now, we were letting the TeXLive installer use the default CTAN-chosen mirror based on the host. But in many cases, this is not efficient and sometimes those servers don't work. With this commit, we manually set the server to use (`rit.edu'), which is relatively fast and up to date. In this way, until we build TeXLive from source, every user will be using the same CTAN mirror.
2019-07-04Configuring wget without libiconvMohammad Akhlaghi-0/+4
On some Fedora systems, Wget's build conflicts with libiconv. With this commit, we use Wget's `--without-libiconv-prefix' configure option so it will be built without an outside libiconv.
2019-07-04libffi always installed in the lib directoryMohammad Akhlaghi-1/+11
On some Fedora systems, libffi installs under `lib64', not `lib'. As a result, Python's Setuptools can't find it and will not built (complaining about not finding the `_ctypes' module). With this commit, we fix this problem by explicitly putting a copy of libffi's installed libraries within the `lib' directory. This issue was found while testing the pipeline with Elham Saremi and Hamed Altafi.
2019-07-04C and Fortran compiler environment set in high-level.mkMohammad Akhlaghi-0/+2
Until now, we weren't explicitly setting the C and Fortran compiler environment variables (`CC' and `F77'). As a result, if the user's system had already set them, there would be a problem (and the system's compilers would be used). With this commit, we are explicitly setting these two environment variables at the start of `high-level.mk'. This bug was found after a discussion with Elham Saremi.
2019-06-30Removing libiconv before building pkg-configMohammad Akhlaghi-0/+8
An existing `libiconv' can cause a conflict with `pkg-config', this is why `libiconv' depends on `pkg-config'. On a clean build, `pkg-config' is built first. But when we don't have a clean build (and `libiconv' exists) there will be a problem. With this commit, before re-building `pkg-config', we'll remove any possibly existing installation of `libiconv'.
2019-06-29Imported recent correction from Pierre Gajda, no conflictsMohammad Akhlaghi-1/+1
There was only a conflict with `.file-metadata'.
2019-06-29Added citation for TIDES, sorted progs alphabeticallyMohammad Akhlaghi-107/+142
While reviewing Prasenjit's commits, I noticed that we had forgot to add the citation for TIDES, also to make things clear, the program/library build rules are now sorted alphabetically. Finally, I noticed that after building the TiKZ PDF figures, it is crashing (like on Prasenjit's computer). After looking around, I noticed its because we were setting the of the `TEXINPUTS' environment variable to be the installed TeX Live directory (which was ultimately redundant because by default TeX will look into where it was installed). The important thing is just that we remove any possible value the host system has, not to set new directories.
2019-06-28Fixed a conflict with libiconv and pkg-configPierre Gajda-1/+1
Libiconv has to be installed before pkg-config otherwise there would be a compilation error: gconvert.c:61:2: error: #error GNU libiconv not in use but included iconv.h is from libiconv 61 | #error GNU libiconv not in use but included iconv.h is from libiconv
2019-06-28tides library addedPrasenjit Saha-1/+11
TIDES is an ODE integrator with multiple-precision arithmetic.
2019-06-28Corrections to basic buildPrasenjit Saha-3/+4
Several corrections were necessary in the basic build: 1) the version of GCC on some systems includes an `_' which would cause a crash when building the PDF. 2) libcharset had to be manually added to the Git build.
2019-06-28Libiconv built as a dependency of GitMohammad Akhlaghi-16/+24
Prasenjit Saha reported that while building on a macOS, Git complained about not finding `libiconv' functions. Therefore with this commit, `libiconv' is also built as a dependency of Git. Also, since the build of Binutils (and the subsequent GCC) doesn't depend on the exact versions of the other basic software, they are set as existance-only prerequsites (ignoring their timestamps). This is done to avoid re-building when it is not needed.
2019-06-20Removed old comment above Gnuastro's build ruleMohammad Akhlaghi-5/+0
The comment above Gnuastro's build rule is not longer relevant, so it has been removed.
2019-06-14Binutils is built alone, before GCCMohammad Akhlaghi-40/+55
Until now, on GNU/Linux systems, GNU Binutils was built in parallel with other programs. As a result, the moment that the Binutils executables (most importantly `ld') are being installed, if another program is using them, we can have a crash. For some reason, this crash doesn't happen on Fedora or Arch Linux, but on some Debian-based distors, it causes the following crash when we are building Git (which happened to be built while Binuntils is being installed): credential-cache--daemon.o: file not recognized: File format not recognized In a following run of `./configure' (when Binutils has been installed), Git will build and install successfully. To fix this problem, with this commit, all basic programs are now a prerequisite of Binutils and Binutils is the sole prerequisite of GCC (which is the final target of `basic.mk'). Also, the GCC configure options were re-ordered to be easier to read (the `--with-*' options under each other, the `--enable-*' options under each other and so on).
2019-06-13Metastore now has rpathMohammad Akhlaghi-10/+16
Until now we hadn't noticed that Metastore's build doesn't include an `rpath'. As a result I got a crash during a Git commit. With this commit, on GNU/Linux systems, we now add rpath to the installed Metastore executable. During the build, I also cleaned up the Metastore build recipe to be more clear and readable (in particular we only check the user name and group name when metastore is actually built).
2019-06-12Imported recent work on Imfit, minor conflicts fixedMohammad Akhlaghi-11/+124
The recent work by Raul to import Imfit is now merged into the master branch. Only a few minor conflicts came up due to the update with CFITSIO (that Raul also done in parallel on this branch).
2019-06-12Imfit and libcurl: manually adding rpath on GNU/Linux systemsRaul Infante-Sainz-5/+18
While trying to debug the installation of Imfit, we found out that libcurl doesn't have rpath and is thus linking with the host system's libraries. So rpath is now manually added to the build of libcurl. We also found out that Imfit executables don't have rpath, so it was added to them too.
2019-06-10TeXlive: allows for updating itself and removes old version fileMohammad Akhlaghi-1/+10
Until now, TeXlive would only attempt to download/update LaTeX packages, but after some time, it will be necessary for it to update itself. When this happens, it complains about running it with `update --self'. To fix this problem, before installing/updating the LaTeX packages, we will first update tlmgr in any case. Also, we weren't removing the TeXlive package version file before appending the values to it. So it was just repeating the packages every time it was updated. With this commit, it is being removed on every new build. Finally, the version of Git was updated to its most recent version.
2019-05-31Modified Imfit installation to not consider default and host pathsRaul Infante-Sainz-8/+15
Until this commit, 'imfit' was installed using proper flags in order to use our own libraries. However, it looks first of all to the default system. As a consecuence, `imfit' program was linked to the host system libraries, even when we told to use our own libraries!! With this commit, the installation of `imfit' has been modified in order to remove the default paths already set in the `SConstruct' script. By doing this, only our paths are take into account for compiling the program. It has been tested on Mac OS laptop and it works fine. Test on GNU/Linux systems remain before merge into the main project branch.
2019-05-31Building single and double precission float libraries of FFTWRaul Infante-Sainz-1/+9
Until this commit, `fftw' was building single precission float library because `scamp' had problem on using the default double precission library. However,`imfit' and maybe other programs will use double precission float libraries. With this commit, `fftw' installation is done twice in order to build single and double precission float libraries. It does not complain and goes up to the end without crashing. However, more tests are needed in order to ensure that there is not problem on having both libraries because in principle, they share the same header file and that could cause problems.
2019-05-31Fixing rpath problem in CFITSIO installationRaul Infante-Sainz-13/+13
`cfitsio' was recently updated and it cames up with a problem when trying to build shared libraries (on Mac OS system laptop). This is because the `configure' script for building `cfitsio' include by default `rpath'. With this commit, this issue is fixed by modifying the installation of `cfitsio'. Using `sed', each ocurrence of `@rpath' in `configure' is replaced by `$(ildir)'. Once it has been done, the installation of `cfitsio' goes as normal. In this commit, the version of the `cfitsio' has been also updated to the most recent one.
2019-05-28Corrected typo in Imfit installation, but not working yetRaul Infante-Sainz-1/+1
With this commit, I have corrected a typo in the installation of `imfit'. In principle it is working on Mac OS system but when I tried to install it in GNU/Linux system it crashed. It complains about a problem with the `cfitsio' library: /pathto/libcfitsio.a: error adding symbols: File format not recognized As in the last commits done by Mohammad in the project, `cfitsio' installation has been upgraded and also the installation has been modified, I am going to try this new version. I will also check if it is a problem about dynamic/static library. It also seems that the installation is taking libraries and variables from the host system, so I will check that everything used is from our own programs/libraries.
2019-05-27Software: CFITSIO 3.47, implemented many common standardsMohammad Akhlaghi-31/+6
Until now, CFITSIO had several peculiar features which needed special attention: 1) Its tarball didn't have a `.' in the version name and would sometimes have extra `0's. 2) When unpacked it was just `cfitsio'. 3) It hard coded the directory to search for cURL library. All of these forced us to do a lot of extra steps for CFITSIO. So I got in touch with them and fortunately they were very open to correcting these issues and with version 3.47 all such issues have been addressed in the tarball. Thanks to these changes, with this commit, CFITSIO's build rule is now very similar to many other programs. I also noticed that it has an optional `--with-bz2' feature to enable unpacking `.fits.bz2' files. So this feature is also added.
2019-05-27Added paper citation of ImfitRaul Infante-Sainz-1/+26
Until this commit, `imfit' paper was not included into the acknowledgements. With this commit, a bib tex file for `imfit' has been added. So, now it is properly cited into the final paper.
2019-05-27Added Imfit into the projectRaul Infante-Sainz-1/+32
With this commit, `imfit' has been included into the project. As described in the official webpage (http://www.mpe.mpg.de/~erwin/code/imfit/): Imfit is a program for fitting astronomical images -- especially images of galaxies, though it can in principle be used for fitting other sources. This program uses the `scons' program to build itself (not the traditional GNU `configure' and `make' system). I have tested it by using an example that can be found in the `examples' directory of the decompressed tarball. It works fine without any error on my Mac OS system.
2019-05-27Adding SCons into the projectRaul Infante-Sainz-1/+18
With this commit, SCons has been included into the project. As in the official webpage of this program says: SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software. The motivation for including this software into the project is because there are some software that use this system for the installation. In particular, the necessity of installing it comes from Imfit. This is an astronomical software that uses this system so that is why it has been included into the project. In principle it is simple, and I have tested it on Mac OS sytem without any problem.
2019-05-24PyYAML decompressed tarball directory name fixedRaul Infante-Sainz-1/+1
Until this commit, the name of the decompressed tarball directory of PyYAML Python package was wrong. It has to be `PyYAML-version' instead of `pyyaml-version'. When I run the installation on Mac OS system it went up to the end of the installation with no error. However, when I tried to install it on a GNU/Linux system, it complained about no finding the `pyyaml-version' directory, which is the expected because the name was wrong! With this commit, I have fixed this issue by writting correctly the name of the decompressed tarball directory.
2019-05-24Added PyYAML Python package into the projectRaul Infante-Sainz-0/+13
With this commit, PyYAML Python package has been added into the project. It is widely used in the Python community and the goal is to have human readable configuration files. As in the web page (https://pypi.org/project/PyYAML/) says: YAML is a data serialization format designed for human readability and interaction with scripting languages. PyYAML is a YAML parser and emitter for Python.
2019-05-24Added LibYAML library into the projectRaul Infante-Sainz-0/+6
With this commit, LibYAML library has been added. This library is a prerequisite of the Python package PyYAML, which is widely used in the Python community when configuration scripts are needed.
2019-05-24Shared CFITSIO installation includedMohammad Akhlaghi-2/+3
By default (when you run `make' within CFITSIO), it only builds a static library. So until now, the template would only install a static version of this library. To be more clear about which libraries are used during the linking of programs that use CFITSIO, it would be helpful to have shared library versions also. Also, if the system has problems with static libraries (like macOS systems), then it won't go looking outside the template to link with them (these paths are hardcoded in the linker, which we don't build on macOS and just use the system's tools). With this commit, we are explicity running `make shared' in the CFITSIO installation to have shared CFITSIO also. I also removed the old `static' element (which we don't use anymore in any of the programs and have to remove some time later!).
2019-05-22TeXLive and Ghostscript also added as dependencies of MatplotlibMohammad Akhlaghi-0/+2
Matplotlib can optionally use LaTeX and Ghostscript as dependencies to render text with LaTeX. Since we already have their build rules, with this commit, they have been added as its prerequisites.
2019-05-21ImageMagick put as prerequisite of MatplotlibRaul Infante-Sainz-0/+1
With this commit, ImageMagick program has been set as prerequisite of Matplotlib Python package. Since some submodules of Matplotlib rely on ImageMagick programs (for example, those to make image animations), and now we have ImageMagick into the project, it is good to install ImageMagick when Matplotlib is requested to be installed.
2019-05-21ImageMagick is now included into the projectRaul Infante-Sainz-0/+13
With this commit, ImageMagick software has been added into the project. This software is useful to deal and treat images from the command line. Since it is widely used and a lot of other programs rely on it, it is worth to have it into the project.
2019-05-09Typo, space instead of TAB, corrected in recipe of MetastoreMohammad Akhlaghi-27/+27
In the previous commits where Metastore's recipe was updated, all the `TAB's in the recipe were saved as spaces just before the commit (due to a copying and pasting from the terminal, not the text editor). With this commit, I am correcting this.
2019-05-09Metastore Git hooks properly written after its installationMohammad Akhlaghi-35/+34
Until now, in the recipe to build Metastore, we would store the current directory in a `current_dir' Bash variable before running `gbuild' to build Metastore. However, in this Makefile, we aren't using `.ONESHELL'. As a result, each un-quoted new-line character creates a new shell and the `current_dir' variable that we used afterwards was empty! This happened because until recently, Metastore was being built in `high-level.mk' (where `.ONESHELL' is activated, because we are using GNU Make to call that Makefile). But in `basic.mk', we are using the host's Make, which may not be GNU Make, so we can't use any GNU-only features. Also, this error was hard to notice, because we weren't using `&&' before writing the final target of Metastore! With this commit, both this issues are addressed: all the lines in the recipe are now quoted in the end with a backslash, and the steps before making the target are separated by `&&', not `;' (which doesn't cause a crash if the command fails). This fixes bug #56295.