aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorLines
2019-08-08Gnuastro's build: removing the `make check' stepMohammad Akhlaghi-2/+1
Until now, unlike most other programs, in Gnuastro we would run `make check', but this causes a crash on some systems, because of its BuildProgram test: a linking error because the library isn't installed yet. Which is natural in cases like this (and must be corrected in future Gnuastro releases). With this commit, the checks of Gnuastro have been removed.
2019-08-08Static PatchELF only built when static C library existsMohammad Akhlaghi-14/+69
Until now, when building PatchELF, we would always require that it be done statically. However, some systems don't have a static C library available for linking. This cause a crash in the static building of PatchELF. But a static PatchELF is necessary for correcting RPATH in GCC's outputs. With this commit, in the configure script we check if a static C library is linkable for the compiler. If it isn't then `host_cc' will be set to 1 and GCC won't be built. We also pass the result of this test to `basic.mk' (through `good_static_lib'), so if a static C library isn't available, it builds a dynamically linked PatchELF. This bug was reported by Elham Saremi.
2019-08-07Removed temporary files to check Fortarn compilerMohammad Akhlaghi-2/+3
Until now, the Fortran compiler check wouldn't delete the files it creates in the temporary software building directory. With this commit, the cleaning steps have been added.
2019-08-06OpenSSH necessary to use OpenMPIMohammad Akhlaghi-2/+15
Until now, OpenMPI was being installed without any dependency. This was fine because it would indeed build. But the moment you tried loading something that depends on it (for example `mpi4py' through `astropy'), you would get an error complaining that SSH isn't present. With this commit, the pipeline now also installs OpenSSH to solve this problem.
2019-08-05WCSLIB needs -lcfitsio on some systemsMohammad Akhlaghi-1/+13
Until now, we were relying on WCSLIB's internal checking and linking with CFITSIO. But on one macOS system (not others that had no problem!), we noticed that it complains with undefined symbol linking errors to CFITSIO libraries. With this commit, as a fast/ugly solution, we are explicity adding `-lcfitsio' to WCSLIB's `LIBS' variable so all binaries are linked with it automatically. We'll be in touch with the WCSLIB author to see if a better solution can be found.
2019-08-05Libffi builds without Java supportMohammad Akhlaghi-1/+2
Until now, we weren't explicitly disabling Java in the build of libffi because there was no Java platform on the systems we tested. But recently Yahya Sefidbakht report a crash in libffi (reported in bug #56716) because of Java not being compiled properly. Unfortunately libffi doesn't have a configure option to disable Java, but it does have an internal macro (`NO_JAVA_RAW_API'). With this commit, we we defined this macro in the `make' step, and it solved the problem. This fixes bug #56716.
2019-08-05Backup repository set to http://akhlaghi.org/reproduce-softwareMohammad Akhlaghi-15/+17
Some of the software tarballs are directly available on their webpage (for example due to build problems on systems, where we had to clone the software and build its tarball ourselves until the next release). Until now, only for these software, these tarballs were hosted on `http://akhlaghi.org/src'. But now, I have moved a clone of the software backup repository (including all its software) to `http://akhlaghi.org/reproduce-software'. To be more clear and have a single place for the backup software, the URL of those software has also been updated in the project source.
2019-08-05Gnuastro updated to version 0.10Mohammad Akhlaghi-2/+2
A new version of Gnuastro was recently released with many improvments and bug fixes, so it is updated here too.
2019-08-05Configuration: Checking that C and Fortran compilers workMohammad Akhlaghi-116/+199
Until now, we were just checking for the existance of a C and Fortran compiler. But it can happen that even if they exist, they don't operate properly, for example see some errors that have been reported until now in P.S. (both on different macOS systems). But finding this source after the programs have started is frustrating for the user. With this commit, before we start building anything, we'll check these two compilers with a simple program and see if they can indeed compile, and if their compiled program can run. If it doesn't work an elaborate error message is printed to help the users navigate to a solution. Also, the building of `flock' within `configure.sh' has been moved just before calling `basic.mk'. This was done so any warning/error message is printed before actually building anything. This fixes bug #56715. P.S. The error messages: C compiler ---------- conftest.c:9:19: fatal error: stdio.h: No such file or directory ^ compilation terminated. ---------- Fortran compiler ---------------- dyld: Library not loaded: @rpath/libisl.10.dylib Referenced from: /path/to/anaconda2/gcc/libexec/gcc/x86_64-apple-darwin15.5.0/4.9.3/f951 Reason: image not found gfortran: internal compiler error: Abort trap: 6 ----------------
2019-08-01Updated version of mpi4py doesn't conflict with OpenMPIMohammad Akhlaghi-14/+5
Until now, in version 3.0.1, mpi4py couldn't be built with the most recent version of OpenMPI. However, after trying the next version (3.0.2), I noticed that it builds successfully without a problem.
2019-08-01Git hooks removed after doing a distcleanMohammad Akhlaghi-0/+5
Until now, when you needed to completely clean a project (with `./project make distclean') the Git hooks that are installed during configure time would cause problems when committing (the `pre-commit' hook in particular won't allow you to commit anything!). With this commit, before deleting the software, the template first removes these Git hooks.
2019-08-01Typo in previous commit correctedMohammad Akhlaghi-1/+0
After making the previous commit, I noticed an extra line (redundantly defining a wrong `BASH_ENV') that should have been deleted. It has been corrected.
2019-08-01Bash startup script for every recipeMohammad Akhlaghi-0/+66
Until now the only way to define the environment of the Make recipes was through the exported Make variables (mostly in `initialize.mk' for the analysis steps for example). However, there is only so much you can do with environment variables! In some situations you want slightly more complicated environment control, like setting an alias or running of scripts (things that are commonly done in the `~/.bashrc' file of users to configure their interactive, non-login shells). With this commit, a `reproduce/software/bash/bashrc.sh' has been defined for this job (which is currently empty!). Every major Make step of the project adds this file as the `BASH_ENV' environment variable, so the shell that is created to execute a recipe first executes this file, then the recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment variable that enables users to limit their envirnoment setup based on the condition it is being setup (in particular in the early phase of `basic.mk', where the user can't make any assumption about the programs and has to write a portable shell script).
2019-07-31GCC: libstdc++ has a link to libiconvMohammad Akhlaghi-0/+1
After adding the libiconv library to the template, the C++ library uses three of its functions (`libiconv', `libiconv_open' and `libiconv_close'). However, it doesn't explicity link with it inside its shared library! I tried by exporting `LIBS=-liconv' before the GCC configure script but it crashed as soon as it went on to the second GCC building stage (because this environment variable was no longer present there). I also tried adding the C++ configure option of `--enable-cstdio' to the GCC configure options (so it doesn't use iconv according to the manual), but it made no change. With this commit, as a brute-force solution, `patchelf --add-needed' is run on the installed `libstdc++.so', so `libiconv.so' is explicitly included inside the `libstdc++' shared library. This bug was found by Roberto Baena Galle while trying to load Matplotlib (which needed the C++ library). This fixes bug #56702.
2019-07-29Checking software tarball checksums before building softwareMohammad Akhlaghi-245/+472
Until now, there was no check on the integrity of the contents of the downloaded/copied software tarballs, we only relied on the tarball name. This could be bad for reproducibility and security, for example on one server the name of a tarball may be the same but with different content. With this commit, the SHA512 checksums of all the software are stored in the newly created `checksums.mk' (similar to how the versions are stored in the `versions.mk'). The resulting variable is then defined for each software and after downloading/copying the file we check to see if the new tarball has the same checksum as the stored value. If it doesn't the script will crash with an error, informing the user of the problem. The only limitation now is a bootstrapping problem: if the host system doesn't already an `sha512sum' executable, we will not do any checksum verification until we install our `sha512sum' (as part of GNU Coreutils). All the tarballs downloaded after GNU Coreutils are built will have their checksums validated. By default almost all GNU/Linux systems will have a usable `sha512sum' (its part of GNU Coreutils after all for a long time: from the Coreutils Changelog file atleast since 2013). This completes task #15347.
2019-07-29Added explanation on .file-metadata within README-hacking.mdMohammad Akhlaghi-0/+36
Until now there was no clear explanation on `.file-metadata' within the project. But since it sometimes appears in the Git changed files and its binary, it was necessary to add a short explanation to inform users. With this commit a section has been added to the "Project architecture" section of `README-hacking.md' to give some context on what it is and how to deal with it. This was suggested by Hamed Altafi.
2019-07-29Tip added at initial configuration notice on how to see progressMohammad Akhlaghi-0/+10
The configuration step (building all the ncessary software) can take some time. It is natual for the user to want to see how the build is going (which software is being built at every moment). So far, we have only put a "Inspecting status" section in `README-hacking.md' that describes a solution, but some early users may not have read it yet. With this commit a short tip was added in the initial installation notice to inform the user of this very useful command.
2019-07-28Corrected typo in environment before running makeMohammad Akhlaghi-4/+4
We recently moved the system's `rm' program absolute address to a shell variable that is found during the `./project' script. But I had forgot to account for the difference between the Make and Bash variable naming differences. I had also forgot to add a value to the HOME variable. With this commit both are corrected: the system's `rm' path is now called `sys_rm' and the HOME variable is set.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-440/+479
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-07-27DYLD_LIBRARY_PATH also fixed for macOS systemsMohammad Akhlaghi-23/+26
Until now we were only setting the `LD_LIBRARY_PATH' environment variable for GNU/Linux systems. But macOS systems use the `DYLD_LIBRARY_PATH'. With this commit, for better control over the environment, we are also fixing `DYLD_LIBRARY_PATH' in all the places that we are setting the general environment variables.
2019-07-26Better explanation of suggested commit messages in REAME-hacking.mdMohammad Akhlaghi-19/+25
Until now the description of the commit message guidelines wasn't clear enough and could cause confusion, in particular because it didn't describe why some basic formatting issues are mandatory. With this commit, it is explained that the main reason we require contributors for follow this format is "consistency" within the project. Also generally it was edited to become more elaborate and explain the points more clearly. I also ran a spell check over the whole file and fixed a few typos. This correction was suggested by Mohammad-reza Khellat.
2019-07-26PatchELF built statically and updatedMohammad Akhlaghi-2/+8
Until now, like all other software, PatchELF would install with dynamic linking. However, PatchELF links with `libstdc++' so on one system, I noticed that PatchELF gives a segmentation fault and corrupts `libstdc++' while correcting its RPATH (after installing GCC). The solution is to build PatchELF statically. With this commit, we force PatchELF to be built statically (it only installs on GNU/Linux systems anyway, so there is no problem with static linking on macOS). This solved the problem on that system. While looking at its documentation, I also noticed that a new version of PatchELF has been released after almost three years, so it has been updated in the template also. This fixes bug #56673.
2019-07-25Updated version of WCSLIB to 6.3, previously 6.2Mohammad Akhlaghi-3/+3
Some bugs have been fixed in the new version of WCSLIB, so it has been updated in the template.
2019-07-25Corrected preparation to build software in shared memoryMohammad Akhlaghi-22/+25
The previous implementation of using shared memory to build software was edited to be more clear and less prone to errors.
2019-07-24libgit2 updated to version 0.28.2, from 0.26.0Mohammad Akhlaghi-3/+3
More than two releases and bug fixes have been made to libgit2. So we are now using a more recent version in the template.
2019-07-24Temporary software building done in shared memory if availableMohammad Akhlaghi-6/+46
Until now, the template would unpack the software and build them directly in the build directory of each project. After installation, the whole unpacked directory is deleted. However, building the software involves the reading and writing of millions of files, so on the long run, it can be bad for the non-volatile memory (HDDs or SSDs), it can also be slightly slow. To fix this, if the system has a shared-memory directory (commonly named `/dev/shm'), we can do the temporary building of the software there. The great thing about this unique directory is that it is actually in the RAM, not on the HDD/SSD. This can slightly improve the speed (not much probably), but more importantly it will not do any long-term harm to the host's HDDs/SSDs. With this commit, when there is a shared memory directory mounted in `/dev/shm', and it has enough space (currently set to 2GB), the `./configure' script will make `.build/software/build-tmp' as a symbolic link to a fixed directory there. Otherwise, it will just build it as a directory in the project's shared directory. The structure has been defined in such a way that we can later easily add different standard shared-memory locations (for different operating systems). This completes task #15336.
2019-07-15Minor corrections in configure and high-levelMohammad Akhlaghi-5/+10
Configure script: when `texlive-ready-tlmgr' is not created, it is similar to not having installed TeXLive. A check was added so in this scenario the `./configure' script doesn't crash. high-level.mk: `cairo' and `pixman' are now installed in parallel and with `V=1' (so the full compilation and linking command is printed).
2019-07-08Corrected typo in the installation of libgit2Mohammad Akhlaghi-1/+1
In one of the last few commits, the commands in the recipe of libgit2 was merged with `&&' so it stops if anything fails. But I had forgot to add a `;' at the end of the `install_name_tool' command. This is corrected with this commit.
2019-07-08Rpath in libstdc++ and passing host_cc to high-levelMohammad Akhlaghi-2/+3
Until this commit, the addion of `-liconv' to `CXXFLAGS' in `high-level.mk' dependend on `host_cc', but I had forgot that `host_cc' isn't defined for `high-level.mk'. It is now defined for this Makefile also in the configure script. Also, the Standard C++ library depends on `libgcc_s.so.1', so after building GCC, it was necessary to add Rpath to it.
2019-07-07Minor corrections in high-level programsMohammad Akhlaghi-23/+24
While doing a clean build, several issues were found in the pipeline and corrected. The main issue was that with the recent installation of `libiconv', the GCC standard C++ library depends on `libiconv', so we need to explicity add an `-liconv' to any C++ compilation. The other corrected points are: - The C++ compiler is now explicitly defined in `CXX'. - libgit2 and WCSLIB's recipes weren't using `&&' between statements, so if there was an error, it would still build the target! - The CMake bootstrapping script now builds much faster in parallel.
2019-07-07Texinfo installed as a dependency of M4Mohammad Akhlaghi-7/+19
While testing on a system with no Texinfo, we noticed that M4 depends on Texinfo. To fix this problem, with this commit, it is now included in the pipeline. While doing a clean build, a few minor issues were also found and corrected in the other rules.
2019-07-07Compiler checks added to the configure scriptMohammad Akhlaghi-0/+77
To build the software, we need a C/C++ compiler. But until now at configure time there was no check on this. As a result, when the system didn't have them, the user would only learn in a crash of the programs. With this commit, some checks have been added at configure time to do basic checks and inform the user if necessary.
2019-07-07Commit message format tip added to README-hacking.mdMohammad Akhlaghi-0/+16
Until now there was no guideline in `README-hacking.md' to describe/suggest a good format for commit messages. With this commit a point has been added in the "Tips" section to help new developers contribute more smoothly. The necessity of this paragraph was pointed out by Mohammad-reza Khellat.
2019-07-07FFTW single-precision library builtMohammad Akhlaghi-8/+7
Until now, we were using the `--enable-single' configure-time option of FFTW to build its single-precision library. The reason was that according to the output of `./configure --help', this was equivalent to `--enable-float'. However, the single precision library wasn't being built. As a result, on systems that already had it, SExtractor would use the system's library and on other systems, it would simply not pass the configure step. With this commit, we are now using `--enable-float' which fixes this problem and installs the `libfftw3f*' libraries (showing that it is not equal to `--enable-single'!). Also, some optimization flags were added to hopefully make it faster. This issue was found thanks to Zahra Sharbaf and Hamed Altafi. This fixes bug #56588.
2019-07-07Using fixed TeX Live repositoryMohammad Akhlaghi-2/+14
Until now, we were letting the TeXLive installer use the default CTAN-chosen mirror based on the host. But in many cases, this is not efficient and sometimes those servers don't work. With this commit, we manually set the server to use (`rit.edu'), which is relatively fast and up to date. In this way, until we build TeXLive from source, every user will be using the same CTAN mirror.
2019-07-04Configuring wget without libiconvMohammad Akhlaghi-0/+4
On some Fedora systems, Wget's build conflicts with libiconv. With this commit, we use Wget's `--without-libiconv-prefix' configure option so it will be built without an outside libiconv.
2019-07-04libffi always installed in the lib directoryMohammad Akhlaghi-1/+11
On some Fedora systems, libffi installs under `lib64', not `lib'. As a result, Python's Setuptools can't find it and will not built (complaining about not finding the `_ctypes' module). With this commit, we fix this problem by explicitly putting a copy of libffi's installed libraries within the `lib' directory. This issue was found while testing the pipeline with Elham Saremi and Hamed Altafi.
2019-07-04C and Fortran compiler environment set in high-level.mkMohammad Akhlaghi-0/+2
Until now, we weren't explicitly setting the C and Fortran compiler environment variables (`CC' and `F77'). As a result, if the user's system had already set them, there would be a problem (and the system's compilers would be used). With this commit, we are explicitly setting these two environment variables at the start of `high-level.mk'. This bug was found after a discussion with Elham Saremi.
2019-06-30Removing libiconv before building pkg-configMohammad Akhlaghi-0/+8
An existing `libiconv' can cause a conflict with `pkg-config', this is why `libiconv' depends on `pkg-config'. On a clean build, `pkg-config' is built first. But when we don't have a clean build (and `libiconv' exists) there will be a problem. With this commit, before re-building `pkg-config', we'll remove any possibly existing installation of `libiconv'.
2019-06-29Imported recent correction from Pierre Gajda, no conflictsMohammad Akhlaghi-1/+1
There was only a conflict with `.file-metadata'.
2019-06-29Added citation for TIDES, sorted progs alphabeticallyMohammad Akhlaghi-113/+152
While reviewing Prasenjit's commits, I noticed that we had forgot to add the citation for TIDES, also to make things clear, the program/library build rules are now sorted alphabetically. Finally, I noticed that after building the TiKZ PDF figures, it is crashing (like on Prasenjit's computer). After looking around, I noticed its because we were setting the of the `TEXINPUTS' environment variable to be the installed TeX Live directory (which was ultimately redundant because by default TeX will look into where it was installed). The important thing is just that we remove any possible value the host system has, not to set new directories.
2019-06-28Fixed a conflict with libiconv and pkg-configPierre Gajda-1/+1
Libiconv has to be installed before pkg-config otherwise there would be a compilation error: gconvert.c:61:2: error: #error GNU libiconv not in use but included iconv.h is from libiconv 61 | #error GNU libiconv not in use but included iconv.h is from libiconv
2019-06-28tides library addedPrasenjit Saha-2/+12
TIDES is an ODE integrator with multiple-precision arithmetic.
2019-06-28Corrections to basic buildPrasenjit Saha-4/+6
Several corrections were necessary in the basic build: 1) the version of GCC on some systems includes an `_' which would cause a crash when building the PDF. 2) libcharset had to be manually added to the Git build.
2019-06-28Libiconv built as a dependency of GitMohammad Akhlaghi-16/+24
Prasenjit Saha reported that while building on a macOS, Git complained about not finding `libiconv' functions. Therefore with this commit, `libiconv' is also built as a dependency of Git. Also, since the build of Binutils (and the subsequent GCC) doesn't depend on the exact versions of the other basic software, they are set as existance-only prerequsites (ignoring their timestamps). This is done to avoid re-building when it is not needed.
2019-06-20Removed old comment above Gnuastro's build ruleMohammad Akhlaghi-5/+0
The comment above Gnuastro's build rule is not longer relevant, so it has been removed.
2019-06-14Minor correction in README-hacking.md for better web-renderingMohammad Akhlaghi-5/+5
The new command-box wasn't being rendered properly, so another correction is made here. I also added the prompt `$' sign in another box of commands.
2019-06-14Minor correction is code demonstration in previous commitMohammad Akhlaghi-1/+2
After checking the previous commit on Gitlab (to see how it is rendered), I noticed that the code has come in the same line, not as a sperate box. Hopefully this commit will fix it.
2019-06-14Software building status check tip added to README-hacking.mdMohammad Akhlaghi-0/+18
It is useful to visually see how the building of software is progressing when running configure. I have been using a simple Bash `while' loop for this, so I added it in the `README-hacking.md' to be useful for others too.
2019-06-14Binutils is built alone, before GCCMohammad Akhlaghi-40/+55
Until now, on GNU/Linux systems, GNU Binutils was built in parallel with other programs. As a result, the moment that the Binutils executables (most importantly `ld') are being installed, if another program is using them, we can have a crash. For some reason, this crash doesn't happen on Fedora or Arch Linux, but on some Debian-based distors, it causes the following crash when we are building Git (which happened to be built while Binuntils is being installed): credential-cache--daemon.o: file not recognized: File format not recognized In a following run of `./configure' (when Binutils has been installed), Git will build and install successfully. To fix this problem, with this commit, all basic programs are now a prerequisite of Binutils and Binutils is the sole prerequisite of GCC (which is the final target of `basic.mk'). Also, the GCC configure options were re-ordered to be easier to read (the `--with-*' options under each other, the `--enable-*' options under each other and so on).