aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/make/high-level.mk
AgeCommit message (Collapse)AuthorLines
2019-08-15WCSLIB 6.4 doesn't need explicit link with CFITSIOMohammad Akhlaghi-15/+3
This was a bug in WCSLIB 6.3 that has been fixed in WCSLIB 6.4. From WCSLIB's changelog: "The rule change to the Fortran makefile in v6.3 to add getwcstab_f.o to the sharable library causes it to depend on CFITSIO to resolve fits_get_wcstab(). Hence backed out of that change.". The actual error was like this: Undefined symbols for architecture x86_64: "_fits_read_wcstab", referenced from: _ftwcst_ in getwcstab_f.o "_gFitsFiles", referenced from: _ftwcst_ in getwcstab_f.o ld: symbol(s) not found for architecture x86_64
2019-08-08libnsl, libtirpc and rpcsvc installed as non-used biber dependenciesMohammad Akhlaghi-6/+43
These three libraries are dependencies of Biber, so we will need them later, but since we don't build biber from source now, we can't control what library it links with. With this commit, we have just added their versions, checksum, download URL and build rule incase they are useful in other software. Later, when we build Biber (and Texlive in general) from source, we'll be able to use these.
2019-08-08Gnuastro's build: removing the `make check' stepMohammad Akhlaghi-2/+1
Until now, unlike most other programs, in Gnuastro we would run `make check', but this causes a crash on some systems, because of its BuildProgram test: a linking error because the library isn't installed yet. Which is natural in cases like this (and must be corrected in future Gnuastro releases). With this commit, the checks of Gnuastro have been removed.
2019-08-06OpenSSH necessary to use OpenMPIMohammad Akhlaghi-2/+13
Until now, OpenMPI was being installed without any dependency. This was fine because it would indeed build. But the moment you tried loading something that depends on it (for example `mpi4py' through `astropy'), you would get an error complaining that SSH isn't present. With this commit, the pipeline now also installs OpenSSH to solve this problem.
2019-08-05WCSLIB needs -lcfitsio on some systemsMohammad Akhlaghi-1/+13
Until now, we were relying on WCSLIB's internal checking and linking with CFITSIO. But on one macOS system (not others that had no problem!), we noticed that it complains with undefined symbol linking errors to CFITSIO libraries. With this commit, as a fast/ugly solution, we are explicity adding `-lcfitsio' to WCSLIB's `LIBS' variable so all binaries are linked with it automatically. We'll be in touch with the WCSLIB author to see if a better solution can be found.
2019-08-05Backup repository set to http://akhlaghi.org/reproduce-softwareMohammad Akhlaghi-4/+4
Some of the software tarballs are directly available on their webpage (for example due to build problems on systems, where we had to clone the software and build its tarball ourselves until the next release). Until now, only for these software, these tarballs were hosted on `http://akhlaghi.org/src'. But now, I have moved a clone of the software backup repository (including all its software) to `http://akhlaghi.org/reproduce-software'. To be more clear and have a single place for the backup software, the URL of those software has also been updated in the project source.
2019-08-01Bash startup script for every recipeMohammad Akhlaghi-0/+4
Until now the only way to define the environment of the Make recipes was through the exported Make variables (mostly in `initialize.mk' for the analysis steps for example). However, there is only so much you can do with environment variables! In some situations you want slightly more complicated environment control, like setting an alias or running of scripts (things that are commonly done in the `~/.bashrc' file of users to configure their interactive, non-login shells). With this commit, a `reproduce/software/bash/bashrc.sh' has been defined for this job (which is currently empty!). Every major Make step of the project adds this file as the `BASH_ENV' environment variable, so the shell that is created to execute a recipe first executes this file, then the recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment variable that enables users to limit their envirnoment setup based on the condition it is being setup (in particular in the early phase of `basic.mk', where the user can't make any assumption about the programs and has to write a portable shell script).
2019-07-29Checking software tarball checksums before building softwareMohammad Akhlaghi-79/+103
Until now, there was no check on the integrity of the contents of the downloaded/copied software tarballs, we only relied on the tarball name. This could be bad for reproducibility and security, for example on one server the name of a tarball may be the same but with different content. With this commit, the SHA512 checksums of all the software are stored in the newly created `checksums.mk' (similar to how the versions are stored in the `versions.mk'). The resulting variable is then defined for each software and after downloading/copying the file we check to see if the new tarball has the same checksum as the stored value. If it doesn't the script will crash with an error, informing the user of the problem. The only limitation now is a bootstrapping problem: if the host system doesn't already an `sha512sum' executable, we will not do any checksum verification until we install our `sha512sum' (as part of GNU Coreutils). All the tarballs downloaded after GNU Coreutils are built will have their checksums validated. By default almost all GNU/Linux systems will have a usable `sha512sum' (its part of GNU Coreutils after all for a long time: from the Coreutils Changelog file atleast since 2013). This completes task #15347.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-3/+10
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-07-27DYLD_LIBRARY_PATH also fixed for macOS systemsMohammad Akhlaghi-10/+11
Until now we were only setting the `LD_LIBRARY_PATH' environment variable for GNU/Linux systems. But macOS systems use the `DYLD_LIBRARY_PATH'. With this commit, for better control over the environment, we are also fixing `DYLD_LIBRARY_PATH' in all the places that we are setting the general environment variables.
2019-07-25Updated version of WCSLIB to 6.3, previously 6.2Mohammad Akhlaghi-2/+2
Some bugs have been fixed in the new version of WCSLIB, so it has been updated in the template.
2019-07-24libgit2 updated to version 0.28.2, from 0.26.0Mohammad Akhlaghi-2/+2
More than two releases and bug fixes have been made to libgit2. So we are now using a more recent version in the template.
2019-07-15Minor corrections in configure and high-levelMohammad Akhlaghi-4/+5
Configure script: when `texlive-ready-tlmgr' is not created, it is similar to not having installed TeXLive. A check was added so in this scenario the `./configure' script doesn't crash. high-level.mk: `cairo' and `pixman' are now installed in parallel and with `V=1' (so the full compilation and linking command is printed).
2019-07-08Corrected typo in the installation of libgit2Mohammad Akhlaghi-1/+1
In one of the last few commits, the commands in the recipe of libgit2 was merged with `&&' so it stops if anything fails. But I had forgot to add a `;' at the end of the `install_name_tool' command. This is corrected with this commit.
2019-07-07Minor corrections in high-level programsMohammad Akhlaghi-23/+24
While doing a clean build, several issues were found in the pipeline and corrected. The main issue was that with the recent installation of `libiconv', the GCC standard C++ library depends on `libiconv', so we need to explicity add an `-liconv' to any C++ compilation. The other corrected points are: - The C++ compiler is now explicitly defined in `CXX'. - libgit2 and WCSLIB's recipes weren't using `&&' between statements, so if there was an error, it would still build the target! - The CMake bootstrapping script now builds much faster in parallel.
2019-07-07FFTW single-precision library builtMohammad Akhlaghi-8/+7
Until now, we were using the `--enable-single' configure-time option of FFTW to build its single-precision library. The reason was that according to the output of `./configure --help', this was equivalent to `--enable-float'. However, the single precision library wasn't being built. As a result, on systems that already had it, SExtractor would use the system's library and on other systems, it would simply not pass the configure step. With this commit, we are now using `--enable-float' which fixes this problem and installs the `libfftw3f*' libraries (showing that it is not equal to `--enable-single'!). Also, some optimization flags were added to hopefully make it faster. This issue was found thanks to Zahra Sharbaf and Hamed Altafi. This fixes bug #56588.
2019-07-07Using fixed TeX Live repositoryMohammad Akhlaghi-2/+14
Until now, we were letting the TeXLive installer use the default CTAN-chosen mirror based on the host. But in many cases, this is not efficient and sometimes those servers don't work. With this commit, we manually set the server to use (`rit.edu'), which is relatively fast and up to date. In this way, until we build TeXLive from source, every user will be using the same CTAN mirror.
2019-07-04C and Fortran compiler environment set in high-level.mkMohammad Akhlaghi-0/+2
Until now, we weren't explicitly setting the C and Fortran compiler environment variables (`CC' and `F77'). As a result, if the user's system had already set them, there would be a problem (and the system's compilers would be used). With this commit, we are explicitly setting these two environment variables at the start of `high-level.mk'. This bug was found after a discussion with Elham Saremi.
2019-06-29Added citation for TIDES, sorted progs alphabeticallyMohammad Akhlaghi-104/+118
While reviewing Prasenjit's commits, I noticed that we had forgot to add the citation for TIDES, also to make things clear, the program/library build rules are now sorted alphabetically. Finally, I noticed that after building the TiKZ PDF figures, it is crashing (like on Prasenjit's computer). After looking around, I noticed its because we were setting the of the `TEXINPUTS' environment variable to be the installed TeX Live directory (which was ultimately redundant because by default TeX will look into where it was installed). The important thing is just that we remove any possible value the host system has, not to set new directories.
2019-06-28tides library addedPrasenjit Saha-0/+9
TIDES is an ODE integrator with multiple-precision arithmetic.
2019-06-20Removed old comment above Gnuastro's build ruleMohammad Akhlaghi-5/+0
The comment above Gnuastro's build rule is not longer relevant, so it has been removed.
2019-06-12Imported recent work on Imfit, minor conflicts fixedMohammad Akhlaghi-8/+91
The recent work by Raul to import Imfit is now merged into the master branch. Only a few minor conflicts came up due to the update with CFITSIO (that Raul also done in parallel on this branch).
2019-06-12Imfit and libcurl: manually adding rpath on GNU/Linux systemsRaul Infante-Sainz-4/+13
While trying to debug the installation of Imfit, we found out that libcurl doesn't have rpath and is thus linking with the host system's libraries. So rpath is now manually added to the build of libcurl. We also found out that Imfit executables don't have rpath, so it was added to them too.
2019-06-10TeXlive: allows for updating itself and removes old version fileMohammad Akhlaghi-0/+9
Until now, TeXlive would only attempt to download/update LaTeX packages, but after some time, it will be necessary for it to update itself. When this happens, it complains about running it with `update --self'. To fix this problem, before installing/updating the LaTeX packages, we will first update tlmgr in any case. Also, we weren't removing the TeXlive package version file before appending the values to it. So it was just repeating the packages every time it was updated. With this commit, it is being removed on every new build. Finally, the version of Git was updated to its most recent version.
2019-05-31Modified Imfit installation to not consider default and host pathsRaul Infante-Sainz-7/+14
Until this commit, 'imfit' was installed using proper flags in order to use our own libraries. However, it looks first of all to the default system. As a consecuence, `imfit' program was linked to the host system libraries, even when we told to use our own libraries!! With this commit, the installation of `imfit' has been modified in order to remove the default paths already set in the `SConstruct' script. By doing this, only our paths are take into account for compiling the program. It has been tested on Mac OS laptop and it works fine. Test on GNU/Linux systems remain before merge into the main project branch.
2019-05-31Building single and double precission float libraries of FFTWRaul Infante-Sainz-1/+9
Until this commit, `fftw' was building single precission float library because `scamp' had problem on using the default double precission library. However,`imfit' and maybe other programs will use double precission float libraries. With this commit, `fftw' installation is done twice in order to build single and double precission float libraries. It does not complain and goes up to the end without crashing. However, more tests are needed in order to ensure that there is not problem on having both libraries because in principle, they share the same header file and that could cause problems.
2019-05-31Fixing rpath problem in CFITSIO installationRaul Infante-Sainz-12/+12
`cfitsio' was recently updated and it cames up with a problem when trying to build shared libraries (on Mac OS system laptop). This is because the `configure' script for building `cfitsio' include by default `rpath'. With this commit, this issue is fixed by modifying the installation of `cfitsio'. Using `sed', each ocurrence of `@rpath' in `configure' is replaced by `$(ildir)'. Once it has been done, the installation of `cfitsio' goes as normal. In this commit, the version of the `cfitsio' has been also updated to the most recent one.
2019-05-28Corrected typo in Imfit installation, but not working yetRaul Infante-Sainz-1/+1
With this commit, I have corrected a typo in the installation of `imfit'. In principle it is working on Mac OS system but when I tried to install it in GNU/Linux system it crashed. It complains about a problem with the `cfitsio' library: /pathto/libcfitsio.a: error adding symbols: File format not recognized As in the last commits done by Mohammad in the project, `cfitsio' installation has been upgraded and also the installation has been modified, I am going to try this new version. I will also check if it is a problem about dynamic/static library. It also seems that the installation is taking libraries and variables from the host system, so I will check that everything used is from our own programs/libraries.
2019-05-27Software: CFITSIO 3.47, implemented many common standardsMohammad Akhlaghi-30/+5
Until now, CFITSIO had several peculiar features which needed special attention: 1) Its tarball didn't have a `.' in the version name and would sometimes have extra `0's. 2) When unpacked it was just `cfitsio'. 3) It hard coded the directory to search for cURL library. All of these forced us to do a lot of extra steps for CFITSIO. So I got in touch with them and fortunately they were very open to correcting these issues and with version 3.47 all such issues have been addressed in the tarball. Thanks to these changes, with this commit, CFITSIO's build rule is now very similar to many other programs. I also noticed that it has an optional `--with-bz2' feature to enable unpacking `.fits.bz2' files. So this feature is also added.
2019-05-27Added paper citation of ImfitRaul Infante-Sainz-1/+2
Until this commit, `imfit' paper was not included into the acknowledgements. With this commit, a bib tex file for `imfit' has been added. So, now it is properly cited into the final paper.
2019-05-27Added Imfit into the projectRaul Infante-Sainz-0/+30
With this commit, `imfit' has been included into the project. As described in the official webpage (http://www.mpe.mpg.de/~erwin/code/imfit/): Imfit is a program for fitting astronomical images -- especially images of galaxies, though it can in principle be used for fitting other sources. This program uses the `scons' program to build itself (not the traditional GNU `configure' and `make' system). I have tested it by using an example that can be found in the `examples' directory of the decompressed tarball. It works fine without any error on my Mac OS system.
2019-05-27Adding SCons into the projectRaul Infante-Sainz-0/+16
With this commit, SCons has been included into the project. As in the official webpage of this program says: SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software. The motivation for including this software into the project is because there are some software that use this system for the installation. In particular, the necessity of installing it comes from Imfit. This is an astronomical software that uses this system so that is why it has been included into the project. In principle it is simple, and I have tested it on Mac OS sytem without any problem.
2019-05-24Added LibYAML library into the projectRaul Infante-Sainz-0/+5
With this commit, LibYAML library has been added. This library is a prerequisite of the Python package PyYAML, which is widely used in the Python community when configuration scripts are needed.
2019-05-24Shared CFITSIO installation includedMohammad Akhlaghi-2/+3
By default (when you run `make' within CFITSIO), it only builds a static library. So until now, the template would only install a static version of this library. To be more clear about which libraries are used during the linking of programs that use CFITSIO, it would be helpful to have shared library versions also. Also, if the system has problems with static libraries (like macOS systems), then it won't go looking outside the template to link with them (these paths are hardcoded in the linker, which we don't build on macOS and just use the system's tools). With this commit, we are explicity running `make shared' in the CFITSIO installation to have shared CFITSIO also. I also removed the old `static' element (which we don't use anymore in any of the programs and have to remove some time later!).
2019-05-21ImageMagick is now included into the projectRaul Infante-Sainz-0/+12
With this commit, ImageMagick software has been added into the project. This software is useful to deal and treat images from the command line. Since it is widely used and a lot of other programs rely on it, it is worth to have it into the project.
2019-05-06Disabled X Window System in netpbm installation for GNU/Linux systemsRaul Infante-Sainz-1/+1
Until this commit, we were installing `netpbm' without disabling the X Windows System in GNU/Linux. That is the default option in the configure step of `netpbm'. However, we are not installing the necessary libraries (X11) and headers for that. As a consecuence, when testing in a system where we don't have these libraries/headers (Ubuntu 14.04), it crashes. With this commit, we disable the X Windows System option in `netpbm' by giving the answer `none' to one of the last answers to that question in the configure step.
2019-05-01Disabled X Window System in Cairo installationRaul Infante-Sainz-1/+2
Until this commit, we were installing `cairo' without disabling the X Windows System. That is the default option in the configure step of `cairo'. However, we are not installing the necessaries libraries (X11) and headers for that. As a consecuence, when testing in a system where we don't have these libraries/headers (Ubuntu 14.04), it crashes. With this commit, we disable the X Windows System option in `cairo' configure step by setting the option `--with-x=no'.
2019-05-01Corrected typo in netpbm tarball extensionRaul Infante-Sainz-1/+1
In the last imported work about fixing all backslashes, the extension of `netpbm' tarball was not the correct one. Now we are using `.tar.gz' instead of `.tgz'. With this commit, we fix this typo by replacing the `.tgz' by `.tar.gz' in the rule of `netpbm'.
2019-05-01Problem in generic CMake rule correctedMohammad Akhlaghi-3/+4
In the recent correction of backslashes, I had missed adding an `&&' before the `rm' command of the generic CMake building rule (`cbuild' of `build-rules.mk'). With this commit, this has been corrected. A multi-threaded Make was also added to build OpenMPI (which takes long and is often built alone because it takes so long and other things depend on it) and the HDF5 library.
2019-04-30TAB-typo corrected in rule to build SCAMPMohammad Akhlaghi-1/+1
The TAB character wasn't used in the second line of the recipe to build SCAMP. This has been corrected with this commit.
2019-04-30New rule to build Coreutils, it is now built alongMohammad Akhlaghi-1/+1
Recently, after the installation of Coreutils, we added a step to manually run PatchELF over all the contents of the `$(ibdir)'. However, after a fresh test, this method failed: because while PatchELF was updating `mv', another program that was using `mv' couldn't do so and crashed. To fix this problem, Coreutils is now installed in a dummy directory also and the exact files that need `rpath' are identified and PatchELF is run on them (to be clean). Also, higher-level `basic.mk' software (higher-level than Bash), were re-organized so they depend on Coreutils. With this, we are no sure that when Coreutils is being built, it is the only program that is being built and PatchELF won't interfere with any other program. Also, OpenSSL and CMake were set to be bulit in parallel to speed up their build.
2019-04-30Imported some recent/parallel work, conflicts fixedMohammad Akhlaghi-27/+34
Especially because of the new convention regarding backslashes, there were many conflicts that are now fixed. But none were substantial.
2019-04-30End-of-line Backslashs no longer right under each otherMohammad Akhlaghi-193/+193
When we need to quote the new-line character we end the line with a backslash (`\'). Until now, our convention has been to put all such backslashes under each other to help in visual inspection. But this causes a lot of confusion in version control: if only one line's length is larger, the whole block will be marked as changed and thus makes it hard to visually see the actual change. It also makes debuging the code (adding some temporary lines) hard. With this commit, I went through all the files and tried to fix all such cases so only a single white space character is between the last command character and the backslash. Where there was an empty line (ending with a backslash, to help in visually separating the code into blocks), I put the backslash right under the previous line's. This completes task #15259.
2019-04-29Corrected spaces/TAB in some rulesRaul Infante-Sainz-61/+61
In some rules there were some spaces when we should use TAB, and also the other way around. We were using TAB instead using spaces. With this commit, we fix that spaces/TAB in the necessary rules.
2019-04-29Modified copyright information of some filesRaul Infante-Sainz-0/+1
With this commit, the copyright information of some files have been modify in order to include the work of Raul Infante-Sainz.
2019-04-27Astrometry-net installation modified to avoid two reporting stepsRaul Infante-Sainz-0/+8
Until this commit, we had a problem in the installation of Astrometry-net in Mac OS systems. During the runing of the Makefile, it stopped at the reporting stage because it tried to do a couple of steps that in Mac OS are not possible: $ cat /proc/cpuinfo $ free The reason is that `/proc/cpuinfo' and `free' doesn't exist in Mac OS. With this commit, we override these two steps. Since they are only for reporting, they are not mandatory. So, we avoid doing them by using `sed' before running the Makefile.
2019-04-27Modified netpbm installation answers for Mac OS systemsRaul Infante-Sainz-6/+4
Until now, we installed `netpbm' by setting the default paths to libraries and headers. That worked fine for GNU/Linux systems, however, in Mac OS systems we have to explicity say to `netpbm' questions what are the paths to the libraries and headers. We found this problem because we tried to run `solve-field' to obtain the astrometry of some images, and then it goes to use `pnmfile', a program that failed because it was not properly linked with `libnetpmb'. With this commit, we fix this problem by answering properly the questions in the configure step of `netpbm' and setting explicity where we want to place the libraries and headers. With this commit, we also update `netpbm' version. To do that, we have to download the last version, for which it doesn't exist a tarball. So, we built this tarball and put it in the Mohammad's webpage.
2019-04-25Added --enable-single option in FFTW installationRaul Infante-Sainz-1/+1
Until this commit, FFTW was installed by default, without the single precission option. However, when installing `sextractor' in a new system we found an error in the installation complaining about not having single precission files of FFTW. More explicity the error was: "FFTW single precision library files not found in [...]". With this commit, we fix this problem by passing the option `--enable-single' in the installation of FFTW.
2019-04-22High-level software now specified in TARGETS.mkMohammad Akhlaghi-65/+47
Until now, to specify which high-level software you want the project to contain, it was necessary to go into the `high-level.mk' Makefile that is complicated and can create bugs. With this commit, a new `reproduce/software/config/installation/TARGETS.mk' file has been created that is easily/cleanly in charge of documenting the final high-level software that must be built for the project. Also, until now, FFTW was set as a dependency of Numpy while we couldn't actually get Numpy to use it! It was just there for future reference and to justify its build rule. But now that many software won't be built and there is no problem with having rules even though a project might not use them, it has been removed.
2019-04-22Git, Metastore, cURL and Libbsd moved to basic softwareMohammad Akhlaghi-113/+1
Git and Metastore are very basic and fundamental tools for the template, so to keep things clean (let the `high-level.mk' software only represent optional software in the template), these two software (and their dependencies: `cURL' and `Libbsd') will now be built in `basic.mk'.