aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/make/high-level.mk
AgeCommit message (Collapse)AuthorLines
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-3/+10
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-07-27DYLD_LIBRARY_PATH also fixed for macOS systemsMohammad Akhlaghi-10/+11
Until now we were only setting the `LD_LIBRARY_PATH' environment variable for GNU/Linux systems. But macOS systems use the `DYLD_LIBRARY_PATH'. With this commit, for better control over the environment, we are also fixing `DYLD_LIBRARY_PATH' in all the places that we are setting the general environment variables.
2019-07-25Updated version of WCSLIB to 6.3, previously 6.2Mohammad Akhlaghi-2/+2
Some bugs have been fixed in the new version of WCSLIB, so it has been updated in the template.
2019-07-24libgit2 updated to version 0.28.2, from 0.26.0Mohammad Akhlaghi-2/+2
More than two releases and bug fixes have been made to libgit2. So we are now using a more recent version in the template.
2019-07-15Minor corrections in configure and high-levelMohammad Akhlaghi-4/+5
Configure script: when `texlive-ready-tlmgr' is not created, it is similar to not having installed TeXLive. A check was added so in this scenario the `./configure' script doesn't crash. high-level.mk: `cairo' and `pixman' are now installed in parallel and with `V=1' (so the full compilation and linking command is printed).
2019-07-08Corrected typo in the installation of libgit2Mohammad Akhlaghi-1/+1
In one of the last few commits, the commands in the recipe of libgit2 was merged with `&&' so it stops if anything fails. But I had forgot to add a `;' at the end of the `install_name_tool' command. This is corrected with this commit.
2019-07-07Minor corrections in high-level programsMohammad Akhlaghi-23/+24
While doing a clean build, several issues were found in the pipeline and corrected. The main issue was that with the recent installation of `libiconv', the GCC standard C++ library depends on `libiconv', so we need to explicity add an `-liconv' to any C++ compilation. The other corrected points are: - The C++ compiler is now explicitly defined in `CXX'. - libgit2 and WCSLIB's recipes weren't using `&&' between statements, so if there was an error, it would still build the target! - The CMake bootstrapping script now builds much faster in parallel.
2019-07-07FFTW single-precision library builtMohammad Akhlaghi-8/+7
Until now, we were using the `--enable-single' configure-time option of FFTW to build its single-precision library. The reason was that according to the output of `./configure --help', this was equivalent to `--enable-float'. However, the single precision library wasn't being built. As a result, on systems that already had it, SExtractor would use the system's library and on other systems, it would simply not pass the configure step. With this commit, we are now using `--enable-float' which fixes this problem and installs the `libfftw3f*' libraries (showing that it is not equal to `--enable-single'!). Also, some optimization flags were added to hopefully make it faster. This issue was found thanks to Zahra Sharbaf and Hamed Altafi. This fixes bug #56588.
2019-07-07Using fixed TeX Live repositoryMohammad Akhlaghi-2/+14
Until now, we were letting the TeXLive installer use the default CTAN-chosen mirror based on the host. But in many cases, this is not efficient and sometimes those servers don't work. With this commit, we manually set the server to use (`rit.edu'), which is relatively fast and up to date. In this way, until we build TeXLive from source, every user will be using the same CTAN mirror.
2019-07-04C and Fortran compiler environment set in high-level.mkMohammad Akhlaghi-0/+2
Until now, we weren't explicitly setting the C and Fortran compiler environment variables (`CC' and `F77'). As a result, if the user's system had already set them, there would be a problem (and the system's compilers would be used). With this commit, we are explicitly setting these two environment variables at the start of `high-level.mk'. This bug was found after a discussion with Elham Saremi.
2019-06-29Added citation for TIDES, sorted progs alphabeticallyMohammad Akhlaghi-104/+118
While reviewing Prasenjit's commits, I noticed that we had forgot to add the citation for TIDES, also to make things clear, the program/library build rules are now sorted alphabetically. Finally, I noticed that after building the TiKZ PDF figures, it is crashing (like on Prasenjit's computer). After looking around, I noticed its because we were setting the of the `TEXINPUTS' environment variable to be the installed TeX Live directory (which was ultimately redundant because by default TeX will look into where it was installed). The important thing is just that we remove any possible value the host system has, not to set new directories.
2019-06-28tides library addedPrasenjit Saha-0/+9
TIDES is an ODE integrator with multiple-precision arithmetic.
2019-06-20Removed old comment above Gnuastro's build ruleMohammad Akhlaghi-5/+0
The comment above Gnuastro's build rule is not longer relevant, so it has been removed.
2019-06-12Imported recent work on Imfit, minor conflicts fixedMohammad Akhlaghi-8/+91
The recent work by Raul to import Imfit is now merged into the master branch. Only a few minor conflicts came up due to the update with CFITSIO (that Raul also done in parallel on this branch).
2019-06-12Imfit and libcurl: manually adding rpath on GNU/Linux systemsRaul Infante-Sainz-4/+13
While trying to debug the installation of Imfit, we found out that libcurl doesn't have rpath and is thus linking with the host system's libraries. So rpath is now manually added to the build of libcurl. We also found out that Imfit executables don't have rpath, so it was added to them too.
2019-06-10TeXlive: allows for updating itself and removes old version fileMohammad Akhlaghi-0/+9
Until now, TeXlive would only attempt to download/update LaTeX packages, but after some time, it will be necessary for it to update itself. When this happens, it complains about running it with `update --self'. To fix this problem, before installing/updating the LaTeX packages, we will first update tlmgr in any case. Also, we weren't removing the TeXlive package version file before appending the values to it. So it was just repeating the packages every time it was updated. With this commit, it is being removed on every new build. Finally, the version of Git was updated to its most recent version.
2019-05-31Modified Imfit installation to not consider default and host pathsRaul Infante-Sainz-7/+14
Until this commit, 'imfit' was installed using proper flags in order to use our own libraries. However, it looks first of all to the default system. As a consecuence, `imfit' program was linked to the host system libraries, even when we told to use our own libraries!! With this commit, the installation of `imfit' has been modified in order to remove the default paths already set in the `SConstruct' script. By doing this, only our paths are take into account for compiling the program. It has been tested on Mac OS laptop and it works fine. Test on GNU/Linux systems remain before merge into the main project branch.
2019-05-31Building single and double precission float libraries of FFTWRaul Infante-Sainz-1/+9
Until this commit, `fftw' was building single precission float library because `scamp' had problem on using the default double precission library. However,`imfit' and maybe other programs will use double precission float libraries. With this commit, `fftw' installation is done twice in order to build single and double precission float libraries. It does not complain and goes up to the end without crashing. However, more tests are needed in order to ensure that there is not problem on having both libraries because in principle, they share the same header file and that could cause problems.
2019-05-31Fixing rpath problem in CFITSIO installationRaul Infante-Sainz-12/+12
`cfitsio' was recently updated and it cames up with a problem when trying to build shared libraries (on Mac OS system laptop). This is because the `configure' script for building `cfitsio' include by default `rpath'. With this commit, this issue is fixed by modifying the installation of `cfitsio'. Using `sed', each ocurrence of `@rpath' in `configure' is replaced by `$(ildir)'. Once it has been done, the installation of `cfitsio' goes as normal. In this commit, the version of the `cfitsio' has been also updated to the most recent one.
2019-05-28Corrected typo in Imfit installation, but not working yetRaul Infante-Sainz-1/+1
With this commit, I have corrected a typo in the installation of `imfit'. In principle it is working on Mac OS system but when I tried to install it in GNU/Linux system it crashed. It complains about a problem with the `cfitsio' library: /pathto/libcfitsio.a: error adding symbols: File format not recognized As in the last commits done by Mohammad in the project, `cfitsio' installation has been upgraded and also the installation has been modified, I am going to try this new version. I will also check if it is a problem about dynamic/static library. It also seems that the installation is taking libraries and variables from the host system, so I will check that everything used is from our own programs/libraries.
2019-05-27Software: CFITSIO 3.47, implemented many common standardsMohammad Akhlaghi-30/+5
Until now, CFITSIO had several peculiar features which needed special attention: 1) Its tarball didn't have a `.' in the version name and would sometimes have extra `0's. 2) When unpacked it was just `cfitsio'. 3) It hard coded the directory to search for cURL library. All of these forced us to do a lot of extra steps for CFITSIO. So I got in touch with them and fortunately they were very open to correcting these issues and with version 3.47 all such issues have been addressed in the tarball. Thanks to these changes, with this commit, CFITSIO's build rule is now very similar to many other programs. I also noticed that it has an optional `--with-bz2' feature to enable unpacking `.fits.bz2' files. So this feature is also added.
2019-05-27Added paper citation of ImfitRaul Infante-Sainz-1/+2
Until this commit, `imfit' paper was not included into the acknowledgements. With this commit, a bib tex file for `imfit' has been added. So, now it is properly cited into the final paper.
2019-05-27Added Imfit into the projectRaul Infante-Sainz-0/+30
With this commit, `imfit' has been included into the project. As described in the official webpage (http://www.mpe.mpg.de/~erwin/code/imfit/): Imfit is a program for fitting astronomical images -- especially images of galaxies, though it can in principle be used for fitting other sources. This program uses the `scons' program to build itself (not the traditional GNU `configure' and `make' system). I have tested it by using an example that can be found in the `examples' directory of the decompressed tarball. It works fine without any error on my Mac OS system.
2019-05-27Adding SCons into the projectRaul Infante-Sainz-0/+16
With this commit, SCons has been included into the project. As in the official webpage of this program says: SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software. The motivation for including this software into the project is because there are some software that use this system for the installation. In particular, the necessity of installing it comes from Imfit. This is an astronomical software that uses this system so that is why it has been included into the project. In principle it is simple, and I have tested it on Mac OS sytem without any problem.
2019-05-24Added LibYAML library into the projectRaul Infante-Sainz-0/+5
With this commit, LibYAML library has been added. This library is a prerequisite of the Python package PyYAML, which is widely used in the Python community when configuration scripts are needed.
2019-05-24Shared CFITSIO installation includedMohammad Akhlaghi-2/+3
By default (when you run `make' within CFITSIO), it only builds a static library. So until now, the template would only install a static version of this library. To be more clear about which libraries are used during the linking of programs that use CFITSIO, it would be helpful to have shared library versions also. Also, if the system has problems with static libraries (like macOS systems), then it won't go looking outside the template to link with them (these paths are hardcoded in the linker, which we don't build on macOS and just use the system's tools). With this commit, we are explicity running `make shared' in the CFITSIO installation to have shared CFITSIO also. I also removed the old `static' element (which we don't use anymore in any of the programs and have to remove some time later!).
2019-05-21ImageMagick is now included into the projectRaul Infante-Sainz-0/+12
With this commit, ImageMagick software has been added into the project. This software is useful to deal and treat images from the command line. Since it is widely used and a lot of other programs rely on it, it is worth to have it into the project.
2019-05-06Disabled X Window System in netpbm installation for GNU/Linux systemsRaul Infante-Sainz-1/+1
Until this commit, we were installing `netpbm' without disabling the X Windows System in GNU/Linux. That is the default option in the configure step of `netpbm'. However, we are not installing the necessary libraries (X11) and headers for that. As a consecuence, when testing in a system where we don't have these libraries/headers (Ubuntu 14.04), it crashes. With this commit, we disable the X Windows System option in `netpbm' by giving the answer `none' to one of the last answers to that question in the configure step.
2019-05-01Disabled X Window System in Cairo installationRaul Infante-Sainz-1/+2
Until this commit, we were installing `cairo' without disabling the X Windows System. That is the default option in the configure step of `cairo'. However, we are not installing the necessaries libraries (X11) and headers for that. As a consecuence, when testing in a system where we don't have these libraries/headers (Ubuntu 14.04), it crashes. With this commit, we disable the X Windows System option in `cairo' configure step by setting the option `--with-x=no'.
2019-05-01Corrected typo in netpbm tarball extensionRaul Infante-Sainz-1/+1
In the last imported work about fixing all backslashes, the extension of `netpbm' tarball was not the correct one. Now we are using `.tar.gz' instead of `.tgz'. With this commit, we fix this typo by replacing the `.tgz' by `.tar.gz' in the rule of `netpbm'.
2019-05-01Problem in generic CMake rule correctedMohammad Akhlaghi-3/+4
In the recent correction of backslashes, I had missed adding an `&&' before the `rm' command of the generic CMake building rule (`cbuild' of `build-rules.mk'). With this commit, this has been corrected. A multi-threaded Make was also added to build OpenMPI (which takes long and is often built alone because it takes so long and other things depend on it) and the HDF5 library.
2019-04-30TAB-typo corrected in rule to build SCAMPMohammad Akhlaghi-1/+1
The TAB character wasn't used in the second line of the recipe to build SCAMP. This has been corrected with this commit.
2019-04-30New rule to build Coreutils, it is now built alongMohammad Akhlaghi-1/+1
Recently, after the installation of Coreutils, we added a step to manually run PatchELF over all the contents of the `$(ibdir)'. However, after a fresh test, this method failed: because while PatchELF was updating `mv', another program that was using `mv' couldn't do so and crashed. To fix this problem, Coreutils is now installed in a dummy directory also and the exact files that need `rpath' are identified and PatchELF is run on them (to be clean). Also, higher-level `basic.mk' software (higher-level than Bash), were re-organized so they depend on Coreutils. With this, we are no sure that when Coreutils is being built, it is the only program that is being built and PatchELF won't interfere with any other program. Also, OpenSSL and CMake were set to be bulit in parallel to speed up their build.
2019-04-30Imported some recent/parallel work, conflicts fixedMohammad Akhlaghi-27/+34
Especially because of the new convention regarding backslashes, there were many conflicts that are now fixed. But none were substantial.
2019-04-30End-of-line Backslashs no longer right under each otherMohammad Akhlaghi-193/+193
When we need to quote the new-line character we end the line with a backslash (`\'). Until now, our convention has been to put all such backslashes under each other to help in visual inspection. But this causes a lot of confusion in version control: if only one line's length is larger, the whole block will be marked as changed and thus makes it hard to visually see the actual change. It also makes debuging the code (adding some temporary lines) hard. With this commit, I went through all the files and tried to fix all such cases so only a single white space character is between the last command character and the backslash. Where there was an empty line (ending with a backslash, to help in visually separating the code into blocks), I put the backslash right under the previous line's. This completes task #15259.
2019-04-29Corrected spaces/TAB in some rulesRaul Infante-Sainz-61/+61
In some rules there were some spaces when we should use TAB, and also the other way around. We were using TAB instead using spaces. With this commit, we fix that spaces/TAB in the necessary rules.
2019-04-29Modified copyright information of some filesRaul Infante-Sainz-0/+1
With this commit, the copyright information of some files have been modify in order to include the work of Raul Infante-Sainz.
2019-04-27Astrometry-net installation modified to avoid two reporting stepsRaul Infante-Sainz-0/+8
Until this commit, we had a problem in the installation of Astrometry-net in Mac OS systems. During the runing of the Makefile, it stopped at the reporting stage because it tried to do a couple of steps that in Mac OS are not possible: $ cat /proc/cpuinfo $ free The reason is that `/proc/cpuinfo' and `free' doesn't exist in Mac OS. With this commit, we override these two steps. Since they are only for reporting, they are not mandatory. So, we avoid doing them by using `sed' before running the Makefile.
2019-04-27Modified netpbm installation answers for Mac OS systemsRaul Infante-Sainz-6/+4
Until now, we installed `netpbm' by setting the default paths to libraries and headers. That worked fine for GNU/Linux systems, however, in Mac OS systems we have to explicity say to `netpbm' questions what are the paths to the libraries and headers. We found this problem because we tried to run `solve-field' to obtain the astrometry of some images, and then it goes to use `pnmfile', a program that failed because it was not properly linked with `libnetpmb'. With this commit, we fix this problem by answering properly the questions in the configure step of `netpbm' and setting explicity where we want to place the libraries and headers. With this commit, we also update `netpbm' version. To do that, we have to download the last version, for which it doesn't exist a tarball. So, we built this tarball and put it in the Mohammad's webpage.
2019-04-25Added --enable-single option in FFTW installationRaul Infante-Sainz-1/+1
Until this commit, FFTW was installed by default, without the single precission option. However, when installing `sextractor' in a new system we found an error in the installation complaining about not having single precission files of FFTW. More explicity the error was: "FFTW single precision library files not found in [...]". With this commit, we fix this problem by passing the option `--enable-single' in the installation of FFTW.
2019-04-22High-level software now specified in TARGETS.mkMohammad Akhlaghi-65/+47
Until now, to specify which high-level software you want the project to contain, it was necessary to go into the `high-level.mk' Makefile that is complicated and can create bugs. With this commit, a new `reproduce/software/config/installation/TARGETS.mk' file has been created that is easily/cleanly in charge of documenting the final high-level software that must be built for the project. Also, until now, FFTW was set as a dependency of Numpy while we couldn't actually get Numpy to use it! It was just there for future reference and to justify its build rule. But now that many software won't be built and there is no problem with having rules even though a project might not use them, it has been removed.
2019-04-22Git, Metastore, cURL and Libbsd moved to basic softwareMohammad Akhlaghi-113/+1
Git and Metastore are very basic and fundamental tools for the template, so to keep things clean (let the `high-level.mk' software only represent optional software in the template), these two software (and their dependencies: `cURL' and `Libbsd') will now be built in `basic.mk'.
2019-04-22Removed ATLAS as a prerequisite of ScampRaul Infante-Sainz-4/+12
Until this commit, ATLAS was a prerequisite of Scamp. The documentation says that. However, we have been able to install Scamp without having ATLAS installed. As a consecuence, ATLAS does not have to be a prerequisite of Scamp anymore. With this commit, we remove ATLAS as prerequisite of Scamp. We also put available software in the template as optional because not all people are going to use these software.
2019-04-22SCAMP and cdsclient have been added into the projectRaul Infante-Sainz-5/+44
With this commit, we add SCAMP into the project. This program is part of the Astromatic software and the main purpose is to make the fine astrometry and the distorsion correction astronomical images. In principle, SCAMP needs ATLAS library to be able to work (that is what the documentation says). However, we had some problems with ATLAS libraries on Mac OS system, and due to that, we set the option `--enable-openblas' in the configure step. By doing that, we are be able to install SCAMP with no problems. One dependency of SCAMP is `cdsclient', it has been also added with this commit. The `cdsclient' package is a set of C and shell routines which can be built on Unix stations or PCs running Linux, which once compiled allow to query some databases located at CDS or on mirrors over the network.
2019-04-17SExtractor is now included into the projectRaul Infante-Sainz-1/+18
With this commit, we add SExtractor to be installed into the project. It is a widely used program to detect object and build catalogues from astronomical images. We had some problems when installing it because it could not link with some ATLAS libraries. But, since we have OpenBLAS installed, we can use it to override the problem with ATLAS.
2019-04-17SWarp is now into the pipelineRaul Infante-Sainz-16/+25
SWarp resamples and co-adds together FITS images using any arbitrary astrometric projection defined in the WCS standard. It is a very useful program for astronomy and that is why we added it to the pipeline. With this commit, we also cite the paper for Astrometry-net that until now was missing.
2019-04-16Typo corrections in Astrometry-net dependency rulesRaul Infante-Sainz-11/+11
Due to the copy/paste of rules from other files, in some rules there were some spaces insteand of tab. There were also a lack of semicolon in the building of CC, and bad extensions in various tarballs. With this commit, all of these typos have been fixed.
2019-04-16Astrometry-net and all its dependencies are now in the projectRaul Infante-Sainz-6/+45
With this commit, we include Astrometry-net software and one remaining dependency of it (Swig). In principle it should work as far as we tested all of this software in GNU/Linux, but we need to test also in Mac OS. As a result, this commit is just for setting all the rules and dependencies, but more tests are necessaries to ensure it works properly, specially in Mac OS systems.
2019-04-16Netpbm and its dependencies are now includedRaul Infante-Sainz-11/+50
With this commit, the installation of `netpbm' has been included. This software has a crazy dialogue installation that we override by giving to it the answers (they differs on GNU/Linux and Mac OS systems). `libxml2' also has been built (a remaining dependency of `netpbm'). In this commit, `libpng' version has been downgraded because it is required by `netpbm'. The motivation of installing `netpbm' is that it is a dependency of Astrometry-net software.
2019-04-16Cairo library and its dependency Pixman, are now includedRaul Infante-Sainz-1/+16
In this commit we add Cairo library. The motivation of including this library is because it is a dependency of Astrometry-net (which is a common software used in astronomy projects). In principle it will not be build by default by de pipeline, but we will keep the rule just in case it is needed. Pixman is another library which is a prerequisite of Cairo, it is also built with this commit.