aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/config
AgeCommit message (Collapse)AuthorLines
2020-01-31Configure step: compiler checks done before basic settingsMohammad Akhlaghi-0/+1
Until now, the project would first ask for the basic directories, then it would start testing the compiler. But that was problematic because the build directory can come from a previous setting (with `./project configure -e'). Also, it could confuse users to first ask for details, then suddently tell them that you don't have a working C library! We also need to store the CPATH variable in the `LOCAL.conf' because in some cases, the compiler won't work without it. With this commit, the compiler checking has been moved at the start of the configure script. Instead of putting the test program in the build directory, we now make a temporary hidden directory in the source directory and delete that directory as soon as the tests are done. In the process, I also noticed that the copyright year of the two hidden files weren't updated and corrected them.
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-9/+9
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19GNU Make updated to version 4.3Mohammad Akhlaghi-2/+2
GNU Make 4.3 was just announced, so I have updated it here is well. This was important because until now the installable version was in alpha-mode (4.2.90), now its a stable version.
2020-01-18Added mweights as a TeXLive packageMohammad Akhlaghi-1/+2
After a new rebuild of the project, I noticed that we now need to also build the `mweights' package.
2020-01-13Updated verion of Astropy to 4.0 the most recent oneRaul Infante-Sainz-2/+2
Newer versions of Astropy package has been released. With this commit, it has been updated. It has been increased from v3.2.1 to v4.0
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-14/+14
Now that its 2020, its necessary to include this year in the copyright statements.
2019-12-14Core R package is now in templateMohammad Akhlaghi-0/+2
With this commit, we now have the core R interpretter within the template. We should later include instructions to install R packages (possibly in a separate top-level Makefile like Python).
2019-12-13Necessary TeXLive packages installed: pdfescape letltxmacro bitsetMohammad Akhlaghi-1/+1
Recent builds of the template need these three packages to build the PDF. This was reported by Hamed Altafi.
2019-12-04Added pdftexmcds to the TeXlive packages to installMohammad Akhlaghi-1/+1
Until now we weren't including this package, but Alberto Madrigal recently reported that his build of the template failed because it needed it. With this commit, it is now being installed with the template.
2019-12-04Updated verions of Ghostscript and GnuastroMohammad Akhlaghi-4/+4
Newer versions of these packages have recently come out with major improvements, so they have been updated in the template.
2019-11-06GNU Project Debugger (GDB) and Pexpect can now be builtMohammad Akhlaghi-0/+4
These two packages are necessary to build the GNU C Library.
2019-11-06Perl is now built as a basic dependency of the templateMohammad Akhlaghi-0/+2
Perl is necessary to build Texinfo and later to build LaTeX. Until now we were just using the host operating system's installation of Perl, but in some instances that Perl can be too old and not suppor the features necessary. With this commit, Perl is now built from source during the basic installation step of the template. This was reported by Idafen Santana Pérez, after trying the pipeline on an Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
2019-11-03Corrected build of log4cxx by re-building the tarball usedMohammad Akhlaghi-2/+2
When building the log4cxx tarball from its Git history, I noticed that files with very long names are not packaged by tar (because by default Automake uses the ancient v7 tar format that only supports file names less than 99 characters). So I build the tarball with the `tar-ustar' option to Automake (by modifying the log4cxx source) and the resulting tarball was able to compile and run successfully. This has been described above the rule to build log4cxx and I also sent an email to their developing mailing list to inform them of this problem. If they address it, I will remove the note on the necessary corrections.
2019-10-31Minor corrections in distribution and autoconf prerequisite of automakeMohammad Akhlaghi-0/+2
Some minor corrections were made in the template: - When making the distribution, `.swp' files (created by Vim) are also removed. - Autoconf is set as a prerequisite of Automake I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball needs some patches, so I have just left it half done until someone actually needs it and we apply the patch.
2019-10-30Tarballs set as order-only prerequisites, not needed for build callsMohammad Akhlaghi-1/+1
Until now, the tarballs were the first normal prerequisite of the software. As a result if their date changed, the whole software would be re-built. However, for tarballs specifically, we actually check their contents with a checksum, so their date is irrelevant (if its newer than the built program, but has the same checksum, there is no need to re-build the software). Also, calling the tarball name as an argument to the building process (for example `gbuild') was redundant. It is now automatically found from the list of order-only prerequisites within `gbuild' and `cbuild' (similar to how it was previously found in the `pybuild' for Python building). A `README.md' file has also been placed in `reproduce/software/make' to help describe the shared properties of the software building Makefiles. This will hopefully grow much larger in the future.
2019-10-29lmfit and its dependencies added, Python tarballs as prerequisiteMohammad Akhlaghi-5/+15
Python's `lmfit' module and all its major dependencies (`asteval', `corner', `emcee' and `uncertainties') have been included in the template. While doing this I noticed that if the tarballs are the last prerequisite of each software building rule, then when building in parallel, the template will immediately start building packages as soon as the first one is downloaded. Not like the current way that it will attempt to download several, then start building. For now, this has been implemented in the Python build rules for all the modules and we'll later do the same for the other programs and libraries. This also motivated a simplification of the `pybuild' function: it now internally looks into the prerequisites and selects the tarball from the prerequisite that is in the tarballs directory. This isn't a problem for the build, but I just don't understand why Python can't recognize the version of `emcee', Python reads the version of `emcee' as `0.0.0'! But it doesn't cause any crash in the build, so for now its fine.
2019-10-28HEALPix, Autoconf and Automake now in libraryMohammad Akhlaghi-0/+8
The tarball of HEALPix includes multiple languages and doesn't include the ready-to-run GNU Build System by default, we actually have to build the `./configure' script for the C/C++ libraries. So it was necessary to also include GNU Autoconf and GNU Automake as prerequisites of HEALPix. However, the official GNU Autoconf tarball (dating from 2012) doesn't build on modern systems, so I just cloned it from its source and bootstrapped it and built its modern tarball which we are using here.
2019-10-24Several more LSST pipeline dependencies are addedMohammad Akhlaghi-0/+18
The following software are added with this commit: eigency, esutil, flake8, future, galsim, lsstdesccoord, pybind11 and pyflakes.
2019-10-24APR, APR Utility, Boost and Eigen are now availableMohammad Akhlaghi-0/+8
As part of an effort to bring in all the dependencies of the LSST Science pipeline (which includes the last commit), these software are now available in the template.
2019-10-24GNU Bison, GNU Help2man and Flex are now availableMohammad Akhlaghi-0/+6
With this commit these three software packages are now installable with this template.
2019-10-10Including trimspaces from TeXLiveMohammad Akhlaghi-1/+2
Carlos Morales Socorro reported that his LaTeX build was missing the `trimspaces' package, so it is now included in the pipeline.
2019-10-01New versions for ImageMagick, Python, Numpy Scipy and MatplotlibMohammad Akhlaghi-10/+10
It was some time since these three software were not updated! With this commit the template now uses the most recent stable release of these packages. Also, the hosting server for ImageMagick was moved to my own webpage because unfortunately ImageMagick removes its tarballs from its own version.
2019-09-06Updated versions of several softwareMohammad Akhlaghi-21/+21
New versions of astropy, bash, cmake, curl, findutils, gawk, gcc, ghostscript, git, make, gsl had recently come so they are updated with this commit. About GNU Findutils and GNU Make: I was bootstrapping (building the tarball of) these two separately separately because their standard tarball release had problems on some systems. Both have been updated now so I am no longer using my own webpage as their main URL. A special note about GNU Make. I just noticed that during bootstrapping, GNU Make would use the fixed version string of `4.2.90' for any commit!!! But fortunately they have officially released their 4.2.90 version, so we are safely using their own webpage. The only difference is the compression format. My old bootstrapped build was `tar.lz', but the standard release is `tar.gz'. Also, all the basic programs (installed in `.local/bin') in `basic.mk' are now existance-only dependencies (after a `|'). Because later programs just use them at a very basic level, so there is no need to rebuild everything when Bash gets updated for example.
2019-08-22Fixed typo in beautifulsoup4 Python package checksum nameRaul Infante-Sainz-2/+2
Until this commit, the name of the variable for `beautifulsoup4' checksum was wrong, and because of that, it was not able to install it. With this commit, `beautifulsoup-checksum' has been replaced for `beautifulsoup4-checksum' in the `reproduce/software/make/python.mk' Makefile, and the problem has been fixed. This was not noticed previously because this Python package is only installed when some high level programs are requested to be installed. With this commit the version of `imagemagick' program has been also updated because the previous version is not available in the official website anymore.
2019-08-21Gnuastro's configuration file updated for version 0.10Mohammad Akhlaghi-2/+8
Recently the version of Gnuastro in the template was updated to version 0.10. However, I had forgot to update the `gnuastro.conf' file to fit with the necessary new features of this version. The new general Gnuastro configuration file is now added instead. This bug was reported by Deepak.
2019-08-15WCSLIB 6.4 doesn't need explicit link with CFITSIOMohammad Akhlaghi-2/+2
This was a bug in WCSLIB 6.3 that has been fixed in WCSLIB 6.4. From WCSLIB's changelog: "The rule change to the Fortran makefile in v6.3 to add getwcstab_f.o to the sharable library causes it to depend on CFITSIO to resolve fits_get_wcstab(). Hence backed out of that change.". The actual error was like this: Undefined symbols for architecture x86_64: "_fits_read_wcstab", referenced from: _ftwcst_ in getwcstab_f.o "_gFitsFiles", referenced from: _ftwcst_ in getwcstab_f.o ld: symbol(s) not found for architecture x86_64
2019-08-08libnsl, libtirpc and rpcsvc installed as non-used biber dependenciesMohammad Akhlaghi-0/+6
These three libraries are dependencies of Biber, so we will need them later, but since we don't build biber from source now, we can't control what library it links with. With this commit, we have just added their versions, checksum, download URL and build rule incase they are useful in other software. Later, when we build Biber (and Texlive in general) from source, we'll be able to use these.
2019-08-06OpenSSH necessary to use OpenMPIMohammad Akhlaghi-0/+2
Until now, OpenMPI was being installed without any dependency. This was fine because it would indeed build. But the moment you tried loading something that depends on it (for example `mpi4py' through `astropy'), you would get an error complaining that SSH isn't present. With this commit, the pipeline now also installs OpenSSH to solve this problem.
2019-08-05Gnuastro updated to version 0.10Mohammad Akhlaghi-2/+2
A new version of Gnuastro was recently released with many improvments and bug fixes, so it is updated here too.
2019-08-01Updated version of mpi4py doesn't conflict with OpenMPIMohammad Akhlaghi-2/+2
Until now, in version 3.0.1, mpi4py couldn't be built with the most recent version of OpenMPI. However, after trying the next version (3.0.2), I noticed that it builds successfully without a problem.
2019-07-29Checking software tarball checksums before building softwareMohammad Akhlaghi-3/+164
Until now, there was no check on the integrity of the contents of the downloaded/copied software tarballs, we only relied on the tarball name. This could be bad for reproducibility and security, for example on one server the name of a tarball may be the same but with different content. With this commit, the SHA512 checksums of all the software are stored in the newly created `checksums.mk' (similar to how the versions are stored in the `versions.mk'). The resulting variable is then defined for each software and after downloading/copying the file we check to see if the new tarball has the same checksum as the stored value. If it doesn't the script will crash with an error, informing the user of the problem. The only limitation now is a bootstrapping problem: if the host system doesn't already an `sha512sum' executable, we will not do any checksum verification until we install our `sha512sum' (as part of GNU Coreutils). All the tarballs downloaded after GNU Coreutils are built will have their checksums validated. By default almost all GNU/Linux systems will have a usable `sha512sum' (its part of GNU Coreutils after all for a long time: from the Coreutils Changelog file atleast since 2013). This completes task #15347.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-2/+2
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-07-26PatchELF built statically and updatedMohammad Akhlaghi-1/+1
Until now, like all other software, PatchELF would install with dynamic linking. However, PatchELF links with `libstdc++' so on one system, I noticed that PatchELF gives a segmentation fault and corrupts `libstdc++' while correcting its RPATH (after installing GCC). The solution is to build PatchELF statically. With this commit, we force PatchELF to be built statically (it only installs on GNU/Linux systems anyway, so there is no problem with static linking on macOS). This solved the problem on that system. While looking at its documentation, I also noticed that a new version of PatchELF has been released after almost three years, so it has been updated in the template also. This fixes bug #56673.
2019-07-25Updated version of WCSLIB to 6.3, previously 6.2Mohammad Akhlaghi-1/+1
Some bugs have been fixed in the new version of WCSLIB, so it has been updated in the template.
2019-07-24libgit2 updated to version 0.28.2, from 0.26.0Mohammad Akhlaghi-1/+1
More than two releases and bug fixes have been made to libgit2. So we are now using a more recent version in the template.
2019-07-07Texinfo installed as a dependency of M4Mohammad Akhlaghi-0/+1
While testing on a system with no Texinfo, we noticed that M4 depends on Texinfo. To fix this problem, with this commit, it is now included in the pipeline. While doing a clean build, a few minor issues were also found and corrected in the other rules.
2019-06-29Added citation for TIDES, sorted progs alphabeticallyMohammad Akhlaghi-2/+2
While reviewing Prasenjit's commits, I noticed that we had forgot to add the citation for TIDES, also to make things clear, the program/library build rules are now sorted alphabetically. Finally, I noticed that after building the TiKZ PDF figures, it is crashing (like on Prasenjit's computer). After looking around, I noticed its because we were setting the of the `TEXINPUTS' environment variable to be the installed TeX Live directory (which was ultimately redundant because by default TeX will look into where it was installed). The important thing is just that we remove any possible value the host system has, not to set new directories.
2019-06-28tides library addedPrasenjit Saha-1/+2
TIDES is an ODE integrator with multiple-precision arithmetic.
2019-06-28Corrections to basic buildPrasenjit Saha-2/+2
Several corrections were necessary in the basic build: 1) the version of GCC on some systems includes an `_' which would cause a crash when building the PDF. 2) libcharset had to be manually added to the Git build.
2019-06-28Libiconv built as a dependency of GitMohammad Akhlaghi-1/+2
Prasenjit Saha reported that while building on a macOS, Git complained about not finding `libiconv' functions. Therefore with this commit, `libiconv' is also built as a dependency of Git. Also, since the build of Binutils (and the subsequent GCC) doesn't depend on the exact versions of the other basic software, they are set as existance-only prerequsites (ignoring their timestamps). This is done to avoid re-building when it is not needed.
2019-06-12Imported recent work on Imfit, minor conflicts fixedMohammad Akhlaghi-1/+3
The recent work by Raul to import Imfit is now merged into the master branch. Only a few minor conflicts came up due to the update with CFITSIO (that Raul also done in parallel on this branch).
2019-06-10TeXlive: allows for updating itself and removes old version fileMohammad Akhlaghi-1/+1
Until now, TeXlive would only attempt to download/update LaTeX packages, but after some time, it will be necessary for it to update itself. When this happens, it complains about running it with `update --self'. To fix this problem, before installing/updating the LaTeX packages, we will first update tlmgr in any case. Also, we weren't removing the TeXlive package version file before appending the values to it. So it was just repeating the packages every time it was updated. With this commit, it is being removed on every new build. Finally, the version of Git was updated to its most recent version.
2019-05-31Modified Imfit installation to not consider default and host pathsRaul Infante-Sainz-1/+1
Until this commit, 'imfit' was installed using proper flags in order to use our own libraries. However, it looks first of all to the default system. As a consecuence, `imfit' program was linked to the host system libraries, even when we told to use our own libraries!! With this commit, the installation of `imfit' has been modified in order to remove the default paths already set in the `SConstruct' script. By doing this, only our paths are take into account for compiling the program. It has been tested on Mac OS laptop and it works fine. Test on GNU/Linux systems remain before merge into the main project branch.
2019-05-31Fixing rpath problem in CFITSIO installationRaul Infante-Sainz-1/+1
`cfitsio' was recently updated and it cames up with a problem when trying to build shared libraries (on Mac OS system laptop). This is because the `configure' script for building `cfitsio' include by default `rpath'. With this commit, this issue is fixed by modifying the installation of `cfitsio'. Using `sed', each ocurrence of `@rpath' in `configure' is replaced by `$(ildir)'. Once it has been done, the installation of `cfitsio' goes as normal. In this commit, the version of the `cfitsio' has been also updated to the most recent one.
2019-05-27Software: CFITSIO 3.47, implemented many common standardsMohammad Akhlaghi-1/+1
Until now, CFITSIO had several peculiar features which needed special attention: 1) Its tarball didn't have a `.' in the version name and would sometimes have extra `0's. 2) When unpacked it was just `cfitsio'. 3) It hard coded the directory to search for cURL library. All of these forced us to do a lot of extra steps for CFITSIO. So I got in touch with them and fortunately they were very open to correcting these issues and with version 3.47 all such issues have been addressed in the tarball. Thanks to these changes, with this commit, CFITSIO's build rule is now very similar to many other programs. I also noticed that it has an optional `--with-bz2' feature to enable unpacking `.fits.bz2' files. So this feature is also added.
2019-05-27Added Imfit into the projectRaul Infante-Sainz-1/+2
With this commit, `imfit' has been included into the project. As described in the official webpage (http://www.mpe.mpg.de/~erwin/code/imfit/): Imfit is a program for fitting astronomical images -- especially images of galaxies, though it can in principle be used for fitting other sources. This program uses the `scons' program to build itself (not the traditional GNU `configure' and `make' system). I have tested it by using an example that can be found in the `examples' directory of the decompressed tarball. It works fine without any error on my Mac OS system.
2019-05-27Adding SCons into the projectRaul Infante-Sainz-1/+2
With this commit, SCons has been included into the project. As in the official webpage of this program says: SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software. The motivation for including this software into the project is because there are some software that use this system for the installation. In particular, the necessity of installing it comes from Imfit. This is an astronomical software that uses this system so that is why it has been included into the project. In principle it is simple, and I have tested it on Mac OS sytem without any problem.
2019-05-24Added PyYAML Python package into the projectRaul Infante-Sainz-0/+1
With this commit, PyYAML Python package has been added into the project. It is widely used in the Python community and the goal is to have human readable configuration files. As in the web page (https://pypi.org/project/PyYAML/) says: YAML is a data serialization format designed for human readability and interaction with scripting languages. PyYAML is a YAML parser and emitter for Python.
2019-05-24Added LibYAML library into the projectRaul Infante-Sainz-0/+1
With this commit, LibYAML library has been added. This library is a prerequisite of the Python package PyYAML, which is widely used in the Python community when configuration scripts are needed.
2019-05-21ImageMagick is now included into the projectRaul Infante-Sainz-0/+1
With this commit, ImageMagick software has been added into the project. This software is useful to deal and treat images from the command line. Since it is widely used and a lot of other programs rely on it, it is worth to have it into the project.