aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/make/high-level.mk
AgeCommit message (Collapse)AuthorLines
2020-04-20Maneage instead of Template in README-hacking.md and copyright noticesMohammad Akhlaghi-11/+8
Until now, throughout Maneage we were using the old name of "Reproducible Paper Template". But we have finally decided to use Maneage, so to avoid confusion, the name has been corrected in `README-hacking.md' and also in the copyright notices. Note also that in `README-hacking.md', the main Maneage branch is now called `maneage', and the main Git remote has been changed to `https://gitlab.com/maneage/project' (this is a new GitLab Group that I have setup for all Maneage-related projects). In this repository there is only one `maneage' branch to avoid complications with the `master' branch of the projects using Maneage later.
2020-04-17Replaced name of directory under akhlaghi.org as backup serverMohammad Akhlaghi-10/+10
Until now, when a the raw tarball of some software wasn't usable, I would put it under my own webpage, or `akhlaghi.org/reproduce-software'. That same address was also used as a backup server. However, now the project has a proper name: Maneage. So I changed the directory on my own server to `akhlaghi.org/maneage-software'. With this commit, this new address has replaced the old one. But to avoid crashes in projects that haven't yet merged with the main Maneage branch, the old `reproduce-software' still works (its actually a symbolic link to the new directory now).
2020-04-17IMPORTANT: software config directly under reproduce/software/configMohammad Akhlaghi-8/+8
Until now the software configuration parameters were defined under the `reproduce/software/config/installation/' directory. This was because the configuration parameters of analysis software (for example Gnuastro's configurations) were placed under there too. But this was terribly confusing, because the run-time options of programs falls under the "analysis" phase of the project. With this commit, the Gnuastro configuration files have been moved under the new `reproduce/analysis/config/gnuastro' directory and the software configuration files are directly under `reproduce/software/config'. A clean build was done with this change and it didn't crash, but it may cause crashes in derived projects, so after merging with Maneage, please re-configure your project to see if anything has been missed. Please let us know if there is a problem.
2020-04-13Configure (TeXLive): year correction also in high-level packagesMohammad Akhlaghi-1/+1
In the previous commit, I removed the year from the basic installation of TeXLive packages, but I forgot to correct this in the high-level TeXLive packages! This is corrected with this commit.
2020-04-13Configure (TeXLive): Year of distribution no longer in directoryMohammad Akhlaghi-6/+4
It is this time of year again: TeXLive has transitioned to its 2020 release and the year is imprinted into the installation directory of TeXLive. Until now, we have had to manually change this year and it caused complications and was very annoying. With this commit, the explicit year has been removed from TeXLive's installation and we now simply put a `maneage' instead of the year. I tried this on another system and it worked nicely. Until the time that we can fully install LaTeX packages from source tarballs, this is the best thing we could do for now.
2020-04-06Astropy now depending on the Expat library to fix internal conflictRaul Infante-Sainz-3/+3
Until now, Astropy was instructed to build its own internal copy of the Expat library. However, with the recent commits before, Maneage now includes an installation of Expat and Astropy can't keep the two (its internal version and the project's version) separate, so they conflict and don't let Astropy get built. With this commit, the problem is fixed by setting the Expat library as an explicit dependency of Astropy and asking Astropy to ignore its internal copy. While doing this, I recognized that it is much easier and elegant to add steps in various stages of the `pybuild' function through hooks instead of variables. So the fifth argument of the `pybuild' function was removed and now it actually checks if hooks are defined as functions and if so, they will be called. The `pyhook_after' function was also implemented in the installation of `pybind11' (which needed it, given that the 5th argument of `pybuild' was removed) and after doing a test-build, I noticed that two lines were not ending with a `\' in `boost' (a dependency of `pybind11'). Commit written originally by Mohammad Akghlaghi
2020-04-05The build of M4 and XLSX I/O on macOS has been fixedRaul Infante-Sainz-4/+11
Until now, the M4 that was built on macOS had internal problems (as discussed in #1): it would simply print `Abort trap: 6' in the output and abort. After looking at the build of Homebrew, I noticed that they apply a patch (correct one line) to fix this problem. To be able to apply that patch on macOS systems, I had fully open up the build recipe of M4 and atleast on the testing system, it was built successfully. Also, after successfully building M4, and thus Autoconf and thus Minizip, we were able to build XLSX I/O on a macOS and found out that the internal library's full address wasn't being put in the libraries and executables. With this commit, we now use macOS's `install_name_tool' to correct the positions of the two `libxlsxio_*' libraries in all its executables. This commit was originally written by Mohammad Akhlaghi
2020-04-05Including version number of Minizip in its installation targetRaul Infante-Sainz-1/+1
Until this commit, only the word `Minizip' was written into the Minizip installation target (without the version number of Minizip). With this commit, this minor bug has been fixed by using the appropiate Make variable: `$(minizip-version)'.
2020-04-05Building Minizip 1.x instead of Minizip 2.xRaul Infante-Sainz-15/+62
Minizip is a dependency of XLSX I/O and until now, I was just using the most recent version I found (2.9.2), but XLSX I/O is written for the Minizip 1.x series, not 2.x. Somehow it didn't cause a crash on my computer!!! I think XLSX I/O's CMake is instructed to look into system directories by default when it doesn't find the directories in the given places. And because I had installed Minizip on my operating system, it did't complain. Upon trying the build on their systems, Yahya, Raul and Zahra reported a failure in the build of XLSX I/O which was due the to the problem above (we were installing the wrong version of Minizip!). With this commit, this has been fixed by installing the 1.x series of Minizip (whish is actually installed within zlib!). This commit was original done by Mohammad Akhlaghi.
2020-04-01Removed multiple tabs in MissFITS tarball definitionRaul Infante-Sainz-1/+1
With this commit, multiples tabs in the definition of MissFITS tarball have been removed. Now they are white spaces.
2020-02-24MissFITS is now added to the templateSurena Fatemi-0/+7
MissFITS is package for manipulating FITS files. I added it as my first commit to the project for educational purposes.
2020-02-16XLSX I/O installed with its two dependencies: expat and minizipMohammad Akhlaghi-0/+32
XLSX I/O is a very simple and fast program and library for reading and writing `.xls' and `.xlsx' files (mainly used by Microsoft Excel) to CSV files. It has two separate executables that can be called for an Excel file and will output a CSV plain text file that can then be used within the pipeline with more standard tools.
2020-02-11Using backup server when original download server failsMohammad Akhlaghi-1/+6
Until now, the main download script could only check one server for the given URL. However, ultimately the actual server that a file is downloaded from is irrelevant for this project: we actually check its checksum. Especially in the case of software (which are distributed over many servers), this can usually be very annoying: the servers may not properly communicate with the running system and even the 10 trials won't be enough. With this commit, the download script `reproduce/analysis/bash/download-multi-try' can take a new optional argument (a 5th argument). It assumes this argument is a space-separated list of server(s) to use as backup for the original URL. When downloading from the original URL fails, it will look into this list and try downloading the same file from each given server.
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-2/+2
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-31Configure step: compiler checks done before basic settingsMohammad Akhlaghi-1/+6
Until now, the project would first ask for the basic directories, then it would start testing the compiler. But that was problematic because the build directory can come from a previous setting (with `./project configure -e'). Also, it could confuse users to first ask for details, then suddently tell them that you don't have a working C library! We also need to store the CPATH variable in the `LOCAL.conf' because in some cases, the compiler won't work without it. With this commit, the compiler checking has been moved at the start of the configure script. Instead of putting the test program in the build directory, we now make a temporary hidden directory in the source directory and delete that directory as soon as the tests are done. In the process, I also noticed that the copyright year of the two hidden files weren't updated and corrected them.
2020-01-22Perl is now a dependency of Coreutils, LD_LIBRARY_PATH in basic.mkMohammad Akhlaghi-1/+1
Until now Perl was built after Coreutils, but I recently noticed that Coreutils actually uses Perl while creating its manpages. So it is now built before Coreutils. Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils can't build its man page for `md5sum'. The problem was found to be due to the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our installed library path in `basic.mk'. Therefore, it would crash because the server had an older version of OpenSSL than the one that the template's Coreutils was built with. In the meantime (while addressing the issues above, because we only had one thread on the AWS server) I also noticed a few programs that were using a summarize compilation command (that just prints `CC xxx.c' instead of the whole command) so I fixed them by adding `V=1'. This bug was found by Idafen Santana Pérez.
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-7/+7
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19LIBRARY_PATH is set accordingly based on the hostMohammad Akhlaghi-0/+7
Until now, GCC wouldn't build properly on Debian-based operating systems because `ld' needed to link with several necessary C library features like `crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to add the directory containing them to `LIBRARY_PATH'. In the previous commit, I actually searched for these files, but while testing on another system, I noticed that it can be problematic (other architectures may exist). With this commit, we are actually finding the build architecture of the running GCC (which is the same as the `ld') and using that to fix a fixed directory to `LIBRARY_PATH'.
2020-01-18TeXLive version printing updated to include revisionMohammad Akhlaghi-7/+26
Until now, when find the versions of the TeXLive packages, we would assume that `cat-date' is always present (because some packages don't have a version!). However, apparently an update has been made in the TeXLive Manager (`tlmgr') and `cat-date' is no longer present! As a result, none of the TeXLive packages were being printed. With this commit, it now assumes that `revision' is always present for every package, but it also attempts to read `cat-date' (for backwards compatability). When `cat-version' isn't present, it will try printing `revision' and if that is also not present, it will print the date.
2020-01-13Adding X11 libraries path to LDFLAGS in Ghostscript installationRaul Infante-Sainz-1/+4
Until this commit, the checking of X11 installation done to ensure that it is already available in the host system was crashing in macOS systems. The reason is that the place of the X11 libraries use to be `/opt/X11/lib' in macOS systems. With this commit, this issue has been fixed by adding this directory to the LDFLAGS.
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-2/+2
Now that its 2020, its necessary to include this year in the copyright statements.
2019-12-18Fixing Libxml2 checksum variable nameRaul Infante-Sainz-1/+1
Until this commit, the number `2' was missing in the checksum variable name of that library. It was `libxml-checksum' but it should be `libxml2-checksum'. With this commit, this issue has been fixed.
2019-12-14R explicitly built without support for OpenMPMohammad Akhlaghi-1/+2
OpenMP takes a LONG TIME to build, so to keep things reproducible we are explicitly disabling OpenMP, if a user needs OpenMP, its trivial to just add it as a prerequisite of R. The problem is that in some scenarios (based on other dependencies and when they were built in the build directory), OpenMP may be present when R is being installed and in other it may not. We don't want the result to be different between the two builds.
2019-12-14Core R package is now in templateMohammad Akhlaghi-0/+14
With this commit, we now have the core R interpretter within the template. We should later include instructions to install R packages (possibly in a separate top-level Makefile like Python).
2019-12-04Checking for necessary X11 libraries before building GhostscriptMohammad Akhlaghi-1/+35
Until now, Ghostscript was using some host system's X11 libraries during its build (and later at run-time). We should ideally install all these necessary libraries within the template (Task #15481). But right now we are too busy. As a temporary work-around we try building a small dummy program that links with some of those libraries before attempting to built Ghostscript. If it fails, then a notice is printed with the cause and explaining a temporary solution is suggested: how to install those libraries on the system when you have root access.
2019-11-08Matplotlib problem on macOS system with GNU GCC fixedMohammad Akhlaghi-1/+1
Until now when building Matplotlib on macOS systems, we were using the default C compiler. However, while Yahya Sefidbakht (previously) and Mahdieh Nabavi (now) were trying to build the template, on their macOS using the GNU Compiler Collection (GCC), we found that Matplotlib needs special macOS headers that GCC doesn't recognize. With this commit, when Matplotlib is being built on macOS systems, it uses `clang' and this fixed the problem (so far checked on Mahdieh's machine).
2019-11-06GNU Project Debugger (GDB) and Pexpect can now be builtMohammad Akhlaghi-0/+7
These two packages are necessary to build the GNU C Library.
2019-11-03Corrected build of log4cxx by re-building the tarball usedMohammad Akhlaghi-13/+29
When building the log4cxx tarball from its Git history, I noticed that files with very long names are not packaged by tar (because by default Automake uses the ancient v7 tar format that only supports file names less than 99 characters). So I build the tarball with the `tar-ustar' option to Automake (by modifying the log4cxx source) and the resulting tarball was able to compile and run successfully. This has been described above the rule to build log4cxx and I also sent an email to their developing mailing list to inform them of this problem. If they address it, I will remove the note on the necessary corrections.
2019-10-31Minor corrections in distribution and autoconf prerequisite of automakeMohammad Akhlaghi-4/+32
Some minor corrections were made in the template: - When making the distribution, `.swp' files (created by Vim) are also removed. - Autoconf is set as a prerequisite of Automake I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball needs some patches, so I have just left it half done until someone actually needs it and we apply the patch.
2019-10-30Tarballs set as order-only prerequisites, not needed for build callsMohammad Akhlaghi-144/+156
Until now, the tarballs were the first normal prerequisite of the software. As a result if their date changed, the whole software would be re-built. However, for tarballs specifically, we actually check their contents with a checksum, so their date is irrelevant (if its newer than the built program, but has the same checksum, there is no need to re-build the software). Also, calling the tarball name as an argument to the building process (for example `gbuild') was redundant. It is now automatically found from the list of order-only prerequisites within `gbuild' and `cbuild' (similar to how it was previously found in the `pybuild' for Python building). A `README.md' file has also been placed in `reproduce/software/make' to help describe the shared properties of the software building Makefiles. This will hopefully grow much larger in the future.
2019-10-28HEALPix, Autoconf and Automake now in libraryMohammad Akhlaghi-1/+62
The tarball of HEALPix includes multiple languages and doesn't include the ready-to-run GNU Build System by default, we actually have to build the `./configure' script for the C/C++ libraries. So it was necessary to also include GNU Autoconf and GNU Automake as prerequisites of HEALPix. However, the official GNU Autoconf tarball (dating from 2012) doesn't build on modern systems, so I just cloned it from its source and bootstrapped it and built its modern tarball which we are using here.
2019-10-24APR, APR Utility, Boost and Eigen are now availableMohammad Akhlaghi-2/+61
As part of an effort to bring in all the dependencies of the LSST Science pipeline (which includes the last commit), these software are now available in the template.
2019-10-24GNU Bison, GNU Help2man and Flex are now availableMohammad Akhlaghi-3/+20
With this commit these three software packages are now installable with this template.
2019-10-01Minor corrections in configure and prepare phaseMohammad Akhlaghi-1/+1
Since ImageMagick can take long to build, we are now building it in parallel. Also, the part where we replace an `_' with `\_' in the software version at the end of the configure script was removed. It is more clear/readable that the actual rule that includes such a name deals with the underline (as is the case for `sip_tpv' which already dealt with it). Finally, I noticed that the checks at the start of `top-prepare' were missing new-lines. I had forgot that the Make single-shell variable isn't activated in this stage yet.
2019-10-01New versions for ImageMagick, Python, Numpy Scipy and MatplotlibMohammad Akhlaghi-4/+1
It was some time since these three software were not updated! With this commit the template now uses the most recent stable release of these packages. Also, the hosting server for ImageMagick was moved to my own webpage because unfortunately ImageMagick removes its tarballs from its own version.
2019-09-28Corrected name of SExtractor in its final targetMohammad Akhlaghi-1/+1
Until now we were calling it `Sextractor', but the official way of writing it is `SExtractor'. With this commit, this has been corrected.
2019-09-06Updated versions of several softwareMohammad Akhlaghi-1/+4
New versions of astropy, bash, cmake, curl, findutils, gawk, gcc, ghostscript, git, make, gsl had recently come so they are updated with this commit. About GNU Findutils and GNU Make: I was bootstrapping (building the tarball of) these two separately separately because their standard tarball release had problems on some systems. Both have been updated now so I am no longer using my own webpage as their main URL. A special note about GNU Make. I just noticed that during bootstrapping, GNU Make would use the fixed version string of `4.2.90' for any commit!!! But fortunately they have officially released their 4.2.90 version, so we are safely using their own webpage. The only difference is the compression format. My old bootstrapped build was `tar.lz', but the standard release is `tar.gz'. Also, all the basic programs (installed in `.local/bin') in `basic.mk' are now existance-only dependencies (after a `|'). Because later programs just use them at a very basic level, so there is no need to rebuild everything when Bash gets updated for example.
2019-08-22OpenMPI environment variable used to disable need for OpenSSHMohammad Akhlaghi-2/+6
Until now, OpenMPI would complain about not having `ssh' or `rsh' as a remote shell feature. However, such features should not be necessary in a reproducible scenario and they also have major security issues. With this commit, we are now using OpenMPI's `OMPI_MCA_plm_rsh_agent' environment variable to disable any remote shell dependency for it (as suggested by Boud). Therefore, any dependency for OpenSSH has been removed. But I thought to keep the build instructions incase it may be useful under some un-foreseen scenario. However, to discourage people from building it, a notice was added ontop of the build instructions. This bug was found, tested and solved thanks to Roberto Baena Gallé and Boud Roukema. This fixes bug #56724.
2019-08-22Minor typo corrected in commentsMohammad Akhlaghi-3/+3
Until now we have followed the convention of using space characters before comment lines in recepies, not tabs. This has been corrected in one case.
2019-08-22Adding fpack and funpack in the installation of CFITSIORaul Infante-Sainz-2/+4
Until this commit, `fpack' and `funpack' were not installed by default with the installation of CFITSIO. It is necessary to explicity do a `make fpack' and `make funpack' to have them installed. With this commit, these two programs have been added.
2019-08-21Using libgit2 implementation of libconv with the option -DUSE_ICONV=OFFRaul Infante-Sainz-1/+5
When trying to install `libgit2' on my macOS system, it complains about not finding `_iconv*' functions! But apparently `libgit2' has its own implementation of `libiconv' that it uses if it can't find `libiconv' on macOS. So, the solution to this problem was to add `-DUSE_ICONV=OFF' to the configuration options of `libgit2'. I reported this issue that now is fixed thanks to the help of Mohammad Akhlaghi.
2019-08-16Libtiff built without JBIG-KIT supportMohammad Akhlaghi-2/+5
Until now, we weren't explicitly telling libtiff to ignore JBIG-KIT. So on some systems, it would try to link withit and would thus fail. With this commit, we have disabled JBIG-KIT support for libtiff. I tried to import it into the template, but it doesn't have use the standard GNU Build system. Maybe later we can add a set of build rules for it, I don't have time now. Also, this problem with libtiff occurred while building Ghostscript. But was fixed after adding this option to libtiff. So libtiff was added as a dependency of Ghostscript. This bug was reported by Sheeraz Ahmad.
2019-08-16OpenSSH privsep-path now set within project's build directoryMohammad Akhlaghi-1/+2
Until now we weren't setting OpenSSH's `privsep-path' configure option. As a result, it would try to use its default (`/var/empty'). Therefore, when the host doesn't have `/var/empty', OpenSSH would crash because of not having permissions to create this directory. With this commit, we are now using OpenSSH's `--with-privsep-path' configure-time option to explicitly use a directory with the project's build directory. This bug was found by Sheeraz Ahmad and Amina Aahil.
2019-08-15WCSLIB 6.4 doesn't need explicit link with CFITSIOMohammad Akhlaghi-15/+3
This was a bug in WCSLIB 6.3 that has been fixed in WCSLIB 6.4. From WCSLIB's changelog: "The rule change to the Fortran makefile in v6.3 to add getwcstab_f.o to the sharable library causes it to depend on CFITSIO to resolve fits_get_wcstab(). Hence backed out of that change.". The actual error was like this: Undefined symbols for architecture x86_64: "_fits_read_wcstab", referenced from: _ftwcst_ in getwcstab_f.o "_gFitsFiles", referenced from: _ftwcst_ in getwcstab_f.o ld: symbol(s) not found for architecture x86_64
2019-08-08libnsl, libtirpc and rpcsvc installed as non-used biber dependenciesMohammad Akhlaghi-6/+43
These three libraries are dependencies of Biber, so we will need them later, but since we don't build biber from source now, we can't control what library it links with. With this commit, we have just added their versions, checksum, download URL and build rule incase they are useful in other software. Later, when we build Biber (and Texlive in general) from source, we'll be able to use these.
2019-08-08Gnuastro's build: removing the `make check' stepMohammad Akhlaghi-2/+1
Until now, unlike most other programs, in Gnuastro we would run `make check', but this causes a crash on some systems, because of its BuildProgram test: a linking error because the library isn't installed yet. Which is natural in cases like this (and must be corrected in future Gnuastro releases). With this commit, the checks of Gnuastro have been removed.
2019-08-06OpenSSH necessary to use OpenMPIMohammad Akhlaghi-2/+13
Until now, OpenMPI was being installed without any dependency. This was fine because it would indeed build. But the moment you tried loading something that depends on it (for example `mpi4py' through `astropy'), you would get an error complaining that SSH isn't present. With this commit, the pipeline now also installs OpenSSH to solve this problem.
2019-08-05WCSLIB needs -lcfitsio on some systemsMohammad Akhlaghi-1/+13
Until now, we were relying on WCSLIB's internal checking and linking with CFITSIO. But on one macOS system (not others that had no problem!), we noticed that it complains with undefined symbol linking errors to CFITSIO libraries. With this commit, as a fast/ugly solution, we are explicity adding `-lcfitsio' to WCSLIB's `LIBS' variable so all binaries are linked with it automatically. We'll be in touch with the WCSLIB author to see if a better solution can be found.
2019-08-05Backup repository set to http://akhlaghi.org/reproduce-softwareMohammad Akhlaghi-4/+4
Some of the software tarballs are directly available on their webpage (for example due to build problems on systems, where we had to clone the software and build its tarball ourselves until the next release). Until now, only for these software, these tarballs were hosted on `http://akhlaghi.org/src'. But now, I have moved a clone of the software backup repository (including all its software) to `http://akhlaghi.org/reproduce-software'. To be more clear and have a single place for the backup software, the URL of those software has also been updated in the project source.
2019-08-01Bash startup script for every recipeMohammad Akhlaghi-0/+4
Until now the only way to define the environment of the Make recipes was through the exported Make variables (mostly in `initialize.mk' for the analysis steps for example). However, there is only so much you can do with environment variables! In some situations you want slightly more complicated environment control, like setting an alias or running of scripts (things that are commonly done in the `~/.bashrc' file of users to configure their interactive, non-login shells). With this commit, a `reproduce/software/bash/bashrc.sh' has been defined for this job (which is currently empty!). Every major Make step of the project adds this file as the `BASH_ENV' environment variable, so the shell that is created to execute a recipe first executes this file, then the recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment variable that enables users to limit their envirnoment setup based on the condition it is being setup (in particular in the early phase of `basic.mk', where the user can't make any assumption about the programs and has to write a portable shell script).