aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/make/high-level.mk
AgeCommit message (Collapse)AuthorLines
2020-08-02OpenMPI build with slurm compatibilityBoud Roukema-1/+4
Prior to this commit, compilation of OpenMPI used the default OpenMPI choices of deciding which libraries should be used in relating to a job scheduler [1] (such as Slurm [2]). Given that the user on a multi-user cluster has to accept the sysadmin's choice of a job scheduler, the question of whether to (1) link with OpenMPI's own libraries (and increase the reproducibility of the science project) or rather (2) link with the sysadmin managed libraries (more likely to be compatible with the host's job scheduler), is an open question of which the best strategy for reproducibility needs to be debated and studied. In this commit, strategy (1) is adopted. The options '--withpmix=internal' and '--with-hwloc=internal' are added to the configure command. The working assumption is that the Maneage version of OpenMPI is likely to be modern enough to be compatible with the native job scheduler such as Slurm. Compilation without any 'pmix' option gave a fail in at least one case; it appears that an external pmix library was sought by the configure script. As of OpenMPI 4.0.1, the internal libevent library is used by default, so there appears to be no option to force it to be chosen internally. This commit also includes the option '--without-verbs'. This option removes a library related to "infiniband", "verbs", "openib" and "BTL"; this library appears to be deprecated. See [3], [4] for discussion. Please add feedback and discussion to the Maneage task about openmpi linking strategies (1) (internal) and (2) (external) at Savannah [5]. [1] https://en.wikipedia.org/wiki/Job_scheduler#Batch_queuing_for_HPC_clusters [2] https://en.wikipedia.org/wiki/Slurm_Workload_Manager - To avoid a name clash, 'slurm-wlm' is the metapackage in Debian for the client commands, the compute node daemon, and the central node daemon. An unrelated package 'slurm' also exists. [3] https://www-lb.open-mpi.org/faq/?category=openfabrics#ofa-device-error [4] https://www-lb.open-mpi.org/faq/?category=building [5] https://savannah.nongnu.org/task/index.php?15737
2020-07-05Configure script prefers clang for macOS systemsMohammad Akhlaghi-0/+1
In the previous commit (Commit 1bc00c9: Only using clang in macOS systems that also have GCC) we set the used C compiler for high-level programs to be 'clang' on macOS systems. But I forgot to do the same kind of change in the configure script (to prefer 'clang' when we are testing for a C compiler on the host). With this commit, the compiler checking phases of the configure script have been improved, so on macOS systems, we now first search for 'clang', then search for 'gcc'. While doing this, I also noticed that the 'rpath' checking command was done before we actually define 'instdir'!!! So in effect, the 'rpath' directory was being set to '/lib'! So with this commit, this test has been taken to after defining 'instdir'.
2020-07-01Minor typo corrected in referencing LibidnRaul Infante-Sainz-1/+1
Until this commit, once Libidn was installed, insted of its own name and version, the name and version of Libjpeg were saved (in the target if Libidn). This robably come from a copy/paste of the rule. With this commit, this minor bug has been corrected. I also added my name as an author of `reproduce/software/make/xorg.mk' Makefile since I added some code there.
2020-06-30Proper deletion of util-linux source after successfully building itMohammad Akhlaghi-1/+3
After recently adding util-linux to Maneage build-tree, we had forgot to delete the unpacked and built source directory after it was installed! This has been corrected with this commit.
2020-06-30Core Xorg libraries necessary for Ghostscript now includedMohammad Akhlaghi-45/+141
Until now, in order to build Ghostscript, the project used the host's Xorg libraries. This was because we hadn't yet added the necessary build rules for them. With this commit, the instructions to build the necessary Xorg libraries for Ghostscript have also been added. Also, the shared Ghostscript library has been built with this commit and two sets of standard fonts are also included, setting us on the path to build TeXLive from source later. This task was done with the help and support of Raul Infante-Sainz.
2020-06-27IMPORTANT: many improvements to low-level software building phaseMohammad Akhlaghi-791/+778
POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause conflicts to your project if you have changed the software building Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and 'python.mk'). If your project has only added analysis, it shouldn't be affected. This is a large commit, involving a long series of corrections in a differnt branch which is now finally being merged into the core Maneage branch. All changes were related and came up naturally as the low-level infrastructure was improved. So separating them in the end for the final merge would have been very time consuming and we are merging them as one commit. In general, the software building Makefiles are now much more easier to read, modify and use, along with several new features that have been added. See below for the full list. - Until now, Maneage needed the host to have a 'make' implementation because Make was necessary to build Lzip. Lzip is then used to uncompress the source of our own GNU Make. However, in the minimalist/slim versions of operating systems (for example used to build Docker images) Make isn't included by default. Since Lzip was the only program before our own GNU Make was installed, we consulting Antonio Diaz Diaz (creator of Lzip) and he kindly added the necessary functionality to a new version of Lzip, which we are using now. Hence we don't need to assume a Make implementation on the host any more. With this commit, Lzip and GNU Make are built without Make, allowing everything else to be safely built with our own custom version of GNU Make and not using the host's 'make' at all. - Until recently (Commit 3d8aa5953c4) GNU Make was built in 'basic.mk'. Therefore 'basic.mk' was written in a way that it can be used with other 'make' implementations also (i.e., important shell commands starting with '&&' and ending in '\' without any comments between them!). Furthermore, to help in style uniformity, the rules in 'high-level.mk' and 'python.mk' also followed a similar structure. But due to the point above, we can now guarantee that GNU Make is used from the very first Makefile, so this hard-to-read structure has been removed in the software build recipes and they are much more readable and edit-friendly now. - Until now, the default backup servers where at some fixed URLs, on our own pages or on Gitlab. But recently we uploaded all the necessary software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is more suitable for this task (it promises longevity, has a fixed DOI, while allowing us to add new content, or new software tarball versions). With this commit, a small script has been written to extract the most recent Zenodo upload link from the Zenodo DOI and use it for downloading the software source codes. - Until now, we primarily used the webpage of each software for downloading its tarball. But this caused many problems: 1) Some of them needed Javascript before the download, 2) Some URLs had a complex dependency on the version number, 3) some servers would be randomly down for maintenance and etc. So thanks to the point above, we now use the Zenodo server as the primary download location. However, if a user wants to use a custom software that is not (yet!) in Zenodo, the download script gives priority to a custom URL that the users can give as Make variables. If that variable is defined, then the script will use that URL before going onto Zenodo. We now have a special place for such URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a good documentation themselves) are preserved here, but are commented by default. - The software source code downloading and checksum verification step has been moved into a Make function called 'import-source' (defined in the 'build-rules.mk' and loaded in all software Makefiles). Having taken all the low-level steps there, I noticed that there is no more need for having the tarball as a separate target! So with this commit, a single rule is the only place that needs to be edited/added (greatly simplifying the software building Makefiles). - Following task #15272, A new option has been added to the './project' script called '--all-highlevel'. When this option is given, the contents of 'TARGETS.conf' are ignored and all the software in Maneage are built (selected by parsing the 'versions.conf' file). This new option was added to confirm the extensive changes made in all the software building recipes and is great for development/testing purposes. - Many of the software hadn't been tested for a long time! So after using the newly added '--all-highlevel', we noticed that some need to be updated. In general, with this commit, 'libpaper' and 'pcre' were added as new software, and the versions of the following software was updated: 'boost', 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in' shell script was added in 'reproduce/software/shell/' which is installed with 'libpaper'. - Even though we intentionally add the necessary flags to add RPATH inside the built executable at compilation time, some software don't do it (different software on different operating systems!). Until now, for historical reasons this check was done in different ways for different software on GNU/Linux sytems. But now it is unified: if 'patchelf' is present we apply it. Because of this, 'patchelf' has been put as a top-level prerequisite, right after Tar and is installed before anything else. - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in 'basic.mk', it was 'glibtool'! This caused many confusions and is corrected with this commit (in 'basic.mk', it is also 'libtool'). - A new argument is added to the './project' script to allow easy loading of the project's shell and environment for fast/temporary testing of things in the same environment as the project. Before activating the project's shell, we completely remove all host environment variables to simulate the project's environment. It can be called with this command: './project shell'. A simple prompt has also been added to highlight that the user is using the Maneage shell!
2020-06-17Software tarballs are downloaded even if not builtMohammad Akhlaghi-6/+0
Some low-level software aren't necessary on some operating systems, for example GCC can't be built on macOS, hence we don't build it and the GCC-only dependencies. Also, on GNU/Linux systems users could configure with '--host-cc' to avoid all the time it takes to build GCC when doing a fast test. Until now, in such cases not only was the software not installed, but the tarballs of the software were also not downloaded. Hence making the output of '--dist-software' incomplete (as in bug #58561). With this commit, we now import all the necessary tarballs, when the software isn't necessary for the particular system, it won't be built or cited, but its tarball will be present anyway, thus allowing the output of '--dist-software' to be complete.
2020-06-16XLSX I/O properly accounts for local buildMohammad Akhlaghi-2/+2
Until now, when adding the necessary library flags to the build of XLSX I/O, we were effectively over-writing the 'LDFLAGS' variables. So the compiler was effectively not being told where to look for the necessary libraries. With this commit, to fix the problem, we now append the new linking flags to LDFLAGS in XLSX I/O's build, not over-write it.
2020-06-02Core software build before using Make to build other softwareMohammad Akhlaghi-2/+10
Until now, Maneage would only build Flock before building everything else using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid parallel downloads during the building of software (which could cause network problems). But after recently trying Maneage on FreeBSD (which is not yet complete, see bug #58465), we noticed that the BSD implemenation of Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq' parts) and its shell also had some peculiarities. It was thus decided to also install our own minimalist shell, Make and compressor program before calling 'basic.mk'. In this way, 'basic.mk' can now assume the same GNU Make features that high-level.mk and python.mk assume. The pre-make building of software is now organized in 'reproduce/software/shell/pre-make-build.sh'. Another nice feature of this commit is for macOS users: until now the default macOS Make had problems for parallel building of software, so 'basic.mk' was built in one thread. But now that we can build the core tools with GNU Make on macOS too, it uses all threads. Furthermore, since we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have to finish every line of a long rule with a backslash to keep variables and such. Generally, the pre-make software are now organized like this: first we build Lzip before anything else: it is downloaded as a simple '.tar' file that is not compressed (only ~400kb). Once Lzip is built, the pre-make phase continues with building GNU Make, Dash (a minimalist shell) and Flock. All of their tarballs are in '.tar.lz'. Maneage then enters 'basic.mk' and the first program it builds is GNU Gzip (itself packaged as '.tar.lz'). Once Gzip is built, we build all the other compression software (all downloaded as '.tar.gz'). Afterwards, any compression standard for other software is fine because we have it. In the process, a bug related to using backup servers was found in 'reproduce/analysis/bash/download-multi-try' for calling outside of 'basic.mk' and removed Bash-specific features. As a result of that bug-fix, because we now have multiple servers for software tarballs, the backup servers now have their own configuration file in 'reproduce/software/config/servers-backup.conf'. This makes it much easier to maintain the backup server list across the multiple places that we need it. Some other minor fixes: - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'. - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD (which doesn't have GCC). - We are now using 'uname -s' to specify if we are on a Linux kernel or not, if not, we are still using the old 'on_mac_os' variable. - While I was trying to build on FreeBSD, I noticed some further corrections that could help. For example the 'makelink' Make-function now takes a third argument which can be a different name compared to the actual program (used for examle to make a link to '/usr/bin/cc' from 'gcc'. - Until now we didn't know if the host's Make implementation supports placing a '@' at the start of the recipe (to avoid printing the actual commands to standard output). Especially in the tarball download phase, there are many lines that are printed for each download which was really annoying. We already used '@' in 'high-level.mk' and 'python.mk' before, but now that we also know that 'basic.mk' is called with our custom GNU Make, we can use it at the start for a cleaner stdout. - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a system where we can't install GCC (or has activated the '--host-cc' option), it may not be present and the project shouldn't break because of this. So with this commit, when a Fortran compiler isn't present, WCSLIB will be built with the '--disable-fortran' configuration option. This commit (task #15667) was completed with help/checks by Raul Infante-Sainz and Boud Roukema.
2020-05-28Fixed TeXLive crash because of differing local and server versionsRaul Infante-Sainz-3/+80
Until this commit, when the user had a previous TeXLive tarball already present (in their software-tarball directory) compared to the CTAN server, the project crashed in the configure phase. This was because TeXLive is updated yearly and we don't yet install TeXLive from source (currently we use its own package manager, but we plan to fix this in task #15267). With this commit, we fix the problem by checking the cause of the crash during the installation of TeX. If the crash is due to this particular error, we ignore the old tarball and download the new one and install it (the old one is still kept in '.build/software/tarballs', but will get a '-OLD' in its name. This probem was recurrent, and every year that TeXLive is updated, the previous tarball had to be removed manually! But with this commit, this is done automatically. The detection and fix of this bug has been possible with the help of Mohammad Akhlaghi, thanks!
2020-05-25Unified reference to GNU/Linux and free softwareMohammad Akhlaghi-1/+1
One of the main reasons to building Maneage is to properly acknowledge/attribute the authors of software in research. So we have adopted a standard of never referring to the GNU-based operating systems running the Linux kernel simply as "Linux", we avoid terms like "Open Sourse" and use Free Software instead (in the same spirit). With this commit, a few instances of the cases above have been corrected, they had slipped through our fingers when we initially imported them into the project. In the special case of the "Journal for Open Source Software", we simply replaced it with its abbreviation (JOSS). This was done because in effect we were generally using journal name abbreviations in almost all the citations already. To avoid any inconsistancies, the names of the three other journals that weren't abbreviated are also abbreviated.
2020-05-23New software: Valgrind and PatchBoud Roukema-0/+36
With this commit, Maneage now includes instructions to build the memory tracing tool Valgrind and the program 'patch' (to apply corrections/patches in text files and in particular the sources of programs). For this version of Valgrind, some patches were necessary for an interface with OpenMPI 2.x (which is the case now). Also note that this version of Valgrind's checks can fail with GCC 10.1.x (when using '--host-cc'), and the failures aren't due to internal problems but due to how the tests are designed (https://bugs.gentoo.org/707598). So currently if any of Valgrind's checks fail, Maneage still assumes that Valgrind was built and installed successfully. While testing on macOS, we noticed that it needs the macOS-specific 'mig' program which we can't build in Maneage. DESCRIPTION: The mig command invokes the Mach Interface Generator to generate Remote Procedure Call (RPC) code for client-server style Mach IPC from specification files. So a symbolic link to the system's 'mig' is now added to the project's programs on macOS systems. This commit's build of Patch and Valgrind has been tested on two GNU/Linux distributions (Debian and ArchLinux) as well as macOS. Work on this commit started by Boud Roukema, but also involved tests and corrections by Mohammad Akhlaghi and Raul Infante-Sainz.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-8/+11
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-05-08Installing Scamp with the option --enable-plplot=noRaul Infante-Sainz-1/+3
Until this commit, Scamp was installed with the option `--enable-plplot=yes' (the default). However, Maneage does not have PLplot included. As it is possible to install Scamp without PLplot (in that case it won't generate plots), with this commit this option has been set to `no'. As a consequence, Scamp will be installed even if the host system does not have PLplot without crashing (but it won't make any plot).
2020-05-08GNU Gettext built as a dependency of BashRaul Infante-Sainz-13/+0
Until now Maneage used the host's GNU Gettext if it was present. Gettext is a relatively low-level software that enables programs to print messages in different languages based on the host environment. Even though it has not direct effect on the running of the software for Maneage and the lanugage environment in Maneage is pre-determined, it is necessary to have it because if the basic programs see it in the host they will link with it and will have problems if/when the host's Gettext is updated. With this commit (which is actually a squashed rebase of 9 commits by Raul and Mohammad), Gettext and its two extra dependencies (libxml2 and libunistring) are now installed within Maneage as a basic software and built before GNU Bash. As a result, all programs built afterwards will successfully link with our own internal version of Gettext and libraries. To get this working, some of the basic software dependencies had to updated and re-ordered and it has been tested in both GNU/Linux and macoS. Some other minor issues that are fixed with this commit - Until this commit, when TeX was not installed, the warning message saying how to run the configure step in order to re-configure the project was not showing the option `-e'. However, the use of this option is more convenient than entering the top-build directory and etc every time. So with this commit, the warning message has been changed in order use the option `-e' in the re-configure of the project. - Until now, on macOS systems, Bash was not linking with our internally built `libncurses'. With this commit, this has been fixed by setting `--withcurses=yes' for Bash's configure script.
2020-05-06Software are rebuilt automatically with change of versionMohammad Akhlaghi-76/+78
Until now, when you changed the version of a software in an already-built system, its tarball would be downloaded, but it wouldn't actually build. The only way would be to force the build by deleting the main target of that file (under `.local/version-info/TYPE/PROGRAM'). This was because the tarballs were an order-only prerequisite which was implemented some time ago based on some theoretical argument that if the tarball dates changes, the software should not be rebuilt (because we check the checksum). However, the problems this causes are more than those it solves: Users may forget to delete the main target of the program and mistakenly think that they are using the new version. The fact that all the numbers going into the paper also contain this number further hides this. With this commit, tarballs are no longer order-only and any time a version of a software is updated, it will be automatically built and not cause confusion and manual intervention by the users. As a result of this change, I also had to correct the way we find the tarball from the list of prerequisites.
2020-04-28Configration bug fixed: other problematic software names from tarballBoud Roukema-5/+4
Similar to the previous commit (e43e3291483699), following a change made yesterday in the identification of software names from their tarballs, a few other problematic names are corrected with this commit: `apr-util', HDF5, TeX Live's installation tarball and `rpcsvc-proto'. Even though we have visually checked the list of software, other unidentified similar cases may remain and will be fixed when found in practice.
2020-04-27Configuration: improved version separation from tarball nameBoud Roukema-8/+11
Until now, the sed script for determining URL download rules in the three software building Makefiles (`basic.mk', `high-level.mk' and `python.mk') considered package names such as `fftw-3...` and `fftw2-2.1...` to be identical. As the example above shows, this would make it hard to include some software that may hav conflicting non-number names. With this commit, the SED script that is used to separate the version from the tarball name only matches numbers that are after a dash (`-'). Therefore considers `fftw-3...` and `fftw-2...` to be identical, but `fftw-3-...` and `fftw2-2.1...` to be different. As a result of this change, the `elif' check for some of the other programs like `m4', or `help2man' was also corrected in all three Makefiles. While doing this check on all the software, we noticed that `zlib-version' is being repeated two times in `version.conf' so it was removed. It caused no complications, because both were the same number, but could lead to bugs later.
2020-04-20Maneage instead of Template in README-hacking.md and copyright noticesMohammad Akhlaghi-11/+8
Until now, throughout Maneage we were using the old name of "Reproducible Paper Template". But we have finally decided to use Maneage, so to avoid confusion, the name has been corrected in `README-hacking.md' and also in the copyright notices. Note also that in `README-hacking.md', the main Maneage branch is now called `maneage', and the main Git remote has been changed to `https://gitlab.com/maneage/project' (this is a new GitLab Group that I have setup for all Maneage-related projects). In this repository there is only one `maneage' branch to avoid complications with the `master' branch of the projects using Maneage later.
2020-04-17Replaced name of directory under akhlaghi.org as backup serverMohammad Akhlaghi-10/+10
Until now, when a the raw tarball of some software wasn't usable, I would put it under my own webpage, or `akhlaghi.org/reproduce-software'. That same address was also used as a backup server. However, now the project has a proper name: Maneage. So I changed the directory on my own server to `akhlaghi.org/maneage-software'. With this commit, this new address has replaced the old one. But to avoid crashes in projects that haven't yet merged with the main Maneage branch, the old `reproduce-software' still works (its actually a symbolic link to the new directory now).
2020-04-17IMPORTANT: software config directly under reproduce/software/configMohammad Akhlaghi-8/+8
Until now the software configuration parameters were defined under the `reproduce/software/config/installation/' directory. This was because the configuration parameters of analysis software (for example Gnuastro's configurations) were placed under there too. But this was terribly confusing, because the run-time options of programs falls under the "analysis" phase of the project. With this commit, the Gnuastro configuration files have been moved under the new `reproduce/analysis/config/gnuastro' directory and the software configuration files are directly under `reproduce/software/config'. A clean build was done with this change and it didn't crash, but it may cause crashes in derived projects, so after merging with Maneage, please re-configure your project to see if anything has been missed. Please let us know if there is a problem.
2020-04-13Configure (TeXLive): year correction also in high-level packagesMohammad Akhlaghi-1/+1
In the previous commit, I removed the year from the basic installation of TeXLive packages, but I forgot to correct this in the high-level TeXLive packages! This is corrected with this commit.
2020-04-13Configure (TeXLive): Year of distribution no longer in directoryMohammad Akhlaghi-6/+4
It is this time of year again: TeXLive has transitioned to its 2020 release and the year is imprinted into the installation directory of TeXLive. Until now, we have had to manually change this year and it caused complications and was very annoying. With this commit, the explicit year has been removed from TeXLive's installation and we now simply put a `maneage' instead of the year. I tried this on another system and it worked nicely. Until the time that we can fully install LaTeX packages from source tarballs, this is the best thing we could do for now.
2020-04-06Astropy now depending on the Expat library to fix internal conflictRaul Infante-Sainz-3/+3
Until now, Astropy was instructed to build its own internal copy of the Expat library. However, with the recent commits before, Maneage now includes an installation of Expat and Astropy can't keep the two (its internal version and the project's version) separate, so they conflict and don't let Astropy get built. With this commit, the problem is fixed by setting the Expat library as an explicit dependency of Astropy and asking Astropy to ignore its internal copy. While doing this, I recognized that it is much easier and elegant to add steps in various stages of the `pybuild' function through hooks instead of variables. So the fifth argument of the `pybuild' function was removed and now it actually checks if hooks are defined as functions and if so, they will be called. The `pyhook_after' function was also implemented in the installation of `pybind11' (which needed it, given that the 5th argument of `pybuild' was removed) and after doing a test-build, I noticed that two lines were not ending with a `\' in `boost' (a dependency of `pybind11'). Commit written originally by Mohammad Akghlaghi
2020-04-05The build of M4 and XLSX I/O on macOS has been fixedRaul Infante-Sainz-4/+11
Until now, the M4 that was built on macOS had internal problems (as discussed in #1): it would simply print `Abort trap: 6' in the output and abort. After looking at the build of Homebrew, I noticed that they apply a patch (correct one line) to fix this problem. To be able to apply that patch on macOS systems, I had fully open up the build recipe of M4 and atleast on the testing system, it was built successfully. Also, after successfully building M4, and thus Autoconf and thus Minizip, we were able to build XLSX I/O on a macOS and found out that the internal library's full address wasn't being put in the libraries and executables. With this commit, we now use macOS's `install_name_tool' to correct the positions of the two `libxlsxio_*' libraries in all its executables. This commit was originally written by Mohammad Akhlaghi
2020-04-05Including version number of Minizip in its installation targetRaul Infante-Sainz-1/+1
Until this commit, only the word `Minizip' was written into the Minizip installation target (without the version number of Minizip). With this commit, this minor bug has been fixed by using the appropiate Make variable: `$(minizip-version)'.
2020-04-05Building Minizip 1.x instead of Minizip 2.xRaul Infante-Sainz-15/+62
Minizip is a dependency of XLSX I/O and until now, I was just using the most recent version I found (2.9.2), but XLSX I/O is written for the Minizip 1.x series, not 2.x. Somehow it didn't cause a crash on my computer!!! I think XLSX I/O's CMake is instructed to look into system directories by default when it doesn't find the directories in the given places. And because I had installed Minizip on my operating system, it did't complain. Upon trying the build on their systems, Yahya, Raul and Zahra reported a failure in the build of XLSX I/O which was due the to the problem above (we were installing the wrong version of Minizip!). With this commit, this has been fixed by installing the 1.x series of Minizip (whish is actually installed within zlib!). This commit was original done by Mohammad Akhlaghi.
2020-04-01Removed multiple tabs in MissFITS tarball definitionRaul Infante-Sainz-1/+1
With this commit, multiples tabs in the definition of MissFITS tarball have been removed. Now they are white spaces.
2020-02-24MissFITS is now added to the templateSurena Fatemi-0/+7
MissFITS is package for manipulating FITS files. I added it as my first commit to the project for educational purposes.
2020-02-16XLSX I/O installed with its two dependencies: expat and minizipMohammad Akhlaghi-0/+32
XLSX I/O is a very simple and fast program and library for reading and writing `.xls' and `.xlsx' files (mainly used by Microsoft Excel) to CSV files. It has two separate executables that can be called for an Excel file and will output a CSV plain text file that can then be used within the pipeline with more standard tools.
2020-02-11Using backup server when original download server failsMohammad Akhlaghi-1/+6
Until now, the main download script could only check one server for the given URL. However, ultimately the actual server that a file is downloaded from is irrelevant for this project: we actually check its checksum. Especially in the case of software (which are distributed over many servers), this can usually be very annoying: the servers may not properly communicate with the running system and even the 10 trials won't be enough. With this commit, the download script `reproduce/analysis/bash/download-multi-try' can take a new optional argument (a 5th argument). It assumes this argument is a space-separated list of server(s) to use as backup for the original URL. When downloading from the original URL fails, it will look into this list and try downloading the same file from each given server.
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-2/+2
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-31Configure step: compiler checks done before basic settingsMohammad Akhlaghi-1/+6
Until now, the project would first ask for the basic directories, then it would start testing the compiler. But that was problematic because the build directory can come from a previous setting (with `./project configure -e'). Also, it could confuse users to first ask for details, then suddently tell them that you don't have a working C library! We also need to store the CPATH variable in the `LOCAL.conf' because in some cases, the compiler won't work without it. With this commit, the compiler checking has been moved at the start of the configure script. Instead of putting the test program in the build directory, we now make a temporary hidden directory in the source directory and delete that directory as soon as the tests are done. In the process, I also noticed that the copyright year of the two hidden files weren't updated and corrected them.
2020-01-22Perl is now a dependency of Coreutils, LD_LIBRARY_PATH in basic.mkMohammad Akhlaghi-1/+1
Until now Perl was built after Coreutils, but I recently noticed that Coreutils actually uses Perl while creating its manpages. So it is now built before Coreutils. Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils can't build its man page for `md5sum'. The problem was found to be due to the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our installed library path in `basic.mk'. Therefore, it would crash because the server had an older version of OpenSSL than the one that the template's Coreutils was built with. In the meantime (while addressing the issues above, because we only had one thread on the AWS server) I also noticed a few programs that were using a summarize compilation command (that just prints `CC xxx.c' instead of the whole command) so I fixed them by adding `V=1'. This bug was found by Idafen Santana PĂ©rez.
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-7/+7
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19LIBRARY_PATH is set accordingly based on the hostMohammad Akhlaghi-0/+7
Until now, GCC wouldn't build properly on Debian-based operating systems because `ld' needed to link with several necessary C library features like `crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to add the directory containing them to `LIBRARY_PATH'. In the previous commit, I actually searched for these files, but while testing on another system, I noticed that it can be problematic (other architectures may exist). With this commit, we are actually finding the build architecture of the running GCC (which is the same as the `ld') and using that to fix a fixed directory to `LIBRARY_PATH'.
2020-01-18TeXLive version printing updated to include revisionMohammad Akhlaghi-7/+26
Until now, when find the versions of the TeXLive packages, we would assume that `cat-date' is always present (because some packages don't have a version!). However, apparently an update has been made in the TeXLive Manager (`tlmgr') and `cat-date' is no longer present! As a result, none of the TeXLive packages were being printed. With this commit, it now assumes that `revision' is always present for every package, but it also attempts to read `cat-date' (for backwards compatability). When `cat-version' isn't present, it will try printing `revision' and if that is also not present, it will print the date.
2020-01-13Adding X11 libraries path to LDFLAGS in Ghostscript installationRaul Infante-Sainz-1/+4
Until this commit, the checking of X11 installation done to ensure that it is already available in the host system was crashing in macOS systems. The reason is that the place of the X11 libraries use to be `/opt/X11/lib' in macOS systems. With this commit, this issue has been fixed by adding this directory to the LDFLAGS.
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-2/+2
Now that its 2020, its necessary to include this year in the copyright statements.
2019-12-18Fixing Libxml2 checksum variable nameRaul Infante-Sainz-1/+1
Until this commit, the number `2' was missing in the checksum variable name of that library. It was `libxml-checksum' but it should be `libxml2-checksum'. With this commit, this issue has been fixed.
2019-12-14R explicitly built without support for OpenMPMohammad Akhlaghi-1/+2
OpenMP takes a LONG TIME to build, so to keep things reproducible we are explicitly disabling OpenMP, if a user needs OpenMP, its trivial to just add it as a prerequisite of R. The problem is that in some scenarios (based on other dependencies and when they were built in the build directory), OpenMP may be present when R is being installed and in other it may not. We don't want the result to be different between the two builds.
2019-12-14Core R package is now in templateMohammad Akhlaghi-0/+14
With this commit, we now have the core R interpretter within the template. We should later include instructions to install R packages (possibly in a separate top-level Makefile like Python).
2019-12-04Checking for necessary X11 libraries before building GhostscriptMohammad Akhlaghi-1/+35
Until now, Ghostscript was using some host system's X11 libraries during its build (and later at run-time). We should ideally install all these necessary libraries within the template (Task #15481). But right now we are too busy. As a temporary work-around we try building a small dummy program that links with some of those libraries before attempting to built Ghostscript. If it fails, then a notice is printed with the cause and explaining a temporary solution is suggested: how to install those libraries on the system when you have root access.
2019-11-08Matplotlib problem on macOS system with GNU GCC fixedMohammad Akhlaghi-1/+1
Until now when building Matplotlib on macOS systems, we were using the default C compiler. However, while Yahya Sefidbakht (previously) and Mahdieh Nabavi (now) were trying to build the template, on their macOS using the GNU Compiler Collection (GCC), we found that Matplotlib needs special macOS headers that GCC doesn't recognize. With this commit, when Matplotlib is being built on macOS systems, it uses `clang' and this fixed the problem (so far checked on Mahdieh's machine).
2019-11-06GNU Project Debugger (GDB) and Pexpect can now be builtMohammad Akhlaghi-0/+7
These two packages are necessary to build the GNU C Library.
2019-11-03Corrected build of log4cxx by re-building the tarball usedMohammad Akhlaghi-13/+29
When building the log4cxx tarball from its Git history, I noticed that files with very long names are not packaged by tar (because by default Automake uses the ancient v7 tar format that only supports file names less than 99 characters). So I build the tarball with the `tar-ustar' option to Automake (by modifying the log4cxx source) and the resulting tarball was able to compile and run successfully. This has been described above the rule to build log4cxx and I also sent an email to their developing mailing list to inform them of this problem. If they address it, I will remove the note on the necessary corrections.
2019-10-31Minor corrections in distribution and autoconf prerequisite of automakeMohammad Akhlaghi-4/+32
Some minor corrections were made in the template: - When making the distribution, `.swp' files (created by Vim) are also removed. - Autoconf is set as a prerequisite of Automake I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball needs some patches, so I have just left it half done until someone actually needs it and we apply the patch.
2019-10-30Tarballs set as order-only prerequisites, not needed for build callsMohammad Akhlaghi-144/+156
Until now, the tarballs were the first normal prerequisite of the software. As a result if their date changed, the whole software would be re-built. However, for tarballs specifically, we actually check their contents with a checksum, so their date is irrelevant (if its newer than the built program, but has the same checksum, there is no need to re-build the software). Also, calling the tarball name as an argument to the building process (for example `gbuild') was redundant. It is now automatically found from the list of order-only prerequisites within `gbuild' and `cbuild' (similar to how it was previously found in the `pybuild' for Python building). A `README.md' file has also been placed in `reproduce/software/make' to help describe the shared properties of the software building Makefiles. This will hopefully grow much larger in the future.
2019-10-28HEALPix, Autoconf and Automake now in libraryMohammad Akhlaghi-1/+62
The tarball of HEALPix includes multiple languages and doesn't include the ready-to-run GNU Build System by default, we actually have to build the `./configure' script for the C/C++ libraries. So it was necessary to also include GNU Autoconf and GNU Automake as prerequisites of HEALPix. However, the official GNU Autoconf tarball (dating from 2012) doesn't build on modern systems, so I just cloned it from its source and bootstrapped it and built its modern tarball which we are using here.
2019-10-24APR, APR Utility, Boost and Eigen are now availableMohammad Akhlaghi-2/+61
As part of an effort to bring in all the dependencies of the LSST Science pipeline (which includes the last commit), these software are now available in the template.