aboutsummaryrefslogtreecommitdiff
path: root/reproduce
AgeCommit message (Collapse)AuthorLines
2020-02-16XLSX I/O installed with its two dependencies: expat and minizipMohammad Akhlaghi-0/+38
XLSX I/O is a very simple and fast program and library for reading and writing `.xls' and `.xlsx' files (mainly used by Microsoft Excel) to CSV files. It has two separate executables that can be called for an Excel file and will output a CSV plain text file that can then be used within the pipeline with more standard tools.
2020-02-13Corrected version of Texinfo when reportingMohammad Akhlaghi-1/+1
Until now we were mistakenly reporting the version of SED instead of Texinfo. With this commit, we corrected it! This was reported by Raul Infante Sainz.
2020-02-13Adding a link to the *crt*.o files in the local install directoryMohammad Akhlaghi-1/+17
Until now, we defined `LIBRARY_PATH' to fix the problem of the `ld' linker of Binutils needing several `*crt*.o' files to run. However, some software (for example ImageMagick) over-write `LIBRARY_PATH', therefore there is no other way than to put a link to these necessary files in our local build directory. With this commit, we fixed the problem by putting a link to the system's relevant files in the local library directory. This fixed the problem with ImageMagick. Later, when we build the GNU C Library in the project, we should remove this step. This bug reported by Raul Castellanos Sanchez.
2020-02-11Configure script won't crash without Fortran compiler, only a warningMohammad Akhlaghi-27/+46
Until now, when a Fortran compiler didn't exist on the host operating system, the configure script would crash with a warning. But some projects may not need Fortran, so this is just an extra/annoying crash! With this commit, it will still print the warning, but instead of a crash, it will just sleep for some seconds, then continue. Later, when if a software needs Fortran, it's building will crash, but atleast the user was warned. In the future, we should add a step to check on the necessary software and see if Fortran is necessary for the project or not. The project configuration should indeed crash if Fortran is necessary, but we should tell the user that software XXXX needs Fortran so we can't continue without a Fortran compiler. Also, a small sentence ("Project's configuration will continue in XXXX seconds.") was added after all the warnings that won't cause a crash, so user's don't think its a crash.
2020-02-11Using backup server when original download server failsMohammad Akhlaghi-7/+40
Until now, the main download script could only check one server for the given URL. However, ultimately the actual server that a file is downloaded from is irrelevant for this project: we actually check its checksum. Especially in the case of software (which are distributed over many servers), this can usually be very annoying: the servers may not properly communicate with the running system and even the 10 trials won't be enough. With this commit, the download script `reproduce/analysis/bash/download-multi-try' can take a new optional argument (a 5th argument). It assumes this argument is a space-separated list of server(s) to use as backup for the original URL. When downloading from the original URL fails, it will look into this list and try downloading the same file from each given server.
2020-02-01Make called with -k during software buildingMohammad Akhlaghi-2/+2
Until now, Make was just run ordinarily on the two Makefiles of the software building phase. Therefore when there was a problem with one software while building in parallel, Make would only complete the running rules and stop afterwards. But when other rules don't depened on the crashed rule, its a waste of time to stop the whole thing. With this commit, both calls to Make in the `configure.sh' script are done with the `-k' option (or `--keep-going' in GNU Make). With this option, if a rule crashes, the other rules that don't depend on it will also be run. Generally, anything that doesn't depend on the crashed rule will be done. The `-k' option is a POSIX definition in Make, so it is present in most implemenetations (for the call to `basic.mk').
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-8/+8
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-31Configure step: compiler checks done before basic settingsMohammad Akhlaghi-418/+409
Until now, the project would first ask for the basic directories, then it would start testing the compiler. But that was problematic because the build directory can come from a previous setting (with `./project configure -e'). Also, it could confuse users to first ask for details, then suddently tell them that you don't have a working C library! We also need to store the CPATH variable in the `LOCAL.conf' because in some cases, the compiler won't work without it. With this commit, the compiler checking has been moved at the start of the configure script. Instead of putting the test program in the build directory, we now make a temporary hidden directory in the source directory and delete that directory as soon as the tests are done. In the process, I also noticed that the copyright year of the two hidden files weren't updated and corrected them.
2020-01-31Architecture-specific C headers on Debian-based OSs now accountedMohammad Akhlaghi-6/+15
Rencely the building of GCC was allowed on Debian-based systems that have their basic C library in architecture-specific directories, like `/usr/lib/x86_64-linux-gnu'. However, these systems also have their headers in non-standard locations, for example `/usr/include/x86_64-linux-gnu' and this caused a crash on a new Ubuntu system. /usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No such file or directory 27 | #include <bits/libc-header-start.h> | ^~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. The reason it didn't cause problems on other Ubuntus that we tested before was historic: In the old days, we would ask Ubuntu systems to install multilib features to have GCC. Because they had installed those features, this problem didn't show up! But this wasn't mandatory! With this commit, the `CPATH' environment variable is set (similar to how `LIBRARY_PATH' was set) and this fixed the problem on a clean Debian virtual machine. This bug was reported by Sebastian Luna Valero.
2020-01-27Initial scripts compatible with Dash (minimalistic POSIX)Mohammad Akhlaghi-35/+47
Until now, the initial project scripts were primarily tested with GNU Bash. But Bash is not generally available on all systems (it has many features beyond POSIX). Because of this, effectively we were imposing the requirement on the user that they must have Bash installed. We recently started this with setting the shebang of `project' and `reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul and Gaspar reported an error on their systems. To fix the problem, I installed Dash (a minimalist POSIX-compliant shell) on my computer and temporarily set the shebangs to `/bin/dash', ran the project configuration step and fixed all issues that came up. With this commit, it can go all the way to building GCC on my system's Dash. After this stage (when `high-level.mk' is called), there is no problem, because we have our own version of GNU Bash and that installed version is used. Probably some more issues still remain and will hopefully be found in the future. While doing this, I also noticed the following two minor issues: - The `./project configure' option `--input-dir' was not recognized because it was mistakenly checking `--inputdir'. It has been corrected. - The test C programs now use the `<<EOF' method instead of `echo'. - In `basic.mk', the extra space between `syspath' and `:=' was removed (it was an ancient relic!).
2020-01-23Hashbangs of project and configure.sh set to /bin/shMohammad Akhlaghi-1/+1
Until now, the hashbang of these two shell scripts was set to `/bin/bash', hence assuming that GNU Bash exists on the host system! But this is an extra requirement on the host operating system and these two scripts should be written such that they operate on a POSIX shell (the generic `/bin/sh' which can point to any shell program). With this commit this has been implemented! We may confront some errors as the system is run on other systems, but we should fix such errors and work hard to make these two scripts as POSIX-compatible as possible (runnable on any shell, so as not to force users to install Bash before running the project). This completes Task #15525.
2020-01-23IMPORTANT: Project preparation is now also done with project makeMohammad Akhlaghi-2/+2
Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system.
2020-01-22Better explanation for missing static C libraryMohammad Akhlaghi-12/+22
Until now, the explanation for a missing static C library didn't actually guide the users to look above and see the error message! So with this commit, I edited it a little to be more clear (and mention to look above). Also, I noticed that on Amazon AWS systems, the static C library is installed as a separate package, so to help the users, I added the necessary command and some better explanation.
2020-01-22Perl is now a dependency of Coreutils, LD_LIBRARY_PATH in basic.mkMohammad Akhlaghi-67/+80
Until now Perl was built after Coreutils, but I recently noticed that Coreutils actually uses Perl while creating its manpages. So it is now built before Coreutils. Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils can't build its man page for `md5sum'. The problem was found to be due to the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our installed library path in `basic.mk'. Therefore, it would crash because the server had an older version of OpenSSL than the one that the template's Coreutils was built with. In the meantime (while addressing the issues above, because we only had one thread on the AWS server) I also noticed a few programs that were using a summarize compilation command (that just prints `CC xxx.c' instead of the whole command) so I fixed them by adding `V=1'. This bug was found by Idafen Santana Pérez.
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-50/+51
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19GNU Make updated to version 4.3Mohammad Akhlaghi-3/+3
GNU Make 4.3 was just announced, so I have updated it here is well. This was important because until now the installable version was in alpha-mode (4.2.90), now its a stable version.
2020-01-19Corrected typo in last commit (forgetting \ at end of line)Mohammad Akhlaghi-1/+1
In the previous commmit, I had forgot to add a `\' after the newly added `sys_library_path' variable to the `high-level.mk' call.
2020-01-19LIBRARY_PATH is set accordingly based on the hostMohammad Akhlaghi-20/+58
Until now, GCC wouldn't build properly on Debian-based operating systems because `ld' needed to link with several necessary C library features like `crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to add the directory containing them to `LIBRARY_PATH'. In the previous commit, I actually searched for these files, but while testing on another system, I noticed that it can be problematic (other architectures may exist). With this commit, we are actually finding the build architecture of the running GCC (which is the same as the `ld') and using that to fix a fixed directory to `LIBRARY_PATH'.
2020-01-19Better search for static C library at start of configurationMohammad Akhlaghi-108/+75
Until now, to see if a working static C library and `sys/cdefs.h' exist, we were checking absolute locations like `/usr/include/sys/cdefs.h' or `/usr/lib/libc.a' and `/usr/lib64/libc.a'. But this is not robust because on different systems, they can be in different locations. With this commit, we actually use `find' to find the location of `libc.a' and use that to add elements to CPPFLAGS and LDFLAGS. This should fix the problem on systems that have them on non-standard locations.
2020-01-18TeXLive version printing updated to include revisionMohammad Akhlaghi-7/+26
Until now, when find the versions of the TeXLive packages, we would assume that `cat-date' is always present (because some packages don't have a version!). However, apparently an update has been made in the TeXLive Manager (`tlmgr') and `cat-date' is no longer present! As a result, none of the TeXLive packages were being printed. With this commit, it now assumes that `revision' is always present for every package, but it also attempts to read `cat-date' (for backwards compatability). When `cat-version' isn't present, it will try printing `revision' and if that is also not present, it will print the date.
2020-01-18Added mweights as a TeXLive packageMohammad Akhlaghi-1/+2
After a new rebuild of the project, I noticed that we now need to also build the `mweights' package.
2020-01-18README-hacking.md: edits and corrections for easier customizationMohammad Akhlaghi-2/+2
The checklist descriptions were slightly edited to be more clear. Also, while following them, I noticed that while removing the "delete-me" parts on `verify.mk', would cause an error: the `if [ $$m == delete-me ];' statement we were saying to delete cause an error because `elif' was the first statement Bash would see. So with this commit, the `download' conditional (which isn't instructed to be deleted) was set to be the top (with an `if') and the `delete-me' conditional now has an `elif'.
2020-01-13Adding X11 libraries path to LDFLAGS in Ghostscript installationRaul Infante-Sainz-1/+4
Until this commit, the checking of X11 installation done to ensure that it is already available in the host system was crashing in macOS systems. The reason is that the place of the X11 libraries use to be `/opt/X11/lib' in macOS systems. With this commit, this issue has been fixed by adding this directory to the LDFLAGS.
2020-01-13Updated verion of Astropy to 4.0 the most recent oneRaul Infante-Sainz-3/+3
Newer versions of Astropy package has been released. With this commit, it has been updated. It has been increased from v3.2.1 to v4.0
2020-01-11Corrected markdown code notations in one README.mdMohammad Akhlaghi-4/+4
In markdown, to specify that part of the text should be shown as code we need to put it between two `. But in `reproduce/software/make/README.md', the code parts started with ` and finished in ' (like the file name above). With this commit, this has been corrected and it will display properly on web browsers.
2020-01-01Verification function checks if file existsMohammad Akhlaghi-3/+13
Until now, if the file to be verified didn't exist, a different checksum would be generated, and it would stop, but it wasn't immediately clear if the differing checksum is because the file doesn't exist at all! With this commit, before calculating the checksum, we first make sure if the file exists. If it doesn't exist an explicit error is printed and thus will help the project editor to find the cause of the problem.
2020-01-01Verification of output values and data added within templateMohammad Akhlaghi-22/+146
Until now, the only verification that the template provided was the published PDF. Users had to manually compare the published and generated PDFs (numbers, plots, tables) and see if they obtained the same result. However, this type of manual verification is not good and is prone to frustration and missing important differences. With this commit, a new Makefile has been added in the analysis steps: `verify.mk'. It provides facilities to easily verify the results that go into the paper. For example tables that go into making the paper's plots, or the LaTeX macros that blend into the text. See the updated parts in `README-hacking.md` for a more complete explanation. This completes task #15497.
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-56/+56
Now that its 2020, its necessary to include this year in the copyright statements.
2019-12-18Fixing Libxml2 checksum variable nameRaul Infante-Sainz-1/+1
Until this commit, the number `2' was missing in the checksum variable name of that library. It was `libxml-checksum' but it should be `libxml2-checksum'. With this commit, this issue has been fixed.
2019-12-18Removed extra backslash in Jeepney prerequisitesRaul Infante-Sainz-1/+1
An extra backslash in the prerequisites of Jeepney Python package was causing a crash in the installation of that software. With this commit, this problem has been fixed by removing the backslash.
2019-12-14R explicitly built without support for OpenMPMohammad Akhlaghi-1/+2
OpenMP takes a LONG TIME to build, so to keep things reproducible we are explicitly disabling OpenMP, if a user needs OpenMP, its trivial to just add it as a prerequisite of R. The problem is that in some scenarios (based on other dependencies and when they were built in the build directory), OpenMP may be present when R is being installed and in other it may not. We don't want the result to be different between the two builds.
2019-12-14Core R package is now in templateMohammad Akhlaghi-0/+16
With this commit, we now have the core R interpretter within the template. We should later include instructions to install R packages (possibly in a separate top-level Makefile like Python).
2019-12-13Necessary TeXLive packages installed: pdfescape letltxmacro bitsetMohammad Akhlaghi-1/+1
Recent builds of the template need these three packages to build the PDF. This was reported by Hamed Altafi.
2019-12-04Added pdftexmcds to the TeXlive packages to installMohammad Akhlaghi-1/+1
Until now we weren't including this package, but Alberto Madrigal recently reported that his build of the template failed because it needed it. With this commit, it is now being installed with the template.
2019-12-04Perl built without -shared on macOSMohammad Akhlaghi-1/+11
Until now, while building Perl, the `-Dlddlflags="-shared $$LDFLAGS"' option was being used for all systems. However, this caused `symbol(s) not found for architecture x86_64' problems on macOS systems. With this commit, this configure option is only passed for GNU/Linux systems. With it, Perl can be built nicely on both GNU/Linux and macOS.
2019-12-04Updated verions of Ghostscript and GnuastroMohammad Akhlaghi-4/+4
Newer versions of these packages have recently come out with major improvements, so they have been updated in the template.
2019-12-04Checking for necessary X11 libraries before building GhostscriptMohammad Akhlaghi-1/+35
Until now, Ghostscript was using some host system's X11 libraries during its build (and later at run-time). We should ideally install all these necessary libraries within the template (Task #15481). But right now we are too busy. As a temporary work-around we try building a small dummy program that links with some of those libraries before attempting to built Ghostscript. If it fails, then a notice is printed with the cause and explaining a temporary solution is suggested: how to install those libraries on the system when you have root access.
2019-11-29Download links directly to actual file if it exists in INDIRMohammad Akhlaghi-2/+8
Until now, when an input dataset already exists in `INDIR', the template would just make a symbolic link to it in the build directory. However, in many cases, the files in INDIR will actually be links to other locations on the filesystem and some programs have problems following too many links. With this commit, the template is now using the `readlink' program (part of GNU Coreutils) to follow a possible link and point the link in the build directory directly to an actual non-link file.
2019-11-08Matplotlib problem on macOS system with GNU GCC fixedMohammad Akhlaghi-3/+9
Until now when building Matplotlib on macOS systems, we were using the default C compiler. However, while Yahya Sefidbakht (previously) and Mahdieh Nabavi (now) were trying to build the template, on their macOS using the GNU Compiler Collection (GCC), we found that Matplotlib needs special macOS headers that GCC doesn't recognize. With this commit, when Matplotlib is being built on macOS systems, it uses `clang' and this fixed the problem (so far checked on Mahdieh's machine).
2019-11-08Minor corrections applied in software buildsMohammad Akhlaghi-8/+13
While working on a different branch to build the GNU C Library, I noticed a few places in the template that need corrections which are now applied: 1. A new-line character after the "C compiler works" notice at the start of the configure script. 2. Removing possible `::' in the `LD_LIBRARY_PATH' definition of `basic.mk'. Note that its not necessary in the other steps because we don't use any outside-defined `LD_LIBRARY_PATH'. 3. Building GMP for C++ and also with `--enable-fat'. 4. Removing the unpacked Perl tarball directory after its installation.
2019-11-06GNU Project Debugger (GDB) and Pexpect can now be builtMohammad Akhlaghi-0/+18
These two packages are necessary to build the GNU C Library.
2019-11-06Perl is now built as a basic dependency of the templateMohammad Akhlaghi-5/+52
Perl is necessary to build Texinfo and later to build LaTeX. Until now we were just using the host operating system's installation of Perl, but in some instances that Perl can be too old and not suppor the features necessary. With this commit, Perl is now built from source during the basic installation step of the template. This was reported by Idafen Santana Pérez, after trying the pipeline on an Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
2019-11-03Corrected build of log4cxx by re-building the tarball usedMohammad Akhlaghi-15/+31
When building the log4cxx tarball from its Git history, I noticed that files with very long names are not packaged by tar (because by default Automake uses the ancient v7 tar format that only supports file names less than 99 characters). So I build the tarball with the `tar-ustar' option to Automake (by modifying the log4cxx source) and the resulting tarball was able to compile and run successfully. This has been described above the rule to build log4cxx and I also sent an email to their developing mailing list to inform them of this problem. If they address it, I will remove the note on the necessary corrections.
2019-10-31Minor corrections in distribution and autoconf prerequisite of automakeMohammad Akhlaghi-6/+37
Some minor corrections were made in the template: - When making the distribution, `.swp' files (created by Vim) are also removed. - Autoconf is set as a prerequisite of Automake I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball needs some patches, so I have just left it half done until someone actually needs it and we apply the patch.
2019-10-30Tarballs set as order-only prerequisites, not needed for build callsMohammad Akhlaghi-371/+427
Until now, the tarballs were the first normal prerequisite of the software. As a result if their date changed, the whole software would be re-built. However, for tarballs specifically, we actually check their contents with a checksum, so their date is irrelevant (if its newer than the built program, but has the same checksum, there is no need to re-build the software). Also, calling the tarball name as an argument to the building process (for example `gbuild') was redundant. It is now automatically found from the list of order-only prerequisites within `gbuild' and `cbuild' (similar to how it was previously found in the `pybuild' for Python building). A `README.md' file has also been placed in `reproduce/software/make' to help describe the shared properties of the software building Makefiles. This will hopefully grow much larger in the future.
2019-10-29lmfit and its dependencies added, Python tarballs as prerequisiteMohammad Akhlaghi-189/+260
Python's `lmfit' module and all its major dependencies (`asteval', `corner', `emcee' and `uncertainties') have been included in the template. While doing this I noticed that if the tarballs are the last prerequisite of each software building rule, then when building in parallel, the template will immediately start building packages as soon as the first one is downloaded. Not like the current way that it will attempt to download several, then start building. For now, this has been implemented in the Python build rules for all the modules and we'll later do the same for the other programs and libraries. This also motivated a simplification of the `pybuild' function: it now internally looks into the prerequisites and selects the tarball from the prerequisite that is in the tarballs directory. This isn't a problem for the build, but I just don't understand why Python can't recognize the version of `emcee', Python reads the version of `emcee' as `0.0.0'! But it doesn't cause any crash in the build, so for now its fine.
2019-10-28HEALPix, Autoconf and Automake now in libraryMohammad Akhlaghi-1/+99
The tarball of HEALPix includes multiple languages and doesn't include the ready-to-run GNU Build System by default, we actually have to build the `./configure' script for the C/C++ libraries. So it was necessary to also include GNU Autoconf and GNU Automake as prerequisites of HEALPix. However, the official GNU Autoconf tarball (dating from 2012) doesn't build on modern systems, so I just cloned it from its source and bootstrapped it and built its modern tarball which we are using here.
2019-10-24Several more LSST pipeline dependencies are addedMohammad Akhlaghi-13/+144
The following software are added with this commit: eigency, esutil, flake8, future, galsim, lsstdesccoord, pybind11 and pyflakes.
2019-10-24APR, APR Utility, Boost and Eigen are now availableMohammad Akhlaghi-2/+70
As part of an effort to bring in all the dependencies of the LSST Science pipeline (which includes the last commit), these software are now available in the template.
2019-10-24GNU Bison, GNU Help2man and Flex are now availableMohammad Akhlaghi-3/+26
With this commit these three software packages are now installable with this template.