aboutsummaryrefslogtreecommitdiff
path: root/reproduce
AgeCommit message (Collapse)AuthorLines
2019-10-01Minor corrections in configure and prepare phaseMohammad Akhlaghi-12/+11
Since ImageMagick can take long to build, we are now building it in parallel. Also, the part where we replace an `_' with `\_' in the software version at the end of the configure script was removed. It is more clear/readable that the actual rule that includes such a name deals with the underline (as is the case for `sip_tpv' which already dealt with it). Finally, I noticed that the checks at the start of `top-prepare' were missing new-lines. I had forgot that the Make single-shell variable isn't activated in this stage yet.
2019-10-01Infrastructure to keep preparation resultsMohammad Akhlaghi-17/+50
A special directory is now defined in `initialize.mk' that can be used in both the preparation and build phases. Also, the contents of prepared results can now be conditionally read during `./project make'.
2019-10-01Preparation phase added before final buildingMohammad Akhlaghi-3/+129
In many real-world scenarios, `./project make' can really benefit from having some basic information about the data before being run. For example when quering a server. If we know how many datasets were downloaded and their general properties, it can greatly optmize the process when we are designing the solution to be run in `./project make'. Therefore with this commit, a new phase has been added to the template's design: `./project prepare'. In the raw template this is empty, because the simple analysis done in the template doesn't warrant it. But everything is ready for projects using the template to add preparation phases prior to the analysis.
2019-10-01New versions for ImageMagick, Python, Numpy Scipy and MatplotlibMohammad Akhlaghi-19/+19
It was some time since these three software were not updated! With this commit the template now uses the most recent stable release of these packages. Also, the hosting server for ImageMagick was moved to my own webpage because unfortunately ImageMagick removes its tarballs from its own version.
2019-09-28Corrected name of SExtractor in its final targetMohammad Akhlaghi-1/+1
Until now we were calling it `Sextractor', but the official way of writing it is `SExtractor'. With this commit, this has been corrected.
2019-09-26Working project when downloaded from arXivMohammad Akhlaghi-14/+23
Until now, we were assuming that the users would just clone the project in Git. But after submitting arXiv:1909.11230, and trying to build directly from the arXiv source, I noticed several problems that wouldn't allow users to build it automatically. So I tried the build step by step and was able to find a fix for the several issues that came up. The scripting parts of the fix were primarily related to the fact that the unpacked arXiv tarball isn't under version control, so some checks had to be put there. Also, we wanted to make it easy to remove the extra files, so an extra `--clean-texdit' option was added to `./project'. Finally, some manual corrections were necessary (prior to running `./project', which are now described in `README.md'. Most of the later steps can be automated and we should do it later, I just don't have enough time now.
2019-09-25Won't copy previous distribution builds in new distributionMohammad Akhlaghi-1/+1
Until now, the pipeline was instructed to only ignore the current temporary project distribution directory. So if there were directories from previous builds, they were wrongly included in the current tarball. With this commit, we don't just ignore the directory of the current distribution, but generally, all directories starting with `paper-v*'.
2019-09-16Configure script won't allow build directory to be under sourceMohammad Akhlaghi-4/+28
Users that are not familiar with the file structure of the project may specify the current directory (to-level source directory) as their build-directory. This will cause a crash right after answering the questions, where `rm' will complain about `tex/build' not being deleted because it exists as a directory. To avoid such confusing situtations, the configure script now checks if the build directory is actually a sub-directory of the source. If it is, it will complain with a short message and abort. Also, a `CAUTION' statment has been put in the initial description, right ontop of the question. This bug was reported Carlos Allende Prieto and David Valls-Gabaud.
2019-09-16Git checksum printed even when on a tagMohammad Akhlaghi-2/+2
Until now, when the commit was tagged, `git describe' would just print the tag and no longer the commit checksum. This is bad because the checksum is a much more robust way to confirm the point in history. With this commit the `--long' option has been added to `git describe' to fix this issue. From now on, when we are on a tag, it will print the tag followed by a `-0-' and the first characters of the checksum.
2019-09-16Distribution tarball now builds in arXivMohammad Akhlaghi-8/+25
`./project make dist' will package all the LaTeX-specific files (and analysis source files) into one `tar.gz' file that is ready to upload to servers like arXiv. However, it wasn't updated for some time, so running it would complain about not having a `configure' script in the top of the project. With this commit, it now works with the new file-structure of the project and also copies all the BibLaTeX source files and `paper.bbl' into the top tarball directory, which allows arXiv to build the paper as intended. The output of `./project make dist' has been uploaded and tested on arXiv and it is built by arXiv perfectly. Also, a short description of all the special `make' targets was added to the output of `./project --help'.
2019-09-15SExtractor BibTex entry updated with ADS's exported formatMohammad Akhlaghi-10/+13
Until now the SExtractor BibTeX entry has the first-name and family name in the same field, while the family-name should be in curly braces. Also, it was missing an ADS URL. So the default ADS generated BibTeX entry for SExtractor is now used instead of the old one.
2019-09-06Updated versions of several softwareMohammad Akhlaghi-59/+61
New versions of astropy, bash, cmake, curl, findutils, gawk, gcc, ghostscript, git, make, gsl had recently come so they are updated with this commit. About GNU Findutils and GNU Make: I was bootstrapping (building the tarball of) these two separately separately because their standard tarball release had problems on some systems. Both have been updated now so I am no longer using my own webpage as their main URL. A special note about GNU Make. I just noticed that during bootstrapping, GNU Make would use the fixed version string of `4.2.90' for any commit!!! But fortunately they have officially released their 4.2.90 version, so we are safely using their own webpage. The only difference is the compression format. My old bootstrapped build was `tar.lz', but the standard release is `tar.gz'. Also, all the basic programs (installed in `.local/bin') in `basic.mk' are now existance-only dependencies (after a `|'). Because later programs just use them at a very basic level, so there is no need to rebuild everything when Bash gets updated for example.
2019-08-22OpenMPI environment variable used to disable need for OpenSSHMohammad Akhlaghi-3/+11
Until now, OpenMPI would complain about not having `ssh' or `rsh' as a remote shell feature. However, such features should not be necessary in a reproducible scenario and they also have major security issues. With this commit, we are now using OpenMPI's `OMPI_MCA_plm_rsh_agent' environment variable to disable any remote shell dependency for it (as suggested by Boud). Therefore, any dependency for OpenSSH has been removed. But I thought to keep the build instructions incase it may be useful under some un-foreseen scenario. However, to discourage people from building it, a notice was added ontop of the build instructions. This bug was found, tested and solved thanks to Roberto Baena Gallé and Boud Roukema. This fixes bug #56724.
2019-08-22Minor typo corrected in commentsMohammad Akhlaghi-3/+3
Until now we have followed the convention of using space characters before comment lines in recepies, not tabs. This has been corrected in one case.
2019-08-22Adding fpack and funpack in the installation of CFITSIORaul Infante-Sainz-2/+4
Until this commit, `fpack' and `funpack' were not installed by default with the installation of CFITSIO. It is necessary to explicity do a `make fpack' and `make funpack' to have them installed. With this commit, these two programs have been added.
2019-08-22Fixed typo in beautifulsoup4 Python package checksum nameRaul Infante-Sainz-3/+3
Until this commit, the name of the variable for `beautifulsoup4' checksum was wrong, and because of that, it was not able to install it. With this commit, `beautifulsoup-checksum' has been replaced for `beautifulsoup4-checksum' in the `reproduce/software/make/python.mk' Makefile, and the problem has been fixed. This was not noticed previously because this Python package is only installed when some high level programs are requested to be installed. With this commit the version of `imagemagick' program has been also updated because the previous version is not available in the official website anymore.
2019-08-22Imported fix for libgit2 on macOS, no conflictsMohammad Akhlaghi-1/+5
There were no conflicts in this merge.
2019-08-21Using libgit2 implementation of libconv with the option -DUSE_ICONV=OFFRaul Infante-Sainz-1/+5
When trying to install `libgit2' on my macOS system, it complains about not finding `_iconv*' functions! But apparently `libgit2' has its own implementation of `libiconv' that it uses if it can't find `libiconv' on macOS. So, the solution to this problem was to add `-DUSE_ICONV=OFF' to the configuration options of `libgit2'. I reported this issue that now is fixed thanks to the help of Mohammad Akhlaghi.
2019-08-21Gnuastro's configuration file updated for version 0.10Mohammad Akhlaghi-2/+8
Recently the version of Gnuastro in the template was updated to version 0.10. However, I had forgot to update the `gnuastro.conf' file to fit with the necessary new features of this version. The new general Gnuastro configuration file is now added instead. This bug was reported by Deepak.
2019-08-16Libtiff built without JBIG-KIT supportMohammad Akhlaghi-2/+5
Until now, we weren't explicitly telling libtiff to ignore JBIG-KIT. So on some systems, it would try to link withit and would thus fail. With this commit, we have disabled JBIG-KIT support for libtiff. I tried to import it into the template, but it doesn't have use the standard GNU Build system. Maybe later we can add a set of build rules for it, I don't have time now. Also, this problem with libtiff occurred while building Ghostscript. But was fixed after adding this option to libtiff. So libtiff was added as a dependency of Ghostscript. This bug was reported by Sheeraz Ahmad.
2019-08-16OpenSSH privsep-path now set within project's build directoryMohammad Akhlaghi-1/+2
Until now we weren't setting OpenSSH's `privsep-path' configure option. As a result, it would try to use its default (`/var/empty'). Therefore, when the host doesn't have `/var/empty', OpenSSH would crash because of not having permissions to create this directory. With this commit, we are now using OpenSSH's `--with-privsep-path' configure-time option to explicitly use a directory with the project's build directory. This bug was found by Sheeraz Ahmad and Amina Aahil.
2019-08-15WCSLIB 6.4 doesn't need explicit link with CFITSIOMohammad Akhlaghi-17/+5
This was a bug in WCSLIB 6.3 that has been fixed in WCSLIB 6.4. From WCSLIB's changelog: "The rule change to the Fortran makefile in v6.3 to add getwcstab_f.o to the sharable library causes it to depend on CFITSIO to resolve fits_get_wcstab(). Hence backed out of that change.". The actual error was like this: Undefined symbols for architecture x86_64: "_fits_read_wcstab", referenced from: _ftwcst_ in getwcstab_f.o "_gFitsFiles", referenced from: _ftwcst_ in getwcstab_f.o ld: symbol(s) not found for architecture x86_64
2019-08-08libnsl, libtirpc and rpcsvc installed as non-used biber dependenciesMohammad Akhlaghi-6/+49
These three libraries are dependencies of Biber, so we will need them later, but since we don't build biber from source now, we can't control what library it links with. With this commit, we have just added their versions, checksum, download URL and build rule incase they are useful in other software. Later, when we build Biber (and Texlive in general) from source, we'll be able to use these.
2019-08-08Corrected typo when fixing libffi's installation directoryMohammad Akhlaghi-1/+1
In the line where we checked if libffi is installed in `lib/' or `lib64/', there was a typo that lead to an incorrect result: we had forgot to add a `-f' to the second check. This is corrected with this commit.
2019-08-08Gnuastro's build: removing the `make check' stepMohammad Akhlaghi-2/+1
Until now, unlike most other programs, in Gnuastro we would run `make check', but this causes a crash on some systems, because of its BuildProgram test: a linking error because the library isn't installed yet. Which is natural in cases like this (and must be corrected in future Gnuastro releases). With this commit, the checks of Gnuastro have been removed.
2019-08-08Static PatchELF only built when static C library existsMohammad Akhlaghi-14/+69
Until now, when building PatchELF, we would always require that it be done statically. However, some systems don't have a static C library available for linking. This cause a crash in the static building of PatchELF. But a static PatchELF is necessary for correcting RPATH in GCC's outputs. With this commit, in the configure script we check if a static C library is linkable for the compiler. If it isn't then `host_cc' will be set to 1 and GCC won't be built. We also pass the result of this test to `basic.mk' (through `good_static_lib'), so if a static C library isn't available, it builds a dynamically linked PatchELF. This bug was reported by Elham Saremi.
2019-08-07Removed temporary files to check Fortarn compilerMohammad Akhlaghi-2/+3
Until now, the Fortran compiler check wouldn't delete the files it creates in the temporary software building directory. With this commit, the cleaning steps have been added.
2019-08-06OpenSSH necessary to use OpenMPIMohammad Akhlaghi-2/+15
Until now, OpenMPI was being installed without any dependency. This was fine because it would indeed build. But the moment you tried loading something that depends on it (for example `mpi4py' through `astropy'), you would get an error complaining that SSH isn't present. With this commit, the pipeline now also installs OpenSSH to solve this problem.
2019-08-05WCSLIB needs -lcfitsio on some systemsMohammad Akhlaghi-1/+13
Until now, we were relying on WCSLIB's internal checking and linking with CFITSIO. But on one macOS system (not others that had no problem!), we noticed that it complains with undefined symbol linking errors to CFITSIO libraries. With this commit, as a fast/ugly solution, we are explicity adding `-lcfitsio' to WCSLIB's `LIBS' variable so all binaries are linked with it automatically. We'll be in touch with the WCSLIB author to see if a better solution can be found.
2019-08-05Libffi builds without Java supportMohammad Akhlaghi-1/+2
Until now, we weren't explicitly disabling Java in the build of libffi because there was no Java platform on the systems we tested. But recently Yahya Sefidbakht report a crash in libffi (reported in bug #56716) because of Java not being compiled properly. Unfortunately libffi doesn't have a configure option to disable Java, but it does have an internal macro (`NO_JAVA_RAW_API'). With this commit, we we defined this macro in the `make' step, and it solved the problem. This fixes bug #56716.
2019-08-05Backup repository set to http://akhlaghi.org/reproduce-softwareMohammad Akhlaghi-15/+17
Some of the software tarballs are directly available on their webpage (for example due to build problems on systems, where we had to clone the software and build its tarball ourselves until the next release). Until now, only for these software, these tarballs were hosted on `http://akhlaghi.org/src'. But now, I have moved a clone of the software backup repository (including all its software) to `http://akhlaghi.org/reproduce-software'. To be more clear and have a single place for the backup software, the URL of those software has also been updated in the project source.
2019-08-05Gnuastro updated to version 0.10Mohammad Akhlaghi-2/+2
A new version of Gnuastro was recently released with many improvments and bug fixes, so it is updated here too.
2019-08-05Configuration: Checking that C and Fortran compilers workMohammad Akhlaghi-116/+199
Until now, we were just checking for the existance of a C and Fortran compiler. But it can happen that even if they exist, they don't operate properly, for example see some errors that have been reported until now in P.S. (both on different macOS systems). But finding this source after the programs have started is frustrating for the user. With this commit, before we start building anything, we'll check these two compilers with a simple program and see if they can indeed compile, and if their compiled program can run. If it doesn't work an elaborate error message is printed to help the users navigate to a solution. Also, the building of `flock' within `configure.sh' has been moved just before calling `basic.mk'. This was done so any warning/error message is printed before actually building anything. This fixes bug #56715. P.S. The error messages: C compiler ---------- conftest.c:9:19: fatal error: stdio.h: No such file or directory ^ compilation terminated. ---------- Fortran compiler ---------------- dyld: Library not loaded: @rpath/libisl.10.dylib Referenced from: /path/to/anaconda2/gcc/libexec/gcc/x86_64-apple-darwin15.5.0/4.9.3/f951 Reason: image not found gfortran: internal compiler error: Abort trap: 6 ----------------
2019-08-01Updated version of mpi4py doesn't conflict with OpenMPIMohammad Akhlaghi-14/+5
Until now, in version 3.0.1, mpi4py couldn't be built with the most recent version of OpenMPI. However, after trying the next version (3.0.2), I noticed that it builds successfully without a problem.
2019-08-01Git hooks removed after doing a distcleanMohammad Akhlaghi-0/+5
Until now, when you needed to completely clean a project (with `./project make distclean') the Git hooks that are installed during configure time would cause problems when committing (the `pre-commit' hook in particular won't allow you to commit anything!). With this commit, before deleting the software, the template first removes these Git hooks.
2019-08-01Typo in previous commit correctedMohammad Akhlaghi-1/+0
After making the previous commit, I noticed an extra line (redundantly defining a wrong `BASH_ENV') that should have been deleted. It has been corrected.
2019-08-01Bash startup script for every recipeMohammad Akhlaghi-0/+58
Until now the only way to define the environment of the Make recipes was through the exported Make variables (mostly in `initialize.mk' for the analysis steps for example). However, there is only so much you can do with environment variables! In some situations you want slightly more complicated environment control, like setting an alias or running of scripts (things that are commonly done in the `~/.bashrc' file of users to configure their interactive, non-login shells). With this commit, a `reproduce/software/bash/bashrc.sh' has been defined for this job (which is currently empty!). Every major Make step of the project adds this file as the `BASH_ENV' environment variable, so the shell that is created to execute a recipe first executes this file, then the recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment variable that enables users to limit their envirnoment setup based on the condition it is being setup (in particular in the early phase of `basic.mk', where the user can't make any assumption about the programs and has to write a portable shell script).
2019-07-31GCC: libstdc++ has a link to libiconvMohammad Akhlaghi-0/+1
After adding the libiconv library to the template, the C++ library uses three of its functions (`libiconv', `libiconv_open' and `libiconv_close'). However, it doesn't explicity link with it inside its shared library! I tried by exporting `LIBS=-liconv' before the GCC configure script but it crashed as soon as it went on to the second GCC building stage (because this environment variable was no longer present there). I also tried adding the C++ configure option of `--enable-cstdio' to the GCC configure options (so it doesn't use iconv according to the manual), but it made no change. With this commit, as a brute-force solution, `patchelf --add-needed' is run on the installed `libstdc++.so', so `libiconv.so' is explicitly included inside the `libstdc++' shared library. This bug was found by Roberto Baena Galle while trying to load Matplotlib (which needed the C++ library). This fixes bug #56702.
2019-07-29Checking software tarball checksums before building softwareMohammad Akhlaghi-245/+472
Until now, there was no check on the integrity of the contents of the downloaded/copied software tarballs, we only relied on the tarball name. This could be bad for reproducibility and security, for example on one server the name of a tarball may be the same but with different content. With this commit, the SHA512 checksums of all the software are stored in the newly created `checksums.mk' (similar to how the versions are stored in the `versions.mk'). The resulting variable is then defined for each software and after downloading/copying the file we check to see if the new tarball has the same checksum as the stored value. If it doesn't the script will crash with an error, informing the user of the problem. The only limitation now is a bootstrapping problem: if the host system doesn't already an `sha512sum' executable, we will not do any checksum verification until we install our `sha512sum' (as part of GNU Coreutils). All the tarballs downloaded after GNU Coreutils are built will have their checksums validated. By default almost all GNU/Linux systems will have a usable `sha512sum' (its part of GNU Coreutils after all for a long time: from the Coreutils Changelog file atleast since 2013). This completes task #15347.
2019-07-29Tip added at initial configuration notice on how to see progressMohammad Akhlaghi-0/+10
The configuration step (building all the ncessary software) can take some time. It is natual for the user to want to see how the build is going (which software is being built at every moment). So far, we have only put a "Inspecting status" section in `README-hacking.md' that describes a solution, but some early users may not have read it yet. With this commit a short tip was added in the initial installation notice to inform the user of this very useful command.
2019-07-28Corrected typo in environment before running makeMohammad Akhlaghi-3/+3
We recently moved the system's `rm' program absolute address to a shell variable that is found during the `./project' script. But I had forgot to account for the difference between the Make and Bash variable naming differences. I had also forgot to add a value to the HOME variable. With this commit both are corrected: the system's `rm' path is now called `sys_rm' and the HOME variable is set.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-32/+1269
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-07-27DYLD_LIBRARY_PATH also fixed for macOS systemsMohammad Akhlaghi-23/+26
Until now we were only setting the `LD_LIBRARY_PATH' environment variable for GNU/Linux systems. But macOS systems use the `DYLD_LIBRARY_PATH'. With this commit, for better control over the environment, we are also fixing `DYLD_LIBRARY_PATH' in all the places that we are setting the general environment variables.
2019-07-26PatchELF built statically and updatedMohammad Akhlaghi-2/+8
Until now, like all other software, PatchELF would install with dynamic linking. However, PatchELF links with `libstdc++' so on one system, I noticed that PatchELF gives a segmentation fault and corrupts `libstdc++' while correcting its RPATH (after installing GCC). The solution is to build PatchELF statically. With this commit, we force PatchELF to be built statically (it only installs on GNU/Linux systems anyway, so there is no problem with static linking on macOS). This solved the problem on that system. While looking at its documentation, I also noticed that a new version of PatchELF has been released after almost three years, so it has been updated in the template also. This fixes bug #56673.
2019-07-25Updated version of WCSLIB to 6.3, previously 6.2Mohammad Akhlaghi-3/+3
Some bugs have been fixed in the new version of WCSLIB, so it has been updated in the template.
2019-07-24libgit2 updated to version 0.28.2, from 0.26.0Mohammad Akhlaghi-3/+3
More than two releases and bug fixes have been made to libgit2. So we are now using a more recent version in the template.
2019-07-15Minor corrections in configure and high-levelMohammad Akhlaghi-4/+5
Configure script: when `texlive-ready-tlmgr' is not created, it is similar to not having installed TeXLive. A check was added so in this scenario the `./configure' script doesn't crash. high-level.mk: `cairo' and `pixman' are now installed in parallel and with `V=1' (so the full compilation and linking command is printed).
2019-07-08Corrected typo in the installation of libgit2Mohammad Akhlaghi-1/+1
In one of the last few commits, the commands in the recipe of libgit2 was merged with `&&' so it stops if anything fails. But I had forgot to add a `;' at the end of the `install_name_tool' command. This is corrected with this commit.
2019-07-08Rpath in libstdc++ and passing host_cc to high-levelMohammad Akhlaghi-1/+1
Until this commit, the addion of `-liconv' to `CXXFLAGS' in `high-level.mk' dependend on `host_cc', but I had forgot that `host_cc' isn't defined for `high-level.mk'. It is now defined for this Makefile also in the configure script. Also, the Standard C++ library depends on `libgcc_s.so.1', so after building GCC, it was necessary to add Rpath to it.
2019-07-07Minor corrections in high-level programsMohammad Akhlaghi-23/+24
While doing a clean build, several issues were found in the pipeline and corrected. The main issue was that with the recent installation of `libiconv', the GCC standard C++ library depends on `libiconv', so we need to explicity add an `-liconv' to any C++ compilation. The other corrected points are: - The C++ compiler is now explicitly defined in `CXX'. - libgit2 and WCSLIB's recipes weren't using `&&' between statements, so if there was an error, it would still build the target! - The CMake bootstrapping script now builds much faster in parallel.