aboutsummaryrefslogtreecommitdiff
path: root/reproduce/src/make/dependencies-basic.mk
AgeCommit message (Collapse)AuthorLines
2019-04-12File is built as a dependency of GCCRaul Infante-Sainz-2/+11
Until now, we did not have `file'. It was in other project, where a problem with `Astrometry-net' software, ends up with the necessity of having `file' into the pipeline. With this commit, we add `file' to the project. Since it is a low level program, it is set in `dependencies-basic.mk' as a prerequisite of GCC.
2019-04-10Imported PatchELF correction, gmp and mpfr also dependencies of AWKMohammad Akhlaghi-63/+53
There weren't any conflicts in this merge. However, while, trying to break a long line into multiple (for better readability), I noticed that the AWK version was mistakenly written as `awk-version' in a few cases, so this was corrected to `gawk-version'. While inspecting the libraries that AWK links to, I noticed that it also links to GNU GMP and GNU MPFR. But since they are built after AWK usually, it was using my host system! So with this commit, AWK has these two libraries as prerequisites. As a result, these two libraries were brought up to the basic program section, and not just GCC prerequisites.
2019-04-10Using bin executable in patchelf for awk and bashRaul Infante-Sainz-11/+12
Until this commit, we were using the target (version number of the program) in the `patchelf' for `awk' and `bash'. This makes an incorrect linking in libraries because the target is not the bin program but just a plain text containing the version number of the program. With this commit we fix this issue by setting in the patchelf of `awk' and `bash' the bin executable, and not the target (version number).
2019-04-08Using Clang on Mac OS systems for pkg-configMohammad Akhlaghi-2/+10
Yahya Sefidbakht reported the following error when building Pkg-config on his Mac OS system (using GCC, not Clang). It is apparently because his version of GCC doesn't support some speical feature on Mac that is necessary to build Glib as part of Pkg-config. With this commit, on Mac systems, for pkg-config we are explicity asking to build with Clang (through the `CC' flag).
2019-04-07GNU M4 now built as a dependency of GNU LibtoolMohammad Akhlaghi-54/+75
On some systems, M4 isn't available, so the linking to the host system fails, as a result, we can't build GNU Libtool. The main reason we weren't building M4 was a bug with the most recent GNU C library (http://lists.gnu.org/archive/html/bug-gnulib/2019-04/msg00004.html). But I found a patch used by Arch Linux which fixes the issue and allows M4 to be built. As a result, the pipeline is now building M4 also and the patched M4 tarball is now uploaded to my own webpage as backup. While doing the steps above, I also noticed that we weren't using a tab at the start of the link definitions of `dependencies-basic.mk'. Although its not necessary, to be consistent, its good for the lines to always start with a tab.
2019-04-07--host-cc configure option to avoid building GCC, M4 mandatoryMohammad Akhlaghi-36/+16
In some cases (specially when debugging the pipeline), its very time-consuming to install GCC. With this commit, a `--host-cc' option has been added to avoid building the C compiler when necessary. The test to see if `sys/cdefs.h' is available on the system (necessary to build GCC) has also been moved to the configure script to print a more visible warning and also use the new `host_cc' variable to let `dependencies-basic.mk' know that GCC shouldn't be built. Finally, we are having problems installing M4 from source, so it has been set as a mandatory dependency.
2019-04-05GCC not building on GNU/Linux system with incomplete complierMohammad Akhlaghi-7/+26
On some GNU/Linux distros, the compiler is separated into `multilib' mode (for 32-bit and 64-bit support) and by default the extra component of the compiler is not installed! In such systems for now, we are just creating symbolic links to the host's compiler (similar to Mac). While testing, I noticed that we weren't passing a "$downloader" option to the downloading script of `dependencies.mk' and `dependencies-python.mk'. Also, I noticed that the Cython and Python-pkg-config packages didn't have setuptools as a dependency! Both have now been fixed. Also, Cython's tarball name is now all small-caps (as in all the other tarballs).
2019-04-05Software acknowledgement section is automatically generatedMohammad Akhlaghi-98/+125
Until now, management of the software names and versions in the paper was done manually (a macro had to be defined in `initialize.mk', then used in `paper.tex', so they had to be manually set in two places). Managing this was not easy. To fix this, with this commit, each software building rule's target is a text file that contains its human-readable name and its version. In the end, the configure script sorts them by their name and writes them into a LaTeX input file that we can easily import as a file into the main paper.
2019-04-04Corrected typo in GCC build recipeMohammad Akhlaghi-3/+2
For the tests, we had just set an absurd value for a test in the GCC build recipe to always fail, but we had forgot to fix it. It is now corrected. Also the order of making `g++' and `gfortran' was reversed for easier readablility (it doesn't matter which one is done first, it only matters that `gcc' be done last).
2019-04-04Numpy and Scipy build on Mac imported into the main branchMohammad Akhlaghi-24/+53
We were developing the build of Numpy and Scipy on Mac in a parallel thread and things seems to be working relatively nice now. There were only two problems: 1) GCC still has some random building issues on Mac. 2) ATLAS shared libraries can't be built on Mac (so we used OpenBLAS to build Numpy and Scipy on both Mac and GNU/Linux). But for now, none of these problems are critical. So, we can progress in one branch. There were only very minor conflicts in the merge.
2019-04-04Using host GCC on Mac, defining compilers in HDF5 libraryRaul Infante-Sainz-9/+10
We wer not able to build `gcc' on Mac, so we are using links to the host compilers. In this commit we also found that on Mac the HDF5 library needs an explicit definition of the compilers.
2019-03-28flock is now built in configure, to allow serial downloadsMohammad Akhlaghi-25/+27
Until now, we were using `flock' (file-lock) for downloading the input datasets in series. But we couldn't do this when downloading the software tarballs because `flock' wasn't yet available. Generally, unlike processing, downloading is much better done in series than in parallel. To enable serial downloads of the software also, with this commit we are installing `flock' in the configure script (not in a Makefile). As a result, besides `flock', we can also benefit from the other good features of the `reproduce/src/bash/download-multi-try' script *(for example attempting download again after some time). Some GNU mirrors may have problems at the time of download, so with this commit, we are using the main GNU FTP server for GNU programs.
2019-03-27GCC is now built on a Mac, not yet ATLASRaul Infante-Sainz-16/+21
Until now, we were simply using the host's GCC for Mac systems. But we found that except for a single step (to fixing `rpath'), it works on Mac!!! So, GCC is now part of the Mac build as well. However, we are still having some problems in building ATLAS on Mac. It works on GNU/Linux, but not in Mac. So for the time being (just temporarily), we are avoiding ATLAS (and thus Scipy) on Mac systems. We just filed an issue on the ATLAS discussion list to hopefully fix the problem soon.
2019-03-25Corrected makelink command to avoid paste commandRaul Infante-Sainz-2/+2
We just noticed that recently the `paste' command on macOS doesn't work with a pipe. So we are now simply using the `tr' command in reverse to re-create the PATH (to find where to link to).
2019-03-22On Mac systems, using a copy of GCC, not a linkRaul Infante-Sainz-3/+4
Until now we were using a symbolic link to replace GCC, but Make doesn't treat symbolic links like files. So it would rebuild the links every time. With this commit, only for GCC on Mac systems, we are actually copying the host's GCC executable to avoid this problem. Also, a wrong comment for cURL was removed.
2019-03-22Imported recent work in the main pipeline, conflicts fixedRaul Infante-Sainz-9/+17
Conflicts in `gcc' build comments and in mentioning software used in paper fixed.
2019-03-21ATLAS and Scipy working on GNU/LinuxRaul Infante-Sainz-2/+9
Numpy needs ATLAS as shared libraries. So we also need to build Python with shared libraries. We also need to define site.cfg for numpy and scipy so we define a master template: `reproduce/config/pipeline/dependency-numpy-scipy.cfg' Also `Openssl' did not have rpath so we added with this commit.
2019-03-20Including ATLAS in the pipeline, not yet completeRaul Infante-Sainz-9/+24
An initial installation of atlas is now included in the pipeline, but we are still trying to make it compile and build smoothly. In the process, we found that GCC also needs some modifications (for example rpath issues).
2019-03-18Reseting path in script to make symbolic links to system programsMohammad Akhlaghi-8/+10
Until recently, there was no problem with the `makelink' script of `dependencies-basic.mk' because it was called on separate recipe lines (and thus separate shells). But recently we added a call to it within a single shell (for GCC on Mac OS systems). So a previous call to it would effect the next call. To fix this, in this commit, we are re-setting PATH to its original value after each call finishes.
2019-03-18No Bzip2 shared libraries on macOS systemsMohammad Akhlaghi-3/+9
Bzip2 has a special/separate Makefile to build shared libraries which didn't work on a macOS. So with this commit, we are allowing Bzip2 shared libraries only on macOS systems. Also, I noticed that macOS's `sed' doesn't have the `-i' option (to do the change in place within the same file). So we are using `-e' to write the changed Makefile in a temporary directory, then rename that.
2019-03-08Using system's GCC on MacMohammad Akhlaghi-49/+63
We still have a few problems with building GCC on a MacOS system. To allow using the pipeline on this operating system, until we find the solution, GCC is only built on non-Mac systems. On Mac, we'll just make a symbolic link to the host's executables.
2019-03-06Imported work on many basic Python modulesMohammad Akhlaghi-49/+73
With the help of Raul, we were able to build many higher-level Python packages to enable the installation of packages like Matplotlib and Astroquery. With this commit, that work is being merged into the master branch.
2019-03-06Astroquery, astropy, matplotlib and numpy are now in the pipelineRaul Infante-Sainz-9/+28
Until this commit, we had some of the python packages intalled but they did not work properly because of the `PYTHONPATH' variables. That is, the pipeline's `python' was the `python' of the system instead of the pipeline's `python'. With this commit this issue has been fixed by setting the correct `PYTHONPATH'. In this commit we also modify the installation of `bzip2' because `CMake' was complaining about some libraries built statically.
2019-02-28GNU GCC is now in the pipelineRaul Infante-Sainz-55/+65
Until now, the pipeline was not installing its own `gcc' but using the system one by making a symbolic link. With this commit, GNU GCC has been added into the pipeline. Right now the installation does not work on Mac OS system beause of some conflicts with `clang', but in principle it should work on GNU Linux distributions.
2019-02-23GCC build rule doesn't depend on BinutilsMohammad Akhlaghi-16/+15
In an attempt to test the GCC build rule (without Binutils, because its too architecture dependent), all the necessary dependencies were moved to GCC (from `ld'). Also `fortran' was also added to the languages supported by GCC. This rule built GCC 8.2.0 nicely on my GNU/Linux system. But `gcc' is still not a final target to built, so the rule is being ignored for now.
2019-02-20Pipeline's Bash and AWK deleted when re-building ncursesMohammad Akhlaghi-0/+7
As in all programs, the build process of ncurses depends on the running shell (Bash) and AWK. At the start of the building of ncurses, we remove its library. But Bash and AWK depend on ncurses to run (this creates a circular dependency). Therefore its necessary to remove the Bash and AWK executables when re-building ncurses. This bug was found by Raul Infante Sainz.
2019-01-23Corrected check for patchelf when building BashMohammad Akhlaghi-6/+9
Until now, the check to see if the patchelf program should be used or not (for GNU/Linux vs. Mac installations) was mistakenly added over the step that we define the `sh' symbolic link, not over the call to patchelf. This is corrected with this commit.
2019-01-22Using fork of metastore to work when getpwuid isn't usableMohammad Akhlaghi-0/+0
After testing the built of Metastore on a server, I noticed that because its `/etc/passwd' doesn't have the list of users, the `getpwuid' call within metastore failed and wouldn't let it finish. So I looked into the code and was able to implement a solution to this problem by adding two options to it for default values for the user and group. Also, file attributes are not necessary in our (current) use case of metastore and caused crashes on our server, so they are also disabled.
2019-01-21Patchelf also built to manually set RPATH in Bash and AWKMohammad Akhlaghi-92/+123
With the current build system, Bash and AWK don't write RPATH into the executables. This causes many problems in the pipeline (for example when using the `$(shell)' function in Make which doesn't have `LD_LIBRARY_PATH'). After consulting the Bash and Make mailing lists, so far, the best solution was to use the Patchelf program to manually write RPATH in these executables. With this commit, Patchelf is now installed in the pipeline and used in Bash and AWK to fix this problem.
2019-01-20Cleaner, more managable, build of BashMohammad Akhlaghi-5/+9
The build of bash has been made a little cleaner to help in readability and management of the code.
2019-01-20Corrected copyright typo in dependencies-basic.mkMohammad Akhlaghi-1/+2
In the previous commit, the copyright year and owner were mistakenly modified. They are corrected now.
2019-01-20Bash is built with its own libreadline, not using oursMohammad Akhlaghi-104/+113
While working on a pipeline based on this, I noticed many linking errors of our installed Bash, complaining that it can't link with libreadline. This was while readline was present in the proper directory and the Bash within a recipe would work properly. After some investigation, I found out that this is because Make's `foreach' function (which was used to define the targets) was apparently calling Bash without setting `LD_LIBRARY_PATH', causing this error. To avoid such sitations, Bash now uses its internal build of readline and we no longer ask it to link with the installed readline.
2019-01-20Corrected symbolic links to the extra pkgconfigs of ncursesMohammad Akhlaghi-6/+6
The targets of the links to have the extra common `ncurses' packages were previously just `pkgconfig/*.pc'! But this would only work when run within the `installed/lib' directory, not any other! So the targets for these packages now use an absolute address.
2019-01-15Many network-related libraries ignored in Wget and cURLMohammad Akhlaghi-5/+19
Wget and cURL depend on many network related libraries by default and if they are present on the host operating system, they will be linked with. This causes problems for the pipeline when these libraries are updated on the host system. With this commit, I went through the configure time options of both Wget and cURL and removed any library that didn't seem related to merely downloading of files (possibly with SSL, because we do build OpenSSL in the pipeline). Also, I noticed a new version of cURL has come, so that is also updated.
2019-01-14Links to the built libncursesw corrected after its buildMohammad Akhlaghi-5/+3
In a previous implementation, we were using a `target' variable to define the final target of several links, but with the new `sov' solution, we just used its base name. However, we had forgot to correct two instances of `target'. This is corrected now. Also, the step to clean all already built outputs of the NCURSES library has been simplified to a platform independent wildcard.
2019-01-14Better linking of different NCURSESW namesMohammad Akhlaghi-45/+50
On Mac OS systems, the full version number is not used in the filename given to libncurses. For example for version 6.1, it is called `libncursesw.6.dylib'. So a more generic and easier to maintain and read script is now used to be able to make links for both Mac and GNU/Linux systems. In short, instead of checking if we are in Mac every time, we just set the suffixes at the start based on the machine once as variables and use those to define the links.
2019-01-14GNU NCURSES and GNU Readline also built before GNU BashMohammad Akhlaghi-46/+153
Readline is a prerequisite of Bash and AWK, while NCURSES is a prerequisite of Readline. With the recent update of GNU Bash (and thus GNU Readline) on my host operating system, the pipeline crashed and I noticed this hole in the pipeline. In particular, AWK (which linked with Readline 7.0) would complain about not finding it and abort.
2019-01-10ccache ignored and disabledMohammad Akhlaghi-2/+13
ccache is a super annoying program in the context of the reproduction pipeline. On systems that use it, the `gcc' and `g++' that are found in PATH are actually calls to `ccache' (so it can manage their call)! Two steps have been taken to ignore and disable ccache (if it isn't ignored properly!): 1) when making symbolic links to compilers, if a directory containing `ccache' is present in the PATH, it is first removed, then we look for the low-level programs that we won't be building. 2) The `CCACHE_DISABLE' environment variable is set to 1 where necessary (with the other environment variables).
2019-01-08When installing Bash, check if the sh link is already presentMohammad Akhlaghi-2/+7
After installing Bash, we would just blindly try to build the $(ibdir)/sh' symbolic link. But that could fail if it already existed. To make things clean, we now remove any link first before attempting to make a new one.
2019-01-02Copyright year updated to 2019Mohammad Akhlaghi-1/+1
Since the current implementation of this pipeline officially started in 2018, all the files only had 2018 in their copyright years. This has now been corrected to 2018-2019.
2019-01-01--with-pc-path for pkg-config configurationMohammad Akhlaghi-1/+1
By giving this option specifically at the build time of Pkg-config, we'll ensure that any package that uses pkg-config will first look into our locally installed build.
2018-12-31Updated Gzip and Gnuastro versions to standard buildsMohammad Akhlaghi-1/+1
Both Gzip and Gnuastro were being bootstrapped personally from their Git repository until now. But fortunately a new release of both came out last week and so to make things standard we are now using their standard tarballs. I also noticed that we weren't checking the version of Gzip or mentioning it in the acknowledgement section. This was also corrected.
2018-12-18Forced linking of bin/sh and verbose Make in GitMohammad Akhlaghi-1/+1
While checking the build of the previous commit, a failure happened when linking `reproduce/build/dependencies/installed/bin/sh' with the built Bash (because the symbolic link already existed!). So a `-f' flag was added to `ln' to just change it without complaining. I also noticed that the Git build was also not in verbose mode. So this has also been corrected.
2018-12-18Gzip compiles in verbose mode to be clear and help in debugsMohammad Akhlaghi-1/+1
While we were testing this pipeline on a Mac OS system, we found and reported a problem in Gzip's build (bug #33689). However, since the Gzip build is not verbose, it was necessary to run its `make' with `V=1'. Generally, since almost all the programs are built in verbose mode (where you can see the compilation commands), we have also set this flag in any build to be clear and make it easier to spot bugs in the future.
2018-12-13Fixed numthreads in dependencies alsoMohammad Akhlaghi-3/+3
Some problems with using the number of threads in dependency building were fixed.
2018-12-11Passing -j build options to dependency building bottle-necksMohammad Akhlaghi-3/+10
Some host Make systems may not allow automatic passing of the number of threads to sub-Makes. So while building the basic dependencies, we'll need to explicity add the `-j' option to the Make files that can benefit most from it: those that are dependencies of many others (Tar & Make), or are the last to build (Coreutils).
2018-12-04Shared library absolute address fixed in Libgit2 and WCSLIB on Mac OSMohammad Akhlaghi-1/+2
The build systems of Libgit2 and WCSLIB on Mac OS does not account for installation in non-standard addresses: `Libgit2' keeps the absolute address of its build directory (not the installation directory) and WCSLIB doesn't write any absolute address at all (so the system uses the first one it finds). To address these issues, we are now using Mac OS's `install_name_tool' program to fix the absolute path within the installed shared library. Since the version of the library is actually present in its shared library name, in `dependency-versions.mk' we have also separated these two libraries so later when their version is changed, we are careful in correcting the shared library name also.
2018-12-04Removed already built symbolic links to OS toolsMohammad Akhlaghi-0/+2
`ln' will complain about a link already existing. So to avoid having to rely on the `-f' option (which may not be portable across systems), when we are making symbolic links to the OS tools that we won't be building, we now remove the file if it exists, then make a new symbolic link.
2018-12-04Low-level links nolonger a prerequisite of programsMohammad Akhlaghi-11/+8
Until now the low-level links that we put in our internal installation from the operating system were a prerequisite of essentially all the basic dependencies. So a change in them would mean a full re-build of all the basic dependencies. But in building the basic dependencies, we already have the operating system's PATH and other environment variables. So unlike the higher-level dependencies, they don't need these links at all! With this commit, the `low-level-links' file is placed in `installed/bin' and is a top-level target of the basic dependencies build. In this way, if it is necessary to update/change to use something from the host operating system, we can simply delete it and run `./configure' again wihout having to re-build all the basic dependencies.
2018-12-04Link to `sw_vers' for Mac OS systems also createdMohammad Akhlaghi-0/+3
On Mac OS systems, the `sw_vers' executable prints information about the operating system. It is used by TeX Live to determine the necessary builds to download and install. We are thus importing it as a low-level tool in `dependency-basics.mk'.