aboutsummaryrefslogtreecommitdiff
path: root/reproduce/src/make/dependencies-basic.mk
AgeCommit message (Collapse)AuthorLines
2019-03-20Including ATLAS in the pipeline, not yet completeRaul Infante-Sainz-9/+24
An initial installation of atlas is now included in the pipeline, but we are still trying to make it compile and build smoothly. In the process, we found that GCC also needs some modifications (for example rpath issues).
2019-03-08Using system's GCC on MacMohammad Akhlaghi-49/+63
We still have a few problems with building GCC on a MacOS system. To allow using the pipeline on this operating system, until we find the solution, GCC is only built on non-Mac systems. On Mac, we'll just make a symbolic link to the host's executables.
2019-03-06Imported work on many basic Python modulesMohammad Akhlaghi-49/+73
With the help of Raul, we were able to build many higher-level Python packages to enable the installation of packages like Matplotlib and Astroquery. With this commit, that work is being merged into the master branch.
2019-03-06Astroquery, astropy, matplotlib and numpy are now in the pipelineRaul Infante-Sainz-9/+28
Until this commit, we had some of the python packages intalled but they did not work properly because of the `PYTHONPATH' variables. That is, the pipeline's `python' was the `python' of the system instead of the pipeline's `python'. With this commit this issue has been fixed by setting the correct `PYTHONPATH'. In this commit we also modify the installation of `bzip2' because `CMake' was complaining about some libraries built statically.
2019-02-28GNU GCC is now in the pipelineRaul Infante-Sainz-55/+65
Until now, the pipeline was not installing its own `gcc' but using the system one by making a symbolic link. With this commit, GNU GCC has been added into the pipeline. Right now the installation does not work on Mac OS system beause of some conflicts with `clang', but in principle it should work on GNU Linux distributions.
2019-02-23GCC build rule doesn't depend on BinutilsMohammad Akhlaghi-16/+15
In an attempt to test the GCC build rule (without Binutils, because its too architecture dependent), all the necessary dependencies were moved to GCC (from `ld'). Also `fortran' was also added to the languages supported by GCC. This rule built GCC 8.2.0 nicely on my GNU/Linux system. But `gcc' is still not a final target to built, so the rule is being ignored for now.
2019-02-20Pipeline's Bash and AWK deleted when re-building ncursesMohammad Akhlaghi-0/+7
As in all programs, the build process of ncurses depends on the running shell (Bash) and AWK. At the start of the building of ncurses, we remove its library. But Bash and AWK depend on ncurses to run (this creates a circular dependency). Therefore its necessary to remove the Bash and AWK executables when re-building ncurses. This bug was found by Raul Infante Sainz.
2019-01-23Corrected check for patchelf when building BashMohammad Akhlaghi-6/+9
Until now, the check to see if the patchelf program should be used or not (for GNU/Linux vs. Mac installations) was mistakenly added over the step that we define the `sh' symbolic link, not over the call to patchelf. This is corrected with this commit.
2019-01-22Using fork of metastore to work when getpwuid isn't usableMohammad Akhlaghi-0/+0
After testing the built of Metastore on a server, I noticed that because its `/etc/passwd' doesn't have the list of users, the `getpwuid' call within metastore failed and wouldn't let it finish. So I looked into the code and was able to implement a solution to this problem by adding two options to it for default values for the user and group. Also, file attributes are not necessary in our (current) use case of metastore and caused crashes on our server, so they are also disabled.
2019-01-21Patchelf also built to manually set RPATH in Bash and AWKMohammad Akhlaghi-92/+123
With the current build system, Bash and AWK don't write RPATH into the executables. This causes many problems in the pipeline (for example when using the `$(shell)' function in Make which doesn't have `LD_LIBRARY_PATH'). After consulting the Bash and Make mailing lists, so far, the best solution was to use the Patchelf program to manually write RPATH in these executables. With this commit, Patchelf is now installed in the pipeline and used in Bash and AWK to fix this problem.
2019-01-20Cleaner, more managable, build of BashMohammad Akhlaghi-5/+9
The build of bash has been made a little cleaner to help in readability and management of the code.
2019-01-20Corrected copyright typo in dependencies-basic.mkMohammad Akhlaghi-1/+2
In the previous commit, the copyright year and owner were mistakenly modified. They are corrected now.
2019-01-20Bash is built with its own libreadline, not using oursMohammad Akhlaghi-104/+113
While working on a pipeline based on this, I noticed many linking errors of our installed Bash, complaining that it can't link with libreadline. This was while readline was present in the proper directory and the Bash within a recipe would work properly. After some investigation, I found out that this is because Make's `foreach' function (which was used to define the targets) was apparently calling Bash without setting `LD_LIBRARY_PATH', causing this error. To avoid such sitations, Bash now uses its internal build of readline and we no longer ask it to link with the installed readline.
2019-01-20Corrected symbolic links to the extra pkgconfigs of ncursesMohammad Akhlaghi-6/+6
The targets of the links to have the extra common `ncurses' packages were previously just `pkgconfig/*.pc'! But this would only work when run within the `installed/lib' directory, not any other! So the targets for these packages now use an absolute address.
2019-01-15Many network-related libraries ignored in Wget and cURLMohammad Akhlaghi-5/+19
Wget and cURL depend on many network related libraries by default and if they are present on the host operating system, they will be linked with. This causes problems for the pipeline when these libraries are updated on the host system. With this commit, I went through the configure time options of both Wget and cURL and removed any library that didn't seem related to merely downloading of files (possibly with SSL, because we do build OpenSSL in the pipeline). Also, I noticed a new version of cURL has come, so that is also updated.
2019-01-14Links to the built libncursesw corrected after its buildMohammad Akhlaghi-5/+3
In a previous implementation, we were using a `target' variable to define the final target of several links, but with the new `sov' solution, we just used its base name. However, we had forgot to correct two instances of `target'. This is corrected now. Also, the step to clean all already built outputs of the NCURSES library has been simplified to a platform independent wildcard.
2019-01-14Better linking of different NCURSESW namesMohammad Akhlaghi-45/+50
On Mac OS systems, the full version number is not used in the filename given to libncurses. For example for version 6.1, it is called `libncursesw.6.dylib'. So a more generic and easier to maintain and read script is now used to be able to make links for both Mac and GNU/Linux systems. In short, instead of checking if we are in Mac every time, we just set the suffixes at the start based on the machine once as variables and use those to define the links.
2019-01-14GNU NCURSES and GNU Readline also built before GNU BashMohammad Akhlaghi-46/+153
Readline is a prerequisite of Bash and AWK, while NCURSES is a prerequisite of Readline. With the recent update of GNU Bash (and thus GNU Readline) on my host operating system, the pipeline crashed and I noticed this hole in the pipeline. In particular, AWK (which linked with Readline 7.0) would complain about not finding it and abort.
2019-01-10ccache ignored and disabledMohammad Akhlaghi-2/+13
ccache is a super annoying program in the context of the reproduction pipeline. On systems that use it, the `gcc' and `g++' that are found in PATH are actually calls to `ccache' (so it can manage their call)! Two steps have been taken to ignore and disable ccache (if it isn't ignored properly!): 1) when making symbolic links to compilers, if a directory containing `ccache' is present in the PATH, it is first removed, then we look for the low-level programs that we won't be building. 2) The `CCACHE_DISABLE' environment variable is set to 1 where necessary (with the other environment variables).
2019-01-08When installing Bash, check if the sh link is already presentMohammad Akhlaghi-2/+7
After installing Bash, we would just blindly try to build the $(ibdir)/sh' symbolic link. But that could fail if it already existed. To make things clean, we now remove any link first before attempting to make a new one.
2019-01-02Copyright year updated to 2019Mohammad Akhlaghi-1/+1
Since the current implementation of this pipeline officially started in 2018, all the files only had 2018 in their copyright years. This has now been corrected to 2018-2019.
2019-01-01--with-pc-path for pkg-config configurationMohammad Akhlaghi-1/+1
By giving this option specifically at the build time of Pkg-config, we'll ensure that any package that uses pkg-config will first look into our locally installed build.
2018-12-31Updated Gzip and Gnuastro versions to standard buildsMohammad Akhlaghi-1/+1
Both Gzip and Gnuastro were being bootstrapped personally from their Git repository until now. But fortunately a new release of both came out last week and so to make things standard we are now using their standard tarballs. I also noticed that we weren't checking the version of Gzip or mentioning it in the acknowledgement section. This was also corrected.
2018-12-18Forced linking of bin/sh and verbose Make in GitMohammad Akhlaghi-1/+1
While checking the build of the previous commit, a failure happened when linking `reproduce/build/dependencies/installed/bin/sh' with the built Bash (because the symbolic link already existed!). So a `-f' flag was added to `ln' to just change it without complaining. I also noticed that the Git build was also not in verbose mode. So this has also been corrected.
2018-12-18Gzip compiles in verbose mode to be clear and help in debugsMohammad Akhlaghi-1/+1
While we were testing this pipeline on a Mac OS system, we found and reported a problem in Gzip's build (bug #33689). However, since the Gzip build is not verbose, it was necessary to run its `make' with `V=1'. Generally, since almost all the programs are built in verbose mode (where you can see the compilation commands), we have also set this flag in any build to be clear and make it easier to spot bugs in the future.
2018-12-13Fixed numthreads in dependencies alsoMohammad Akhlaghi-3/+3
Some problems with using the number of threads in dependency building were fixed.
2018-12-11Passing -j build options to dependency building bottle-necksMohammad Akhlaghi-3/+10
Some host Make systems may not allow automatic passing of the number of threads to sub-Makes. So while building the basic dependencies, we'll need to explicity add the `-j' option to the Make files that can benefit most from it: those that are dependencies of many others (Tar & Make), or are the last to build (Coreutils).
2018-12-04Shared library absolute address fixed in Libgit2 and WCSLIB on Mac OSMohammad Akhlaghi-1/+2
The build systems of Libgit2 and WCSLIB on Mac OS does not account for installation in non-standard addresses: `Libgit2' keeps the absolute address of its build directory (not the installation directory) and WCSLIB doesn't write any absolute address at all (so the system uses the first one it finds). To address these issues, we are now using Mac OS's `install_name_tool' program to fix the absolute path within the installed shared library. Since the version of the library is actually present in its shared library name, in `dependency-versions.mk' we have also separated these two libraries so later when their version is changed, we are careful in correcting the shared library name also.
2018-12-04Removed already built symbolic links to OS toolsMohammad Akhlaghi-0/+2
`ln' will complain about a link already existing. So to avoid having to rely on the `-f' option (which may not be portable across systems), when we are making symbolic links to the OS tools that we won't be building, we now remove the file if it exists, then make a new symbolic link.
2018-12-04Low-level links nolonger a prerequisite of programsMohammad Akhlaghi-11/+8
Until now the low-level links that we put in our internal installation from the operating system were a prerequisite of essentially all the basic dependencies. So a change in them would mean a full re-build of all the basic dependencies. But in building the basic dependencies, we already have the operating system's PATH and other environment variables. So unlike the higher-level dependencies, they don't need these links at all! With this commit, the `low-level-links' file is placed in `installed/bin' and is a top-level target of the basic dependencies build. In this way, if it is necessary to update/change to use something from the host operating system, we can simply delete it and run `./configure' again wihout having to re-build all the basic dependencies.
2018-12-04Link to `sw_vers' for Mac OS systems also createdMohammad Akhlaghi-0/+3
On Mac OS systems, the `sw_vers' executable prints information about the operating system. It is used by TeX Live to determine the necessary builds to download and install. We are thus importing it as a low-level tool in `dependency-basics.mk'.
2018-12-04Configure script for GNU Build system now an input argumentMohammad Akhlaghi-2/+3
Until now, we were checking the existance of the `configure' file and if it wasn't present, we would check for `config' (for OpenSSL which also has a lower-level "Configure" script that is called by the `config' script). But after two tests on Mac machines by Raul Infante Sainz and Cristina Martínez Lombilla, we found out that Mac Os's file names aren't case sensitive and thus the build wouldn't use `config', but `Configure'. Now, the exact configuration script can be specified as the 7th argument to the `gbuild' script. If it isn't given, the standard `configure' name will be used, but when it is, the given name will be used.
2018-12-03Checking Mac OS host for configuring OpenSSLMohammad Akhlaghi-0/+12
OpenSSL can't automatically detect the architecture of Mac OS systems, so as it suggests on its Wiki, it needs some help for doing that. With this commit, we are checking the build on Mac OS with the presence of `otool' (Mac OS's linker). If it's there, we'll add the OpenSSL configuration options suggested by OpenSSL's Wiki.
2018-12-03Added rpath in basic dependencies, remove input if download failsMohammad Akhlaghi-3/+6
Until now, we weren't including the `rpath' linking options to the basic dependencies. They are now added. Also, when the download of an input file fails for any reason, an empty file won't be replaced there any more.
2018-12-03Minor typo correction in building CoreutilsMohammad Akhlaghi-1/+1
We were missing a `\' at the end of the `$(call' function of Coreutils to connect the two lines. It has been fixed now.
2018-12-03Trusted CA certificates also downloaded for Wget usageMohammad Akhlaghi-8/+18
To enable easy downloading of HTTPS links with Wget (this pipeline's defaut downloader), we need a set of trusted CA certificates. Until the time that we can generate one ourselves, one generic set of trusted CA certificates is now downloaded like a tarball and placed in the OpenSSL configuration directory. With these CA certificates, within the pipeline we can now safely use the pipeline's own installed Wget.
2018-12-03Preference for shared library linkingMohammad Akhlaghi-24/+35
Some high-level programs like Wget and cURL need to be built in shared mode because they also include dynamic loading of libraries. Therefore, if we only build the lower-level libraries in static mode, our own build will be ignored and they will go and find the system's shared libraries to link with. Because of this, for now, we have manually set the `static_build' variable in the configure script to `no'. Also, if the downloader fails, we'll delete the output (an empty file in the case of Wget) because it interefers with a target definition.
2018-12-02Wget and OpenSSL now installed as a basic dependencyMohammad Akhlaghi-33/+93
The TeX Live installer needs Wget to operate smoothly, especially on recent Mac OS systems that don't have Wget pre-installed. Also, it would be good for the pipeline to have its own downloader. So with this commit, the pipeline also installs Wget and OpenSSL which is a dependency. Many other small changes/fixes were done in this process.
2018-11-30cURL now also available as downloader with -L flagMohammad Akhlaghi-1/+5
The main reason I wasn't using cURL as a downloading tool was that I wasn't familar with how to ask it to follow a re-direct. But I just found out that its with the `-L' configure time option. So it is now added as a downloader tool to the pipeline.
2018-11-29Ignoring building of GCC for pipelineMohammad Akhlaghi-9/+33
GNU Binutils (which provides the GNU Linker) is not ported to Mac OS systems. GCC also takes a very long time to build, and if we are to still have linking problems with LLVM's linker, it would be better to just ignore GCC also and use the system's C compiler and linker together. So for the time being, GCC isn't a main target of the basic dependencies and won't be installed. But we have kept the rules that were checked on a GNU/Linux operating system.
2018-11-29GCC is now installed by the pipelineMohammad Akhlaghi-39/+120
The pipeline now installs GCC and all its necessary prerequisites.
2018-11-28Changed -rpath to -rpath-link for LLVMMohammad Akhlaghi-0/+5
The linker of LLVM version 10.0.0 (clang-1000.11.45.5) doesn't recognize the `-rpath' linker option! After some searching, apparently it does recognize `-rpath-link' so we are testing with that now.
2018-11-28Better control of shared library linkingMohammad Akhlaghi-6/+22
Until now we weren't explicity writing the full path of the dynamic libraries necessary for linking a program. But now with `-Wl,-rpath=$(ildir)' we ensure that the linker keeps the address of the dynamic libraries necessary for linking at linking time, not running time. Also, `pkg-config' is also built when preparing the basics. Several other minor corrections were made thanks to the great help of Raúl Infante Sainz.
2018-11-26High-level dependencies build without system's PATHMohammad Akhlaghi-36/+107
The high-level dependencies are now built without having access to the system's PATH. To do this, all the necessary software that we aren't building ourselves are now brought into the installed `bin/' directory using a symbolic link to the corresponding software on the host. To do this, it was also necessary to increase the number of basic/low-level packages that we are building, and add several more (Diffutils and Findutils). With this process in place, we now have a list of the exact software packages that we are not building our selves, enabling easy building of all such dependencies in the future.
2018-11-26sh executable now available in PATHMohammad Akhlaghi-0/+15
While working on a research project using this pipeline, I noticed that we don't have any `sh' executable within our PATH. However, some programs (including Gnuastro's configure script, when it is checking for shells to use with Libtool) check and use it. So after building Bash, we also build an `sh' symbolic link to point to the built Bash executable.
2018-11-20Not including system's libraries in high-level dependenciesMohammad Akhlaghi-1/+1
The system's libraries are no longer used in building the higher-level dependencies. Also, thanks to Raul Infante Sainz, we found out that Bash's build script was still removing the extra directory information (not good!).
2018-11-20GNU Coreutils now built in basic dependenciesMohammad Akhlaghi-19/+29
GNU Coreutils are basic programs that can help in the configuration of higher-level programs. Because of that, it was a dependency of almost all software built in `dependencies.mk'. To make things more clear, easier to read and faster (when building in parallel), the building of Coreutils is now moved to the `dependencies-basic.mk' rules. There, it is built along-side Bash. Since `dependenceis-basic.mk' is run and completed before `dependencies.mk', with this, we can be sure that Coreutils is present by the time we want to build the higher-level programs. Also, Zlib is now added as a dependency of Git also (it is necessary for its build).
2018-11-19Removed GNU Binutils, CMake's built with its own bootstrapMohammad Akhlaghi-43/+10
When the C compiler is not GNU GCC, linking with GNU Binutils is going to cause problems. So until the time that we can include GCC into this pipeline, its best to avoid Binutils also. Also, for building CMake, we were relying on an installed CMake, but now, we are using its own `./bootstrap' script, so it can be built even if the host system doesn't have CMake. Also, for TeX Live, we are now setting a custom file as main target to avoid complications with symbolic links as targets in Make. Finally, when the user says they don't want to re-write an existing configuration file, no extra notices will be printed and the configure script will immediately start building programs.
2018-11-19Gzip's tarball in tar.gz instead of tar.lzMohammad Akhlaghi-27/+27
Until now, we were using a customized `tar.lz' tarball for Gzip. But on systems that don't have GNU Tar, this will cause a problem (non-GNU Tar doesn't recognize `.tar.lz'). So to keep things simple, we are using the customized gzip in `tar.gz' format. After the internal build of GNU Tar and Lzip, the default method of unpacking (`tar xf XXXXX.tar.XX') will work nicely on all the standard compression algorithms and we don't have to modify our commands based on the algorithm (nice feature of GNU Tar).
2018-11-18Pipeline also installs TeX live and necessary packagesMohammad Akhlaghi-0/+11
Since the final product of the pipeline is a LaTeX-created PDF file, it was necessary to also have LaTeX within the pipeline. With this commit, TeX Live is also built as part of the configuration and all the necessary packages to build the PDF are also installed and mentioned in the paper along with their versions.
generated by cgit v1.2.1 (git 2.18.0) at 2024-11-19 23:28:29 +0000