| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, we defined `LIBRARY_PATH' to fix the problem of the `ld' linker
of Binutils needing several `*crt*.o' files to run. However, some software
(for example ImageMagick) over-write `LIBRARY_PATH', therefore there is no
other way than to put a link to these necessary files in our local build
directory.
With this commit, we fixed the problem by putting a link to the system's
relevant files in the local library directory. This fixed the problem with
ImageMagick. Later, when we build the GNU C Library in the project, we
should remove this step.
This bug reported by Raul Castellanos Sanchez.
|
|
Until now, the main download script could only check one server for the
given URL. However, ultimately the actual server that a file is downloaded
from is irrelevant for this project: we actually check its
checksum. Especially in the case of software (which are distributed over
many servers), this can usually be very annoying: the servers may not
properly communicate with the running system and even the 10 trials won't
be enough.
With this commit, the download script
`reproduce/analysis/bash/download-multi-try' can take a new optional
argument (a 5th argument). It assumes this argument is a space-separated
list of server(s) to use as backup for the original URL. When downloading
from the original URL fails, it will look into this list and try
downloading the same file from each given server.
|
|
Until now the shell scripts in the software building phase were in the
`reproduce/software/bash' directory. But given our recent change to a
POSIX-only start, the `configure.sh' shell script (which is the main
component of this directory) is no longer written with Bash.
With this commit, to fix that problem, that directory's name has been
changed to `reproduce/software/shell'.
|
|
Until now, the initial project scripts were primarily tested with GNU
Bash. But Bash is not generally available on all systems (it has many
features beyond POSIX). Because of this, effectively we were imposing the
requirement on the user that they must have Bash installed. We recently
started this with setting the shebang of `project' and
`reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul
and Gaspar reported an error on their systems.
To fix the problem, I installed Dash (a minimalist POSIX-compliant shell)
on my computer and temporarily set the shebangs to `/bin/dash', ran the
project configuration step and fixed all issues that came up. With this
commit, it can go all the way to building GCC on my system's Dash. After
this stage (when `high-level.mk' is called), there is no problem, because
we have our own version of GNU Bash and that installed version is used.
Probably some more issues still remain and will hopefully be found in the
future.
While doing this, I also noticed the following two minor issues:
- The `./project configure' option `--input-dir' was not recognized
because it was mistakenly checking `--inputdir'. It has been corrected.
- The test C programs now use the `<<EOF' method instead of `echo'.
- In `basic.mk', the extra space between `syspath' and `:=' was removed
(it was an ancient relic!).
|
|
Until now Perl was built after Coreutils, but I recently noticed that
Coreutils actually uses Perl while creating its manpages. So it is now
built before Coreutils.
Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils
can't build its man page for `md5sum'. The problem was found to be due to
the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our
installed library path in `basic.mk'. Therefore, it would crash because the
server had an older version of OpenSSL than the one that the template's
Coreutils was built with.
In the meantime (while addressing the issues above, because we only had one
thread on the AWS server) I also noticed a few programs that were using a
summarize compilation command (that just prints `CC xxx.c' instead of the
whole command) so I fixed them by adding `V=1'.
This bug was found by Idafen Santana Pérez.
|
|
Until now, the configuration Makefiles (in
`reproduce/software/config/installation' and `reproduce/analysis/config')
had a `.mk' suffix, similar to the workhorse Makefiles. Although they are
indeed Makefiles, but given their nature (to only keep configuration
parameters), it is confusing (especially to early users) for them to also
have a `.mk' (similar to the analysis or software building Makefiles).
To address this issue, with this commit, all the configuration Makefiles
(in those directories) are now given a `.conf' suffix. This is also assumed
for all the files that are loaded.
The configuration (software building) and running of the template have been
checked with this change from scratch, but please report any error that may
not have been noticed.
THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS
FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE
SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES
ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
|
|
GNU Make 4.3 was just announced, so I have updated it here is well. This
was important because until now the installable version was in alpha-mode
(4.2.90), now its a stable version.
|
|
Until now, GCC wouldn't build properly on Debian-based operating systems
because `ld' needed to link with several necessary C library features like
`crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to
add the directory containing them to `LIBRARY_PATH'. In the previous
commit, I actually searched for these files, but while testing on another
system, I noticed that it can be problematic (other architectures may
exist).
With this commit, we are actually finding the build architecture of the
running GCC (which is the same as the `ld') and using that to fix a fixed
directory to `LIBRARY_PATH'.
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
Until now, while building Perl, the `-Dlddlflags="-shared $$LDFLAGS"'
option was being used for all systems. However, this caused `symbol(s) not
found for architecture x86_64' problems on macOS systems.
With this commit, this configure option is only passed for GNU/Linux
systems. With it, Perl can be built nicely on both GNU/Linux and macOS.
|
|
Until now when building Matplotlib on macOS systems, we were using the
default C compiler. However, while Yahya Sefidbakht (previously) and
Mahdieh Nabavi (now) were trying to build the template, on their macOS
using the GNU Compiler Collection (GCC), we found that Matplotlib needs
special macOS headers that GCC doesn't recognize.
With this commit, when Matplotlib is being built on macOS systems, it uses
`clang' and this fixed the problem (so far checked on Mahdieh's machine).
|
|
While working on a different branch to build the GNU C Library, I noticed a
few places in the template that need corrections which are now applied:
1. A new-line character after the "C compiler works" notice at the start
of the configure script.
2. Removing possible `::' in the `LD_LIBRARY_PATH' definition of
`basic.mk'. Note that its not necessary in the other steps because we
don't use any outside-defined `LD_LIBRARY_PATH'.
3. Building GMP for C++ and also with `--enable-fat'.
4. Removing the unpacked Perl tarball directory after its installation.
|
|
Perl is necessary to build Texinfo and later to build LaTeX. Until now we
were just using the host operating system's installation of Perl, but in
some instances that Perl can be too old and not suppor the features
necessary. With this commit, Perl is now built from source during the basic
installation step of the template.
This was reported by Idafen Santana Pérez, after trying the pipeline on an
Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
|
|
Until now, the tarballs were the first normal prerequisite of the
software. As a result if their date changed, the whole software would be
re-built. However, for tarballs specifically, we actually check their
contents with a checksum, so their date is irrelevant (if its newer than
the built program, but has the same checksum, there is no need to re-build
the software).
Also, calling the tarball name as an argument to the building process (for
example `gbuild') was redundant. It is now automatically found from the
list of order-only prerequisites within `gbuild' and `cbuild' (similar to
how it was previously found in the `pybuild' for Python building).
A `README.md' file has also been placed in `reproduce/software/make' to
help describe the shared properties of the software building
Makefiles. This will hopefully grow much larger in the future.
|
|
The tarball of HEALPix includes multiple languages and doesn't include the
ready-to-run GNU Build System by default, we actually have to build the
`./configure' script for the C/C++ libraries. So it was necessary to also
include GNU Autoconf and GNU Automake as prerequisites of HEALPix.
However, the official GNU Autoconf tarball (dating from 2012) doesn't build
on modern systems, so I just cloned it from its source and bootstrapped it
and built its modern tarball which we are using here.
|
|
Until now, we were assuming that the users would just clone the project in
Git. But after submitting arXiv:1909.11230, and trying to build directly
from the arXiv source, I noticed several problems that wouldn't allow users
to build it automatically. So I tried the build step by step and was able
to find a fix for the several issues that came up.
The scripting parts of the fix were primarily related to the fact that the
unpacked arXiv tarball isn't under version control, so some checks had to
be put there. Also, we wanted to make it easy to remove the extra files, so
an extra `--clean-texdit' option was added to `./project'.
Finally, some manual corrections were necessary (prior to running
`./project', which are now described in `README.md'. Most of the later
steps can be automated and we should do it later, I just don't have enough
time now.
|
|
New versions of astropy, bash, cmake, curl, findutils, gawk, gcc,
ghostscript, git, make, gsl had recently come so they are updated with this
commit.
About GNU Findutils and GNU Make: I was bootstrapping (building the tarball
of) these two separately separately because their standard tarball release
had problems on some systems. Both have been updated now so I am no longer
using my own webpage as their main URL.
A special note about GNU Make. I just noticed that during bootstrapping,
GNU Make would use the fixed version string of `4.2.90' for any commit!!!
But fortunately they have officially released their 4.2.90 version, so we
are safely using their own webpage. The only difference is the compression
format. My old bootstrapped build was `tar.lz', but the standard release is
`tar.gz'.
Also, all the basic programs (installed in `.local/bin') in `basic.mk' are
now existance-only dependencies (after a `|'). Because later programs just
use them at a very basic level, so there is no need to rebuild everything
when Bash gets updated for example.
|
|
Until now, when building PatchELF, we would always require that it be done
statically. However, some systems don't have a static C library available
for linking. This cause a crash in the static building of PatchELF. But a
static PatchELF is necessary for correcting RPATH in GCC's outputs.
With this commit, in the configure script we check if a static C library is
linkable for the compiler. If it isn't then `host_cc' will be set to 1 and
GCC won't be built. We also pass the result of this test to `basic.mk'
(through `good_static_lib'), so if a static C library isn't available, it
builds a dynamically linked PatchELF.
This bug was reported by Elham Saremi.
|
|
Some of the software tarballs are directly available on their webpage (for
example due to build problems on systems, where we had to clone the
software and build its tarball ourselves until the next release). Until
now, only for these software, these tarballs were hosted on
`http://akhlaghi.org/src'. But now, I have moved a clone of the software
backup repository (including all its software) to
`http://akhlaghi.org/reproduce-software'.
To be more clear and have a single place for the backup software, the URL
of those software has also been updated in the project source.
|
|
After making the previous commit, I noticed an extra line (redundantly
defining a wrong `BASH_ENV') that should have been deleted. It has been
corrected.
|
|
Until now the only way to define the environment of the Make recipes was
through the exported Make variables (mostly in `initialize.mk' for the
analysis steps for example). However, there is only so much you can do with
environment variables! In some situations you want slightly more
complicated environment control, like setting an alias or running of
scripts (things that are commonly done in the `~/.bashrc' file of users to
configure their interactive, non-login shells).
With this commit, a `reproduce/software/bash/bashrc.sh' has been defined
for this job (which is currently empty!). Every major Make step of the
project adds this file as the `BASH_ENV' environment variable, so the shell
that is created to execute a recipe first executes this file, then the
recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment
variable that enables users to limit their envirnoment setup based on the
condition it is being setup (in particular in the early phase of
`basic.mk', where the user can't make any assumption about the programs and
has to write a portable shell script).
|
|
After adding the libiconv library to the template, the C++ library uses
three of its functions (`libiconv', `libiconv_open' and
`libiconv_close'). However, it doesn't explicity link with it inside its
shared library!
I tried by exporting `LIBS=-liconv' before the GCC configure script but it
crashed as soon as it went on to the second GCC building stage (because
this environment variable was no longer present there). I also tried adding
the C++ configure option of `--enable-cstdio' to the GCC configure options
(so it doesn't use iconv according to the manual), but it made no change.
With this commit, as a brute-force solution, `patchelf --add-needed' is run
on the installed `libstdc++.so', so `libiconv.so' is explicitly included
inside the `libstdc++' shared library.
This bug was found by Roberto Baena Galle while trying to load Matplotlib
(which needed the C++ library).
This fixes bug #56702.
|
|
Until now, there was no check on the integrity of the contents of the
downloaded/copied software tarballs, we only relied on the tarball
name. This could be bad for reproducibility and security, for example on
one server the name of a tarball may be the same but with different
content.
With this commit, the SHA512 checksums of all the software are stored in
the newly created `checksums.mk' (similar to how the versions are stored in
the `versions.mk'). The resulting variable is then defined for each
software and after downloading/copying the file we check to see if the new
tarball has the same checksum as the stored value. If it doesn't the script
will crash with an error, informing the user of the problem.
The only limitation now is a bootstrapping problem: if the host system
doesn't already an `sha512sum' executable, we will not do any checksum
verification until we install our `sha512sum' (as part of GNU
Coreutils). All the tarballs downloaded after GNU Coreutils are built will
have their checksums validated. By default almost all GNU/Linux systems
will have a usable `sha512sum' (its part of GNU Coreutils after all for a
long time: from the Coreutils Changelog file atleast since 2013).
This completes task #15347.
|
|
Until now, to work on a project, it was necessary to `./configure' it and
build the software. Then we had to run `.local/bin/make' to run the project
and do the analysis every time. If the project was a shared project between
many users on a large server, it was necessary to call the `./for-group'
script.
This way of managing the project had a major problem: since the user
directly called the lower-level `./configure' or `.local/bin/make' it was
not possible to provide high-level control (for example limiting the
environment variables). This was especially noticed recently with a bug
that was related to environment variables (bug #56682).
With this commit, this problem is solved using a single script called
`project' in the top directory. To configure and build the project, users
can now run these commands:
$ ./project configure
$ ./project make
To work on the project with other users in a group these commands can be
used:
$ ./project configure --group=GROUPNAME
$ ./project make --group=GROUPNAME
The old options to both configure and make the project are still valid. Run
`./project --help' to see a list. For example:
$ ./project configure -e --host-cc
$ ./project make -j8
The old `configure' script has been moved to
`reproduce/software/bash/configure.sh' and is called by the new `./project'
script. The `./project' script now just manages the options, then passes
control to the `configure.sh' script. For the "make" step, it also reads
the options, then calls Make. So in the lower-level nothing has
changed. Only the `./project' script is now the single/direct user
interface of the project.
On a parallel note: as part of bug #56682, we also found out that on some
macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set
to blank. This is no problem because RPATH is automatically set in macOS
and the executables and libraries contain the absolute address of the
libraries they should link with. But having `DYLD_LIBRARY_PATH' can
conflict with some low-level system libraries and cause very hard to debug
linking errors (like that reported in the bug report).
This fixes bug #56682.
|
|
Until now we were only setting the `LD_LIBRARY_PATH' environment variable
for GNU/Linux systems. But macOS systems use the `DYLD_LIBRARY_PATH'.
With this commit, for better control over the environment, we are also
fixing `DYLD_LIBRARY_PATH' in all the places that we are setting the
general environment variables.
|
|
Until now, like all other software, PatchELF would install with dynamic
linking. However, PatchELF links with `libstdc++' so on one system, I
noticed that PatchELF gives a segmentation fault and corrupts `libstdc++'
while correcting its RPATH (after installing GCC). The solution is to build
PatchELF statically.
With this commit, we force PatchELF to be built statically (it only
installs on GNU/Linux systems anyway, so there is no problem with static
linking on macOS). This solved the problem on that system.
While looking at its documentation, I also noticed that a new version of
PatchELF has been released after almost three years, so it has been updated
in the template also.
This fixes bug #56673.
|
|
Until this commit, the addion of `-liconv' to `CXXFLAGS' in `high-level.mk'
dependend on `host_cc', but I had forgot that `host_cc' isn't defined for
`high-level.mk'. It is now defined for this Makefile also in the configure
script.
Also, the Standard C++ library depends on `libgcc_s.so.1', so after
building GCC, it was necessary to add Rpath to it.
|
|
While testing on a system with no Texinfo, we noticed that M4 depends on
Texinfo. To fix this problem, with this commit, it is now included in the
pipeline.
While doing a clean build, a few minor issues were also found and corrected
in the other rules.
|
|
On some Fedora systems, Wget's build conflicts with libiconv.
With this commit, we use Wget's `--without-libiconv-prefix' configure
option so it will be built without an outside libiconv.
|
|
An existing `libiconv' can cause a conflict with `pkg-config', this is why
`libiconv' depends on `pkg-config'. On a clean build, `pkg-config' is built
first. But when we don't have a clean build (and `libiconv' exists) there
will be a problem.
With this commit, before re-building `pkg-config', we'll remove any
possibly existing installation of `libiconv'.
|
|
There was only a conflict with `.file-metadata'.
|
|
While reviewing Prasenjit's commits, I noticed that we had forgot to add
the citation for TIDES, also to make things clear, the program/library
build rules are now sorted alphabetically.
Finally, I noticed that after building the TiKZ PDF figures, it is crashing
(like on Prasenjit's computer). After looking around, I noticed its because
we were setting the of the `TEXINPUTS' environment variable to be the
installed TeX Live directory (which was ultimately redundant because by
default TeX will look into where it was installed). The important thing is
just that we remove any possible value the host system has, not to set new
directories.
|
|
Libiconv has to be installed before pkg-config otherwise there
would be a compilation error:
gconvert.c:61:2: error: #error GNU libiconv not in use but included
iconv.h is from libiconv
61 | #error GNU libiconv not in use but included iconv.h is from
libiconv
|
|
Several corrections were necessary in the basic build: 1) the
version of GCC on some systems includes an `_' which would cause
a crash when building the PDF. 2) libcharset had to be manually
added to the Git build.
|
|
Prasenjit Saha reported that while building on a macOS, Git complained
about not finding `libiconv' functions. Therefore with this commit,
`libiconv' is also built as a dependency of Git.
Also, since the build of Binutils (and the subsequent GCC) doesn't depend
on the exact versions of the other basic software, they are set as
existance-only prerequsites (ignoring their timestamps). This is done to
avoid re-building when it is not needed.
|
|
Until now, on GNU/Linux systems, GNU Binutils was built in parallel with
other programs. As a result, the moment that the Binutils executables (most
importantly `ld') are being installed, if another program is using them, we
can have a crash.
For some reason, this crash doesn't happen on Fedora or Arch Linux, but on
some Debian-based distors, it causes the following crash when we are
building Git (which happened to be built while Binuntils is being
installed):
credential-cache--daemon.o: file not recognized: File format not recognized
In a following run of `./configure' (when Binutils has been installed), Git
will build and install successfully.
To fix this problem, with this commit, all basic programs are now a
prerequisite of Binutils and Binutils is the sole prerequisite of GCC
(which is the final target of `basic.mk').
Also, the GCC configure options were re-ordered to be easier to read (the
`--with-*' options under each other, the `--enable-*' options under each
other and so on).
|
|
Until now we hadn't noticed that Metastore's build doesn't include an
`rpath'. As a result I got a crash during a Git commit.
With this commit, on GNU/Linux systems, we now add rpath to the installed
Metastore executable. During the build, I also cleaned up the Metastore
build recipe to be more clear and readable (in particular we only check the
user name and group name when metastore is actually built).
|
|
The recent work by Raul to import Imfit is now merged into the master
branch. Only a few minor conflicts came up due to the update with CFITSIO
(that Raul also done in parallel on this branch).
|
|
While trying to debug the installation of Imfit, we found out
that libcurl doesn't have rpath and is thus linking with the
host system's libraries. So rpath is now manually added to the
build of libcurl. We also found out that Imfit executables
don't have rpath, so it was added to them too.
|
|
In the previous commits where Metastore's recipe was updated, all the
`TAB's in the recipe were saved as spaces just before the commit (due to a
copying and pasting from the terminal, not the text editor). With this
commit, I am correcting this.
|
|
Until now, in the recipe to build Metastore, we would store the current
directory in a `current_dir' Bash variable before running `gbuild' to build
Metastore. However, in this Makefile, we aren't using `.ONESHELL'. As a
result, each un-quoted new-line character creates a new shell and the
`current_dir' variable that we used afterwards was empty!
This happened because until recently, Metastore was being built in
`high-level.mk' (where `.ONESHELL' is activated, because we are using GNU
Make to call that Makefile). But in `basic.mk', we are using the host's
Make, which may not be GNU Make, so we can't use any GNU-only features.
Also, this error was hard to notice, because we weren't using `&&' before
writing the final target of Metastore!
With this commit, both this issues are addressed: all the lines in the
recipe are now quoted in the end with a backslash, and the steps before
making the target are separated by `&&', not `;' (which doesn't cause a
crash if the command fails).
This fixes bug #56295.
|
|
Until this commit, Binutils was brough up as a GCC prerequisite two times:
Once as a special (`host_cc') dependency (for when we want to build GCC),
another time as a dependency in every build. The first one is now obsolete:
on MacOS systems where we don't build Binutils, we define its symbolic
links in the same recipe.
With this commit, we remove the first one, so Binutils is always a
dependency of GCC.
|
|
GCC 9.1 was just released and includes many new optimization features that
can benefit projects in this template also.
Also, the build directory of GCC has been moved inside the unpacked
directory (similar to all the other programs). As a result, while GCC is
being built, the `.build/software/build-tmp' directory will only have one
directory for GCC. Until now, there was one `gcc-build' directory and one
unpacked tarball.
|
|
Recently, after the installation of Coreutils, we added a step to manually
run PatchELF over all the contents of the `$(ibdir)'. However, after a
fresh test, this method failed: because while PatchELF was updating `mv',
another program that was using `mv' couldn't do so and crashed.
To fix this problem, Coreutils is now installed in a dummy directory also
and the exact files that need `rpath' are identified and PatchELF is run on
them (to be clean). Also, higher-level `basic.mk' software (higher-level
than Bash), were re-organized so they depend on Coreutils. With this, we
are no sure that when Coreutils is being built, it is the only program that
is being built and PatchELF won't interfere with any other program.
Also, OpenSSL and CMake were set to be bulit in parallel to speed up their
build.
|
|
Especially because of the new convention regarding backslashes, there were
many conflicts that are now fixed. But none were substantial.
|
|
When we need to quote the new-line character we end the line with a
backslash (`\'). Until now, our convention has been to put all such
backslashes under each other to help in visual inspection.
But this causes a lot of confusion in version control: if only one line's
length is larger, the whole block will be marked as changed and thus makes
it hard to visually see the actual change. It also makes debuging the code
(adding some temporary lines) hard.
With this commit, I went through all the files and tried to fix all such
cases so only a single white space character is between the last command
character and the backslash. Where there was an empty line (ending with a
backslash, to help in visually separating the code into blocks), I put the
backslash right under the previous line's.
This completes task #15259.
|
|
Until now, to test if GCC can use `sys/cdefs.h', we were building
a small test program using it. But after testing on an Ubuntu 14.04,
we noticed that the GCC test during the configure script passes, but
GCC still can't be built. After some investigation we noticed its
available in other directories, but during the build of GCC, those
directories aren't used, and it only assumes it to be under
`/usr/include'. So with this commit, we are only checking this
particular location for this header, not a test run of GCC.
After fixing this, we noticed that GCC's build crashed again because
it couldn't link with `libc.a' (or `libc.so'). So we also added a
for this library and added a new warning to inform the user what they
might be able to do.
Finally, we noticed that in one of the last steps of building GCC,
we weren't using `&&', but `;', so the GCC name file would be
built, even when the GCC build failed.
|
|
Even though the Coreutils configure script says that it adds `rpath' to its
executables, its executables don't have it. As a result, it goes to use the
host operating system libraries, causing failures when they don't match
with the template.
Unfortunately after some tests with the configure script, I couldn't find
any way to include `rpath'! Even though `-rpath-link' is present in all
linking commands during Coreutils' build, the installed files still don't
have it!
So I was forced to use PatchELF. However, one problem is that Coreutils
installs many programs, not just one like Bash or AWK.
As a brute-force solution, with this commit, we are running PatchELF on all
the installed programs. It won't hurt those that already have it, but it
will fix all those that don't.
With this commit, I am also making the following non-related small changes:
- In `build-rules.mk', the `&&' characters were placed at the start of the
line for better readability.
- Bash, Readline and NCURSES are now built in parallel on systems that
support it (during `basic.mk').
|
|
Until this commit, `m4' was not a prerequisite of `gmp'. However, during
a test in Ubuntu 14.04 using one single core in the configure step, it
crashed complaining about not having `m4' installed.
With this commit, we set `m4' as a prerequisite of `gmp'.
|
|
With this commit, the copyright information of some files have been
modify in order to include the work of Raul Infante-Sainz.
|
