| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, the software building and analysis steps of the pipeline were
intertwined. However, these steps (of how to build a software, and how to
use it) are logically completely independent.
Therefore with this commit, the pipeline now has a new architecture
(particularly in the `reproduce' directory) to emphasize this distinction:
The `reproduce' directory now has the two `software' and `analysis'
subdirectories and the respective parts of the previous architecture have
been broken up between these two based on their function. There is also no
more `src' directory. The `config' directory for software and analysis is
now mixed with the language-specific directories.
Also, some of the software versions were also updated after some checks
with their webpages.
This new architecture will allow much more focused work on each part of the
pipeline (to install the software and to run them for an analysis).
|
|
All occurances of "pipeline" have been chanaged to "project" or "template"
withint the text (comments, READMEs, and comments) of the template. The
main template branch is now also named `template'.
This was all because `pipeline' is too generic and couldn't be
distinguished from the base, and customized project.
|
|
Until now, the files where the people were meant to change didn't have a
proper copyright notice (for example `Copyright (C) YOUR NAME.'). This was
wrong because the license does not convey copyright ownership. So the name
of the file's original author must always be included and when people
modify it (and add their own copyright-able modifications).
With this commit, the file's original author (and email) are added to the
copyright notice and when more than one person modified a file, both names
have their individual copyright notice.
Based on this, the description for adding a copyright notice in
`README-hacking.md' has also been modified.
|
|
All dependencies for building astroquery package have been done.
Until nowthe Python dependencies were built in the same Makefile
as the high level libraries and programs. But, because astroquery
has many dependencies we split the Python and Python packages
installation in a new Makefile.
The installation of differents packages are done using Python and
not pip, because we found some problems when doing it with pip.
Apparently there are some interferences between the packages
installed by the pip of the system and the pip installed as part
of Python in the pipeline.
|
|
Astropy was added and one very important thing is that we have to
use the pypi tarball (https://pypi.org/) (which is bootstrapped)
and not the github tarball.
|
|
The pipeline heavily depends on file meta data (and in particular the
modification dates), for example the configuration-Makefiles within the
pipeline are set as prerequisites to the rules of the pipeline.
However, when Git checks out a branch, it doesn't preserve the meta-data of
the files unique to that branch (for example program source files or
configuration-Makefiles). As a result, the rules that depend on them will
be re-done.
This is especially troublesome in the scenario of this reproducible paper
project because we commonly need to switch between branches (for example to
import recent work in the pipeline into the projects). After some
searching, I think the Metastore program is the best solution. Metastore is
now built as part of the pipeline and through two Git hooks, it is called
by Git to store the original meta-data of files into a binary file that is
version controlled (and managed by Metastore).
|
|
The build of bash has been made a little cleaner to help in readability and
management of the code.
|
|
The call to SED in `dependencies-build-rules.mk' had the file name before
the options. On some verions of SED, this would cause problems. So the
filename is now given after the options.
|
|
Since the current implementation of this pipeline officially started in
2018, all the files only had 2018 in their copyright years. This has now
been corrected to 2018-2019.
|
|
Until now, we were checking the existance of the `configure' file and if it
wasn't present, we would check for `config' (for OpenSSL which also has a
lower-level "Configure" script that is called by the `config' script). But
after two tests on Mac machines by Raul Infante Sainz and Cristina Martínez
Lombilla, we found out that Mac Os's file names aren't case sensitive and
thus the build wouldn't use `config', but `Configure'.
Now, the exact configuration script can be specified as the 7th argument to
the `gbuild' script. If it isn't given, the standard `configure' name will
be used, but when it is, the given name will be used.
|
|
OpenSSL can't automatically detect the architecture of Mac OS systems, so
as it suggests on its Wiki, it needs some help for doing that. With this
commit, we are checking the build on Mac OS with the presence of `otool'
(Mac OS's linker). If it's there, we'll add the OpenSSL configuration
options suggested by OpenSSL's Wiki.
|
|
To enable easy downloading of HTTPS links with Wget (this pipeline's defaut
downloader), we need a set of trusted CA certificates. Until the time that
we can generate one ourselves, one generic set of trusted CA certificates
is now downloaded like a tarball and placed in the OpenSSL configuration
directory.
With these CA certificates, within the pipeline we can now safely use the
pipeline's own installed Wget.
|
|
Some high-level programs like Wget and cURL need to be built in shared mode
because they also include dynamic loading of libraries. Therefore, if we
only build the lower-level libraries in static mode, our own build will be
ignored and they will go and find the system's shared libraries to link
with. Because of this, for now, we have manually set the `static_build'
variable in the configure script to `no'.
Also, if the downloader fails, we'll delete the output (an empty file in
the case of Wget) because it interefers with a target definition.
|
|
The TeX Live installer needs Wget to operate smoothly, especially on recent
Mac OS systems that don't have Wget pre-installed. Also, it would be good
for the pipeline to have its own downloader. So with this commit, the
pipeline also installs Wget and OpenSSL which is a dependency.
Many other small changes/fixes were done in this process.
|
|
On the Libgit2 webpage, it has recommended to build it statically on Mac
systems. By default we are doing this on Linux systems, but the `-static'
flag failed on Mac. But apparently CMake might be able to deal with the
issue in a different way.
|
|
Thanks to a test build on Raul Infante Sainz's Mac OS computer, we were
able to address some issues and will be trying them after this commit:
a) The LLVM linker on that computer didn't recognize `-rpath-link'! So at
configure time we now check for it and only include it when the linker
recognizes it.
b) CMake corrections: 1) `CMAKE_LIBRARY_PATH' is now defined so CMake can
look in our custom directory to find the necessary libraries. 2) To
build and install the CMake built programs, we now simply use `make'
and `make install'.
c) To avoid particular linking problems with WCSLIB (which has special
problems compared to other libraries), we are now deleting the shared
library version (both on GNU and Mac systems).
|
|
GNU Binutils (which provides the GNU Linker) is not ported to Mac OS
systems. GCC also takes a very long time to build, and if we are to still
have linking problems with LLVM's linker, it would be better to just ignore
GCC also and use the system's C compiler and linker together.
So for the time being, GCC isn't a main target of the basic dependencies
and won't be installed. But we have kept the rules that were checked on a
GNU/Linux operating system.
|
|
Previously the SHELL environment variable was only set in `./configure',
`make', and `make install' after our internal Bash was installed. This
caused problems with Lzip's configure script on older shells (tested on a
Mac OS by Raul Infante Sainz).
So now, before having our own Bash, we set it to a possibly existing Bash
on the system and if that isn't present, just the standard `/bin/sh'.
|
|
The pipeline now installs GCC and all its necessary prerequisites.
|
|
Until now we weren't explicity writing the full path of the dynamic
libraries necessary for linking a program. But now with
`-Wl,-rpath=$(ildir)' we ensure that the linker keeps the address of the
dynamic libraries necessary for linking at linking time, not running
time. Also, `pkg-config' is also built when preparing the basics. Several
other minor corrections were made thanks to the great help of Raúl Infante
Sainz.
|
|
The high-level dependencies are now built without having access to the
system's PATH. To do this, all the necessary software that we aren't
building ourselves are now brought into the installed `bin/' directory
using a symbolic link to the corresponding software on the host. To do
this, it was also necessary to increase the number of basic/low-level
packages that we are building, and add several more (Diffutils and
Findutils).
With this process in place, we now have a list of the exact software
packages that we are not building our selves, enabling easy building of all
such dependencies in the future.
|
|
When the C compiler is not GNU GCC, linking with GNU Binutils is going to
cause problems. So until the time that we can include GCC into this
pipeline, its best to avoid Binutils also.
Also, for building CMake, we were relying on an installed CMake, but now,
we are using its own `./bootstrap' script, so it can be built even if the
host system doesn't have CMake.
Also, for TeX Live, we are now setting a custom file as main target to
avoid complications with symbolic links as targets in Make.
Finally, when the user says they don't want to re-write an existing
configuration file, no extra notices will be printed and the configure
script will immediately start building programs.
|
|
TeX Live is now also downloaded and built by the reproduction
pipeline. Currently on the basic (TeX and LaTeX) source is built but no
extra packages, so the PDF building will fail. We'll add them in the next
commit.
|
|
If the Makefile `$(static_build)' variable in
'/reproduce/src/make/dependencies-build-rules.mk' isn't defined (by
mistake), it will default to blank space, then the Shell will complain
about a bad formatted operator (needing two operands). So an `x' was added
before it and before `yes' which will allow us to safely pass such cases
without a terrible crash.
|
|
The default Mac compiler has problems building static libraries. Since we
are not yet building the GNU C Compiler as part of the pipeline, we'll have
to rely on the host system's compiler. Therefore, a check is now added a
the start of the configure script that will build a minimal program with
the `-static' flag and if it fails, it will print a warning. Afterwards,
none of the dependencies will be built with the `-static' flag.
|
|
To have better control over the build, GNU Binutils, Bzip2, GNU Gzip, and
XZ Utils have also been added to the pipeline. Some other minor cleanups
and fixes were also implemented throughout the process.
|
|
Until now, when a package was to be built statically, we were adding the
`--static' option to `CFLAGS'. This was the wrong place to put it! It
should be in the linking step (thus `LDFLAGS'). Also, based on Bash's
configure script, we are now using the more generic form of `-static'
(single dash, not double dash).
On the other hand, the `--disable-shared' option isn't available in many of
the packages and it is highly redundant with the `-static' option, so it
has been removed to avoid an extra warning in such packages.
|
|
To ensure the easy unpacking and building of the programs, Lzip and Tar are
now also build during the initial setup phase.
Some minor corrections were also applied to make things cleaner and
smoother.
|
|
Until now, we used semicolons in Make's Call function definitions to build
the programs with GNU build system or CMake. Therefore, if any step of the
process failed, the rest would be ignorant to it and pass. Now, we use `&&'
to separate the different processing steps. In this way, we can be sure
that if any of them fails (during configuration, or building for example),
the pipeline will also stop and not continue to the next command (in the
same recipe).
Since the two Make Call functions were identical in the two
`dependencies-basic.mk' and `dependencies.mk', they are now in one file to
be imported in both.
This bug was found by Raul Infante Sainz.
|
