| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, we were using `flock' (file-lock) for downloading the input
datasets in series. But we couldn't do this when downloading the software
tarballs because `flock' wasn't yet available. Generally, unlike
processing, downloading is much better done in series than in parallel.
To enable serial downloads of the software also, with this commit we are
installing `flock' in the configure script (not in a Makefile). As a
result, besides `flock', we can also benefit from the other good features
of the `reproduce/src/bash/download-multi-try' script *(for example
attempting download again after some time).
Some GNU mirrors may have problems at the time of download, so with this
commit, we are using the main GNU FTP server for GNU programs.
|
|
Until this commit, we had some of the python packages intalled
but they did not work properly because of the `PYTHONPATH' variables.
That is, the pipeline's `python' was the `python' of the system
instead of the pipeline's `python'.
With this commit this issue has been fixed by setting the correct
`PYTHONPATH'. In this commit we also modify the installation of
`bzip2' because `CMake' was complaining about some libraries built
statically.
|
|
In the libpng installation there was `ilibdir' instead of `ilidir'.
|
|
Until now the installation of Python and its packages (numpy, astropy,
astroquery, etc.) were done in the same `makefile'.
With this commit the installation of Python and its packages have been
split and now it is independent of the other programs. The installation
of all Python packages needs to be written explicitely because pip is
not used anymore.
|
|
All dependencies for building astroquery package have been done.
Until nowthe Python dependencies were built in the same Makefile
as the high level libraries and programs. But, because astroquery
has many dependencies we split the Python and Python packages
installation in a new Makefile.
The installation of differents packages are done using Python and
not pip, because we found some problems when doing it with pip.
Apparently there are some interferences between the packages
installed by the pip of the system and the pip installed as part
of Python in the pipeline.
|
|
Raul Infante-Sainz added the building of Python (along with the Numpy and
Astropy packages) into the pipeline. That work is now being merged into the
main pipeline branch.
There was only this small problem that needed to be fixed: the Python
tarball's name after unpacking is actually `Python-X.X.X' (with a captial
P), not `python-X.X.X'. This has been corrected with this merge.
|
|
The zip program wasn't placed correctly (in alphabetical order) and its URL
command had the wrong indentation! Both have no effect at all on the
processing and are only cosmetic (to help in readability).
|
|
Astropy was added and one very important thing is that we have to
use the pypi tarball (https://pypi.org/) (which is bootstrapped)
and not the github tarball.
|
|
Python needs some packages to be really useful. Numpy is the most
important package for using Python and a lot of other packages
depend on it.
In this commit we add numpy to the pipeline. The tarball of numpy
right now is fossies.
|
|
Many projects use Python so it is necessary include it in the
pipeline.
|
|
With this commit, it is now possible to package the project into a tarball
or zip file, ready to be distributed to collaborators who only want to
modify the final paper (and not do the analysis technicalities), or for
uploading to sites like arXiv, or online LaTeX sharing pages.
|
|
I recently found another fork of metastore that allows its build on macOS
systems (https://github.com/mpctx/metastore). So I forked it into my own
fork with several other corrections (mostly cosmetic!), so it is now much
better suited for this pipeline.
Raul Infante-Sainz has already tested the building of metastore on his
macOS. In a previous test, we also noticed that libbsd should not be built
on Mac systems, so it is now a conditional prerequisite to metastore.
|
|
While editing files, some editors create temporary `~' files that can cause
problems in metastore's ability to delete their host directory if its not
on the other branch. With this commit, a `find' call was added to the post
checkout Git hook to remove such temporary files before metastore is
called.
Also, some comments were added to both git hooks to make them easier to
understand for a beginner.
|
|
Two corrections were made in the Git hooks of Metastore.
1) The shebang at the start of the scripts now uses the absolute adress of
our installed bash, not the relative `.local/bin/bash'. Note that it is
possible to use Git within subdirectories and in that scenario, the
`.local' will fail.
2) The `$$user' section was removed from the command to find the user's
group. With the user as an argument, `groups' may print the user's name
first, then their list of groups. When this happens, the script would
be just repeating the user's name. But the raw `groups' command will
list the groups of the running user.
|
|
After testing the built of Metastore on a server, I noticed that because
its `/etc/passwd' doesn't have the list of users, the `getpwuid' call
within metastore failed and wouldn't let it finish.
So I looked into the code and was able to implement a solution to this
problem by adding two options to it for default values for the user and
group. Also, file attributes are not necessary in our (current) use case of
metastore and caused crashes on our server, so they are also disabled.
|
|
Metastore depends on `bsd/string.h' to work properly (atleast on GNU/Linux
systems). The first system I tried building with had that library, so I
didn't notice! With this commit, we also build `libbsd' as part of the
pipeline.
Also, I couldn't find libbsd's version in any of its installed headers, so
like Libjpeg, we can't actually check and will directly write our internal
version into the paper.
|
|
The pipeline heavily depends on file meta data (and in particular the
modification dates), for example the configuration-Makefiles within the
pipeline are set as prerequisites to the rules of the pipeline.
However, when Git checks out a branch, it doesn't preserve the meta-data of
the files unique to that branch (for example program source files or
configuration-Makefiles). As a result, the rules that depend on them will
be re-done.
This is especially troublesome in the scenario of this reproducible paper
project because we commonly need to switch between branches (for example to
import recent work in the pipeline into the projects). After some
searching, I think the Metastore program is the best solution. Metastore is
now built as part of the pipeline and through two Git hooks, it is called
by Git to store the original meta-data of files into a binary file that is
version controlled (and managed by Metastore).
|
|
The TIFF library can optionally depend on webp [1] and zstd [2]. But these
aren't commonly used in scientific datasets so to avoid a longer build and
managing of extra dependencies (atleast for now!), we are disabling
them. The problem is that they cause a dependency on the host system and if
they are updated/removed, the relevant pipeline programs will crash.
[1] https://en.wikipedia.org/wiki/WebP
[2] https://en.wikipedia.org/wiki/Zstandard
|
|
Wget and cURL depend on many network related libraries by default and if
they are present on the host operating system, they will be linked
with. This causes problems for the pipeline when these libraries are
updated on the host system.
With this commit, I went through the configure time options of both Wget
and cURL and removed any library that didn't seem related to merely
downloading of files (possibly with SSL, because we do build OpenSSL in the
pipeline).
Also, I noticed a new version of cURL has come, so that is also updated.
|
|
Git needs cURL in its build. Until now, by chance cURL was always built
before Git, but while building this pipeline on a system, Git was built
before cURL and we found the problem.
I also noticed that we hadn't added `Your name <your@email.address>' to the
`for-group' script. This has been corrected now.
|
|
ccache is a super annoying program in the context of the reproduction
pipeline. On systems that use it, the `gcc' and `g++' that are found in
PATH are actually calls to `ccache' (so it can manage their call)!
Two steps have been taken to ignore and disable ccache (if it isn't ignored
properly!): 1) when making symbolic links to compilers, if a directory
containing `ccache' is present in the PATH, it is first removed, then we
look for the low-level programs that we won't be building. 2) The
`CCACHE_DISABLE' environment variable is set to 1 where necessary (with the
other environment variables).
|
|
Since the current implementation of this pipeline officially started in
2018, all the files only had 2018 in their copyright years. This has now
been corrected to 2018-2019.
|
|
Both Gzip and Gnuastro were being bootstrapped personally from their Git
repository until now. But fortunately a new release of both came out last
week and so to make things standard we are now using their standard
tarballs.
I also noticed that we weren't checking the version of Gzip or mentioning
it in the acknowledgement section. This was also corrected.
|
|
While checking the build of the previous commit, a failure happened when
linking `reproduce/build/dependencies/installed/bin/sh' with the built Bash
(because the symbolic link already existed!). So a `-f' flag was added to
`ln' to just change it without complaining.
I also noticed that the Git build was also not in verbose mode. So this has
also been corrected.
|
|
Some problems with using the number of threads in dependency building were
fixed.
|
|
On Mac OS systems, CFITSIO doesn't use path to find the `curl-config'
program (used by to give the library header and linking options), but uses
an absolute path. Therefore the only way we can ask CFITSIO to look into
our build of cURL is to manually change that absolute address.
Also, since all the libraries are now linked dynamically, we don't need the
extra linking flags when building WCSLIB (so it finds CFITSIO).
|
|
Mac OS's `install_name_tool' program's command is broken up into two lines,
but I had forgotten to add a line-break so the command would fail. I didn't
notice it myself because this error only shows up on Mac OS systems that
actually need to parse it.
|
|
The build systems of Libgit2 and WCSLIB on Mac OS does not account for
installation in non-standard addresses: `Libgit2' keeps the absolute
address of its build directory (not the installation directory) and WCSLIB
doesn't write any absolute address at all (so the system uses the first one
it finds).
To address these issues, we are now using Mac OS's `install_name_tool'
program to fix the absolute path within the installed shared library.
Since the version of the library is actually present in its shared library
name, in `dependency-versions.mk' we have also separated these two
libraries so later when their version is changed, we are careful in
correcting the shared library name also.
|
|
In the previous commit, I forgot to actually add some changes to the
staging area before committing an pushing. So some of the changes discussed
in the previous commit and now commited.
|
|
Make builds the dependencies of each package based on the order in the
prerequisites list. So when building in parallel, it can greatly help the
over-all build speed if larger packages are built first. Therefore the
three larger Gnuastro dependencies are now placed at earlier places of the
prerequisites.
|
|
Some high-level programs like Wget and cURL need to be built in shared mode
because they also include dynamic loading of libraries. Therefore, if we
only build the lower-level libraries in static mode, our own build will be
ignored and they will go and find the system's shared libraries to link
with. Because of this, for now, we have manually set the `static_build'
variable in the configure script to `no'.
Also, if the downloader fails, we'll delete the output (an empty file in
the case of Wget) because it interefers with a target definition.
|
|
The TeX Live installer needs Wget to operate smoothly, especially on recent
Mac OS systems that don't have Wget pre-installed. Also, it would be good
for the pipeline to have its own downloader. So with this commit, the
pipeline also installs Wget and OpenSSL which is a dependency.
Many other small changes/fixes were done in this process.
|
|
Thanks to the check by Cristina Martínez, some corrections were made when
we attempt to download and install TeXLive. Further checks and corrections
will be in due time.
|
|
Until now, we were downloading TeX Live's tarball within the same rule that
unpacked it. But this causes problems for situations were it cannot be
downloaded within the pipeline (and manually placed in the tarball
directory). So now, the TeX Live downloader is treated like all the other
downloaders.
|
|
On the Libgit2 webpage, it has recommended to build it statically on Mac
systems. By default we are doing this on Linux systems, but the `-static'
flag failed on Mac. But apparently CMake might be able to deal with the
issue in a different way.
|
|
Thanks to a test build on Raul Infante Sainz's Mac OS computer, we were
able to address some issues and will be trying them after this commit:
a) The LLVM linker on that computer didn't recognize `-rpath-link'! So at
configure time we now check for it and only include it when the linker
recognizes it.
b) CMake corrections: 1) `CMAKE_LIBRARY_PATH' is now defined so CMake can
look in our custom directory to find the necessary libraries. 2) To
build and install the CMake built programs, we now simply use `make'
and `make install'.
c) To avoid particular linking problems with WCSLIB (which has special
problems compared to other libraries), we are now deleting the shared
library version (both on GNU and Mac systems).
|
|
GNU Binutils (which provides the GNU Linker) is not ported to Mac OS
systems. GCC also takes a very long time to build, and if we are to still
have linking problems with LLVM's linker, it would be better to just ignore
GCC also and use the system's C compiler and linker together.
So for the time being, GCC isn't a main target of the basic dependencies
and won't be installed. But we have kept the rules that were checked on a
GNU/Linux operating system.
|
|
The pipeline now installs GCC and all its necessary prerequisites.
|
|
The linker of LLVM version 10.0.0 (clang-1000.11.45.5) doesn't recognize
the `-rpath' linker option! After some searching, apparently it does
recognize `-rpath-link' so we are testing with that now.
|
|
Until now we weren't explicity writing the full path of the dynamic
libraries necessary for linking a program. But now with
`-Wl,-rpath=$(ildir)' we ensure that the linker keeps the address of the
dynamic libraries necessary for linking at linking time, not running
time. Also, `pkg-config' is also built when preparing the basics. Several
other minor corrections were made thanks to the great help of Raúl Infante
Sainz.
|
|
The high-level dependencies are now built without having access to the
system's PATH. To do this, all the necessary software that we aren't
building ourselves are now brought into the installed `bin/' directory
using a symbolic link to the corresponding software on the host. To do
this, it was also necessary to increase the number of basic/low-level
packages that we are building, and add several more (Diffutils and
Findutils).
With this process in place, we now have a list of the exact software
packages that we are not building our selves, enabling easy building of all
such dependencies in the future.
|
|
In most analysis situations (except for simulations), an input dataset is
necessary, but that part of the pipeline was just left out and a general
`SURVEY' variable was set and never used. So with this commit, we actually
use a sample FITS file from the FITS standard webpage, show it (as well as
its histogram) and do some basic calculations on it.
This preparation of the input datasets is done in a generic way to enable
easy addition of more datasets if necessary.
|
|
A new version of the ghostscript package is now available, so the used
version in the pipeline (previously 9.25) has been incremented to 9.26.
|
|
When you point to this project, the `README.md' file is the default file
that opens on GitLab and other online git repositories. Since a
reproduction pipeline project is different from the actual pipeline, its
best for the default text that opens to describe the paper, not the
pipeline. The old `README.md' is also kept, but its now called
`REAME-pipeline.md'.
|
|
When there is a problem in creating the final TeX Live installation, the
previous version of the pipeline would not understand and just finish! We
would later have problems in building the paper.
So the following series of steps were taken: to keep the recipes in a
shorter and easier to understand way, the steps to install TeX Live are now
one rule (that produce `.local/bin/texlive-ready-tlmgr' when its
successful), and the steps to install the necessary packages are in another
rule (that produce `.local/bin/texlive-ready' when its successful).
When control comes back inside configure, if `.local/bin/texlive-ready'
isn't there (something failed during the TeX Live installation, or building
packages), then the whole TeX Live installation directory
(`.local/texlive') will be deleted along with the two output files. This
will help ensure that future steps can check the availablility of a working
TeX Live in the pipeline.
|
|
The system's libraries are no longer used in building the higher-level
dependencies. Also, thanks to Raul Infante Sainz, we found out that Bash's
build script was still removing the extra directory information (not
good!).
|
|
GNU Coreutils are basic programs that can help in the configuration of
higher-level programs. Because of that, it was a dependency of almost all
software built in `dependencies.mk'. To make things more clear, easier to
read and faster (when building in parallel), the building of Coreutils is
now moved to the `dependencies-basic.mk' rules. There, it is built
along-side Bash. Since `dependenceis-basic.mk' is run and completed before
`dependencies.mk', with this, we can be sure that Coreutils is present by
the time we want to build the higher-level programs.
Also, Zlib is now added as a dependency of Git also (it is necessary for
its build).
|
|
Until now, we were building Libtool as a high-level `top-level-programs'
software. But all tools that use the `./configure' script already have a
version of Libtool in them. So ultimately the `libtool' in the PATH is not
used. However, in the case of Gnuastro, we need libtool for running
BulidProgram. So in effect, its a dependency of Gnuastro.
|
|
When the C compiler is not GNU GCC, linking with GNU Binutils is going to
cause problems. So until the time that we can include GCC into this
pipeline, its best to avoid Binutils also.
Also, for building CMake, we were relying on an installed CMake, but now,
we are using its own `./bootstrap' script, so it can be built even if the
host system doesn't have CMake.
Also, for TeX Live, we are now setting a custom file as main target to
avoid complications with symbolic links as targets in Make.
Finally, when the user says they don't want to re-write an existing
configuration file, no extra notices will be printed and the configure
script will immediately start building programs.
|
|
The two README files have been updated to explain the new feature of
downloading and building dependencies.
|
