| Age | Commit message (Collapse) | Author | Lines |
|---|
|
A few minor conflicts came up that were easily fixed.
|
|
Until now the software configuration parameters were defined under the
`reproduce/software/config/installation/' directory. This was because the
configuration parameters of analysis software (for example Gnuastro's
configurations) were placed under there too. But this was terribly
confusing, because the run-time options of programs falls under the
"analysis" phase of the project.
With this commit, the Gnuastro configuration files have been moved under
the new `reproduce/analysis/config/gnuastro' directory and the software
configuration files are directly under `reproduce/software/config'. A clean
build was done with this change and it didn't crash, but it may cause
crashes in derived projects, so after merging with Maneage, please
re-configure your project to see if anything has been missed. Please let us
know if there is a problem.
|
|
TeXLive recently transitioned from its 2019 version to its 2020 version
thanks to Elham Saremi's trial of the this project. The fact that
traditionally Maneage installs all TeXLive packages in a per-year directory
is very annoying and required an update in the core Maneage system every
year. So I suddently recognized that we can fix this by setting a different
name for the directory holding the release year. This has been implemented
with this commit.
I have also done this change in the main Maneage branch for other projects
to also benefit from this correction.
|
|
It is this time of year again: TeXLive has transitioned to its 2020 release
and the year is imprinted into the installation directory of TeXLive. Until
now, we have had to manually change this year and it caused complications
and was very annoying.
With this commit, the explicit year has been removed from TeXLive's
installation and we now simply put a `maneage' instead of the year. I tried
this on another system and it worked nicely. Until the time that we can
fully install LaTeX packages from source tarballs, this is the best thing
we could do for now.
|
|
Minizip is a dependency of XLSX I/O and until now, I was just using the
most recent version I found (2.9.2), but XLSX I/O is written for the
Minizip 1.x series, not 2.x. Somehow it didn't cause a crash on my
computer!!! I think XLSX I/O's CMake is instructed to look into system
directories by default when it doesn't find the directories in the given
places. And because I had installed Minizip on my operating system, it
did't complain.
Upon trying the build on their systems, Yahya, Raul and Zahra reported a
failure in the build of XLSX I/O which was due the to the problem above (we
were installing the wrong version of Minizip!).
With this commit, this has been fixed by installing the 1.x series of
Minizip (whish is actually installed within zlib!).
This commit was original done by Mohammad Akhlaghi.
|
|
Minizip is a dependency of XLSX I/O and until now, I was just using the
most recent version I found (2.9.2), but XLSX I/O is written for the
Minizip 1.x series, not 2.x. Somehow it didn't cause a crash on my
computer!!! I think XLSX I/O's CMake is instructed to look into system
directories by default when it doesn't find the directories in the given
places. And because I had installed Minizip on my operating system, it
did't complain.
Upon trying the build on their systems, Yahya, Raul and Zahra reported a
failure in the build of XLSX I/O which was due the to the problem above (we
were installing the wrong version of Minizip!).
With this commit, this has been fixed by installing the 1.x series of
Minizip (whish is actually installed within zlib!).
|
|
With this commit, CMake has been updated to its most recent version.
This upgrade has been done because in the installation of XLSX I/O on
macOS laptop, it crashes complaining about C compiler "not able to
compile a simple test program". After a fast search, I found it could be
possible to just use the most recent version of CMake to solve the
problem. But it didn't work. In any case, it is good to have the most
recent version of CMake included.
|
|
A few minor conflicts occurred and were fixed.
|
|
In the last update of Astropy to version 4.0 they removed some things
that the previous version of Astroquery needs. As a consequence, it is
also necessary to update the Astroquery version to be a ble to run with
the Astropy 4.0. With this commit, the update of Astroquery to it most
recent version (0.4) has been done.
|
|
Until now, there was no explanation on an actual analysis phase, therefore
with this commit an example scenario with a readable Makefile is included.
The Data lineage graph was also simplified to both be more readable, and
also to correspond to this new explanation and subMakefile.
Some random edits/typos were also corrected and some references added for
discussion.
|
|
MissFITS is package for manipulating FITS files.
I added it as my first commit to the project for educational
purposes.
|
|
Until now, there was no easy way to read/write `.xlsx' files (Microsoft
Excel spreadsheets) within the template. But XLSX I/O provides to simple
programs and some libraries to easily convert `.xlsx' files to CSV that can
easily be read by any tool.
This has also been implemented in the core template branch.
|
|
XLSX I/O is a very simple and fast program and library for reading and
writing `.xls' and `.xlsx' files (mainly used by Microsoft Excel) to CSV
files. It has two separate executables that can be called for an Excel file
and will output a CSV plain text file that can then be used within the
pipeline with more standard tools.
|
|
Until now, the project would first ask for the basic directories, then it
would start testing the compiler. But that was problematic because the
build directory can come from a previous setting (with `./project configure
-e'). Also, it could confuse users to first ask for details, then suddently
tell them that you don't have a working C library! We also need to store
the CPATH variable in the `LOCAL.conf' because in some cases, the compiler
won't work without it.
With this commit, the compiler checking has been moved at the start of the
configure script. Instead of putting the test program in the build
directory, we now make a temporary hidden directory in the source directory
and delete that directory as soon as the tests are done.
In the process, I also noticed that the copyright year of the two hidden
files weren't updated and corrected them.
|
|
In the last few days I have been writing these two sections in the middle
of other work. But I am making this commit because it has already become a
lot! I am now going onto the description of `./project make'.
|
|
Until now, the configuration Makefiles (in
`reproduce/software/config/installation' and `reproduce/analysis/config')
had a `.mk' suffix, similar to the workhorse Makefiles. Although they are
indeed Makefiles, but given their nature (to only keep configuration
parameters), it is confusing (especially to early users) for them to also
have a `.mk' (similar to the analysis or software building Makefiles).
To address this issue, with this commit, all the configuration Makefiles
(in those directories) are now given a `.conf' suffix. This is also assumed
for all the files that are loaded.
The configuration (software building) and running of the template have been
checked with this change from scratch, but please report any error that may
not have been noticed.
THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS
FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE
SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES
ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
|
|
GNU Make 4.3 was just announced, so I have updated it here is well. This
was important because until now the installable version was in alpha-mode
(4.2.90), now its a stable version.
|
|
Until now, I was writing the paper without the template. But we will soon
be adding a tutorial to the template, and I thought it will be good to have
an example demonstration here too. So I just brought the hole project into
the template structure, allowing us to add the template analysis later when
its ready, and also allowing us to easily reproduce this paper ofcourse
(without having to worry about the host's TeXLive installation.
|
|
The unnecessary parts were removed and the project now runs.
|
|
After a new rebuild of the project, I noticed that we now need to also
build the `mweights' package.
|
|
Newer versions of Astropy package has been released. With this commit,
it has been updated. It has been increased from v3.2.1 to v4.0
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
With this commit, we now have the core R interpretter within the
template. We should later include instructions to install R packages
(possibly in a separate top-level Makefile like Python).
|
|
Recent builds of the template need these three packages to build the PDF.
This was reported by Hamed Altafi.
|
|
Until now we weren't including this package, but Alberto Madrigal recently
reported that his build of the template failed because it needed it.
With this commit, it is now being installed with the template.
|
|
Newer versions of these packages have recently come out with major
improvements, so they have been updated in the template.
|
|
These two packages are necessary to build the GNU C Library.
|
|
Perl is necessary to build Texinfo and later to build LaTeX. Until now we
were just using the host operating system's installation of Perl, but in
some instances that Perl can be too old and not suppor the features
necessary. With this commit, Perl is now built from source during the basic
installation step of the template.
This was reported by Idafen Santana Pérez, after trying the pipeline on an
Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
|
|
When building the log4cxx tarball from its Git history, I noticed that
files with very long names are not packaged by tar (because by default
Automake uses the ancient v7 tar format that only supports file names less
than 99 characters).
So I build the tarball with the `tar-ustar' option to Automake (by
modifying the log4cxx source) and the resulting tarball was able to compile
and run successfully. This has been described above the rule to build
log4cxx and I also sent an email to their developing mailing list to inform
them of this problem. If they address it, I will remove the note on the
necessary corrections.
|
|
Some minor corrections were made in the template:
- When making the distribution, `.swp' files (created by Vim) are also
removed.
- Autoconf is set as a prerequisite of Automake
I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball
needs some patches, so I have just left it half done until someone actually
needs it and we apply the patch.
|
|
Until now, the tarballs were the first normal prerequisite of the
software. As a result if their date changed, the whole software would be
re-built. However, for tarballs specifically, we actually check their
contents with a checksum, so their date is irrelevant (if its newer than
the built program, but has the same checksum, there is no need to re-build
the software).
Also, calling the tarball name as an argument to the building process (for
example `gbuild') was redundant. It is now automatically found from the
list of order-only prerequisites within `gbuild' and `cbuild' (similar to
how it was previously found in the `pybuild' for Python building).
A `README.md' file has also been placed in `reproduce/software/make' to
help describe the shared properties of the software building
Makefiles. This will hopefully grow much larger in the future.
|
|
Python's `lmfit' module and all its major dependencies (`asteval',
`corner', `emcee' and `uncertainties') have been included in the template.
While doing this I noticed that if the tarballs are the last prerequisite
of each software building rule, then when building in parallel, the
template will immediately start building packages as soon as the first one
is downloaded. Not like the current way that it will attempt to download
several, then start building. For now, this has been implemented in the
Python build rules for all the modules and we'll later do the same for the
other programs and libraries. This also motivated a simplification of the
`pybuild' function: it now internally looks into the prerequisites and
selects the tarball from the prerequisite that is in the tarballs
directory.
This isn't a problem for the build, but I just don't understand why Python
can't recognize the version of `emcee', Python reads the version of `emcee'
as `0.0.0'! But it doesn't cause any crash in the build, so for now its
fine.
|
|
The tarball of HEALPix includes multiple languages and doesn't include the
ready-to-run GNU Build System by default, we actually have to build the
`./configure' script for the C/C++ libraries. So it was necessary to also
include GNU Autoconf and GNU Automake as prerequisites of HEALPix.
However, the official GNU Autoconf tarball (dating from 2012) doesn't build
on modern systems, so I just cloned it from its source and bootstrapped it
and built its modern tarball which we are using here.
|
|
The following software are added with this commit: eigency, esutil, flake8,
future, galsim, lsstdesccoord, pybind11 and pyflakes.
|
|
As part of an effort to bring in all the dependencies of the LSST Science
pipeline (which includes the last commit), these software are now available
in the template.
|
|
With this commit these three software packages are now installable with
this template.
|
|
Carlos Morales Socorro reported that his LaTeX build was missing the
`trimspaces' package, so it is now included in the pipeline.
|
|
It was some time since these three software were not updated! With this
commit the template now uses the most recent stable release of these
packages.
Also, the hosting server for ImageMagick was moved to my own webpage
because unfortunately ImageMagick removes its tarballs from its own
version.
|
|
New versions of astropy, bash, cmake, curl, findutils, gawk, gcc,
ghostscript, git, make, gsl had recently come so they are updated with this
commit.
About GNU Findutils and GNU Make: I was bootstrapping (building the tarball
of) these two separately separately because their standard tarball release
had problems on some systems. Both have been updated now so I am no longer
using my own webpage as their main URL.
A special note about GNU Make. I just noticed that during bootstrapping,
GNU Make would use the fixed version string of `4.2.90' for any commit!!!
But fortunately they have officially released their 4.2.90 version, so we
are safely using their own webpage. The only difference is the compression
format. My old bootstrapped build was `tar.lz', but the standard release is
`tar.gz'.
Also, all the basic programs (installed in `.local/bin') in `basic.mk' are
now existance-only dependencies (after a `|'). Because later programs just
use them at a very basic level, so there is no need to rebuild everything
when Bash gets updated for example.
|
|
Until this commit, the name of the variable for `beautifulsoup4'
checksum was wrong, and because of that, it was not able to install it.
With this commit, `beautifulsoup-checksum' has been replaced for
`beautifulsoup4-checksum' in the `reproduce/software/make/python.mk'
Makefile, and the problem has been fixed. This was not noticed
previously because this Python package is only installed when some high
level programs are requested to be installed.
With this commit the version of `imagemagick' program has been also
updated because the previous version is not available in the official
website anymore.
|
|
Recently the version of Gnuastro in the template was updated to version
0.10. However, I had forgot to update the `gnuastro.conf' file to fit with
the necessary new features of this version. The new general Gnuastro
configuration file is now added instead.
This bug was reported by Deepak.
|
|
This was a bug in WCSLIB 6.3 that has been fixed in WCSLIB 6.4. From
WCSLIB's changelog: "The rule change to the Fortran makefile in v6.3 to add
getwcstab_f.o to the sharable library causes it to depend on CFITSIO to
resolve fits_get_wcstab(). Hence backed out of that change.".
The actual error was like this:
Undefined symbols for architecture x86_64:
"_fits_read_wcstab", referenced from:
_ftwcst_ in getwcstab_f.o
"_gFitsFiles", referenced from:
_ftwcst_ in getwcstab_f.o
ld: symbol(s) not found for architecture x86_64
|
|
These three libraries are dependencies of Biber, so we will need them
later, but since we don't build biber from source now, we can't control
what library it links with. With this commit, we have just added their
versions, checksum, download URL and build rule incase they are useful in
other software.
Later, when we build Biber (and Texlive in general) from source, we'll be
able to use these.
|
|
Until now, OpenMPI was being installed without any dependency. This was
fine because it would indeed build. But the moment you tried loading
something that depends on it (for example `mpi4py' through `astropy'), you
would get an error complaining that SSH isn't present.
With this commit, the pipeline now also installs OpenSSH to solve this
problem.
|
|
A new version of Gnuastro was recently released with many improvments and
bug fixes, so it is updated here too.
|
|
Until now, in version 3.0.1, mpi4py couldn't be built with the most recent
version of OpenMPI. However, after trying the next version (3.0.2), I
noticed that it builds successfully without a problem.
|
|
Until now, there was no check on the integrity of the contents of the
downloaded/copied software tarballs, we only relied on the tarball
name. This could be bad for reproducibility and security, for example on
one server the name of a tarball may be the same but with different
content.
With this commit, the SHA512 checksums of all the software are stored in
the newly created `checksums.mk' (similar to how the versions are stored in
the `versions.mk'). The resulting variable is then defined for each
software and after downloading/copying the file we check to see if the new
tarball has the same checksum as the stored value. If it doesn't the script
will crash with an error, informing the user of the problem.
The only limitation now is a bootstrapping problem: if the host system
doesn't already an `sha512sum' executable, we will not do any checksum
verification until we install our `sha512sum' (as part of GNU
Coreutils). All the tarballs downloaded after GNU Coreutils are built will
have their checksums validated. By default almost all GNU/Linux systems
will have a usable `sha512sum' (its part of GNU Coreutils after all for a
long time: from the Coreutils Changelog file atleast since 2013).
This completes task #15347.
|
|
Until now, to work on a project, it was necessary to `./configure' it and
build the software. Then we had to run `.local/bin/make' to run the project
and do the analysis every time. If the project was a shared project between
many users on a large server, it was necessary to call the `./for-group'
script.
This way of managing the project had a major problem: since the user
directly called the lower-level `./configure' or `.local/bin/make' it was
not possible to provide high-level control (for example limiting the
environment variables). This was especially noticed recently with a bug
that was related to environment variables (bug #56682).
With this commit, this problem is solved using a single script called
`project' in the top directory. To configure and build the project, users
can now run these commands:
$ ./project configure
$ ./project make
To work on the project with other users in a group these commands can be
used:
$ ./project configure --group=GROUPNAME
$ ./project make --group=GROUPNAME
The old options to both configure and make the project are still valid. Run
`./project --help' to see a list. For example:
$ ./project configure -e --host-cc
$ ./project make -j8
The old `configure' script has been moved to
`reproduce/software/bash/configure.sh' and is called by the new `./project'
script. The `./project' script now just manages the options, then passes
control to the `configure.sh' script. For the "make" step, it also reads
the options, then calls Make. So in the lower-level nothing has
changed. Only the `./project' script is now the single/direct user
interface of the project.
On a parallel note: as part of bug #56682, we also found out that on some
macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set
to blank. This is no problem because RPATH is automatically set in macOS
and the executables and libraries contain the absolute address of the
libraries they should link with. But having `DYLD_LIBRARY_PATH' can
conflict with some low-level system libraries and cause very hard to debug
linking errors (like that reported in the bug report).
This fixes bug #56682.
|
|
Until now, like all other software, PatchELF would install with dynamic
linking. However, PatchELF links with `libstdc++' so on one system, I
noticed that PatchELF gives a segmentation fault and corrupts `libstdc++'
while correcting its RPATH (after installing GCC). The solution is to build
PatchELF statically.
With this commit, we force PatchELF to be built statically (it only
installs on GNU/Linux systems anyway, so there is no problem with static
linking on macOS). This solved the problem on that system.
While looking at its documentation, I also noticed that a new version of
PatchELF has been released after almost three years, so it has been updated
in the template also.
This fixes bug #56673.
|
|
Some bugs have been fixed in the new version of WCSLIB, so it has been
updated in the template.
|
