| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, the only verification that the template provided was the
published PDF. Users had to manually compare the published and generated
PDFs (numbers, plots, tables) and see if they obtained the same
result. However, this type of manual verification is not good and is prone
to frustration and missing important differences.
With this commit, a new Makefile has been added in the analysis steps:
`verify.mk'. It provides facilities to easily verify the results that go
into the paper. For example tables that go into making the paper's plots,
or the LaTeX macros that blend into the text. See the updated parts in
`README-hacking.md` for a more complete explanation.
This completes task #15497.
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
Until this commit, the number `2' was missing in the checksum variable
name of that library. It was `libxml-checksum' but it should be
`libxml2-checksum'. With this commit, this issue has been fixed.
|
|
An extra backslash in the prerequisites of Jeepney Python package was
causing a crash in the installation of that software. With this commit,
this problem has been fixed by removing the backslash.
|
|
OpenMP takes a LONG TIME to build, so to keep things reproducible we are
explicitly disabling OpenMP, if a user needs OpenMP, its trivial to just
add it as a prerequisite of R. The problem is that in some scenarios (based
on other dependencies and when they were built in the build directory),
OpenMP may be present when R is being installed and in other it may not. We
don't want the result to be different between the two builds.
|
|
With this commit, we now have the core R interpretter within the
template. We should later include instructions to install R packages
(possibly in a separate top-level Makefile like Python).
|
|
Recent builds of the template need these three packages to build the PDF.
This was reported by Hamed Altafi.
|
|
Until now we weren't including this package, but Alberto Madrigal recently
reported that his build of the template failed because it needed it.
With this commit, it is now being installed with the template.
|
|
Until now, while building Perl, the `-Dlddlflags="-shared $$LDFLAGS"'
option was being used for all systems. However, this caused `symbol(s) not
found for architecture x86_64' problems on macOS systems.
With this commit, this configure option is only passed for GNU/Linux
systems. With it, Perl can be built nicely on both GNU/Linux and macOS.
|
|
Newer versions of these packages have recently come out with major
improvements, so they have been updated in the template.
|
|
Until now, Ghostscript was using some host system's X11 libraries during
its build (and later at run-time). We should ideally install all these
necessary libraries within the template (Task #15481). But right now we are
too busy.
As a temporary work-around we try building a small dummy program that links
with some of those libraries before attempting to built Ghostscript. If it
fails, then a notice is printed with the cause and explaining a temporary
solution is suggested: how to install those libraries on the system when
you have root access.
|
|
Until now, when an input dataset already exists in `INDIR', the template
would just make a symbolic link to it in the build directory. However, in
many cases, the files in INDIR will actually be links to other locations on
the filesystem and some programs have problems following too many links.
With this commit, the template is now using the `readlink' program (part of
GNU Coreutils) to follow a possible link and point the link in the build
directory directly to an actual non-link file.
|
|
Until now when building Matplotlib on macOS systems, we were using the
default C compiler. However, while Yahya Sefidbakht (previously) and
Mahdieh Nabavi (now) were trying to build the template, on their macOS
using the GNU Compiler Collection (GCC), we found that Matplotlib needs
special macOS headers that GCC doesn't recognize.
With this commit, when Matplotlib is being built on macOS systems, it uses
`clang' and this fixed the problem (so far checked on Mahdieh's machine).
|
|
While working on a different branch to build the GNU C Library, I noticed a
few places in the template that need corrections which are now applied:
1. A new-line character after the "C compiler works" notice at the start
of the configure script.
2. Removing possible `::' in the `LD_LIBRARY_PATH' definition of
`basic.mk'. Note that its not necessary in the other steps because we
don't use any outside-defined `LD_LIBRARY_PATH'.
3. Building GMP for C++ and also with `--enable-fat'.
4. Removing the unpacked Perl tarball directory after its installation.
|
|
These two packages are necessary to build the GNU C Library.
|
|
Perl is necessary to build Texinfo and later to build LaTeX. Until now we
were just using the host operating system's installation of Perl, but in
some instances that Perl can be too old and not suppor the features
necessary. With this commit, Perl is now built from source during the basic
installation step of the template.
This was reported by Idafen Santana Pérez, after trying the pipeline on an
Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
|
|
When building the log4cxx tarball from its Git history, I noticed that
files with very long names are not packaged by tar (because by default
Automake uses the ancient v7 tar format that only supports file names less
than 99 characters).
So I build the tarball with the `tar-ustar' option to Automake (by
modifying the log4cxx source) and the resulting tarball was able to compile
and run successfully. This has been described above the rule to build
log4cxx and I also sent an email to their developing mailing list to inform
them of this problem. If they address it, I will remove the note on the
necessary corrections.
|
|
Some minor corrections were made in the template:
- When making the distribution, `.swp' files (created by Vim) are also
removed.
- Autoconf is set as a prerequisite of Automake
I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball
needs some patches, so I have just left it half done until someone actually
needs it and we apply the patch.
|
|
Until now, the tarballs were the first normal prerequisite of the
software. As a result if their date changed, the whole software would be
re-built. However, for tarballs specifically, we actually check their
contents with a checksum, so their date is irrelevant (if its newer than
the built program, but has the same checksum, there is no need to re-build
the software).
Also, calling the tarball name as an argument to the building process (for
example `gbuild') was redundant. It is now automatically found from the
list of order-only prerequisites within `gbuild' and `cbuild' (similar to
how it was previously found in the `pybuild' for Python building).
A `README.md' file has also been placed in `reproduce/software/make' to
help describe the shared properties of the software building
Makefiles. This will hopefully grow much larger in the future.
|
|
Python's `lmfit' module and all its major dependencies (`asteval',
`corner', `emcee' and `uncertainties') have been included in the template.
While doing this I noticed that if the tarballs are the last prerequisite
of each software building rule, then when building in parallel, the
template will immediately start building packages as soon as the first one
is downloaded. Not like the current way that it will attempt to download
several, then start building. For now, this has been implemented in the
Python build rules for all the modules and we'll later do the same for the
other programs and libraries. This also motivated a simplification of the
`pybuild' function: it now internally looks into the prerequisites and
selects the tarball from the prerequisite that is in the tarballs
directory.
This isn't a problem for the build, but I just don't understand why Python
can't recognize the version of `emcee', Python reads the version of `emcee'
as `0.0.0'! But it doesn't cause any crash in the build, so for now its
fine.
|
|
The tarball of HEALPix includes multiple languages and doesn't include the
ready-to-run GNU Build System by default, we actually have to build the
`./configure' script for the C/C++ libraries. So it was necessary to also
include GNU Autoconf and GNU Automake as prerequisites of HEALPix.
However, the official GNU Autoconf tarball (dating from 2012) doesn't build
on modern systems, so I just cloned it from its source and bootstrapped it
and built its modern tarball which we are using here.
|
|
The following software are added with this commit: eigency, esutil, flake8,
future, galsim, lsstdesccoord, pybind11 and pyflakes.
|
|
As part of an effort to bring in all the dependencies of the LSST Science
pipeline (which includes the last commit), these software are now available
in the template.
|
|
With this commit these three software packages are now installable with
this template.
|
|
The steps to package the project have been made slightly more clear and
also the temporary directory that is created for packaging is deleted after
the tarball is made.
|
|
Until now, when you ran `make clean', all the directories under
`$(BDIR)/tex/' would be deleted except for `macros' and `build'. This was
good for the single-user mode. But in group mode, this would delete the
user-specific TeX build directory because its called `build-USER', not
`build'.
With this commit, to fix the problem, we define the new `texbtopdir' and
based on the group condition, and use that to specify which directory to
not delete.
|
|
Carlos Morales Socorro reported that his LaTeX build was missing the
`trimspaces' package, so it is now included in the pipeline.
|
|
Until now, some software were using the depreciated ADS URL
(`http://adsabs.harvard.edu....') and some were using the modern one
(`http://ui.adsabs.harvard.edu').
With this commit, to be consistent they all link to the modern URL.
|
|
Until now, this script would always only work with a file-lock. But in some
scenarios, we might want to download in parallel. For example when the
system has multiple ports to the internet.
With this commit, we have added this feature: when the lockfile name is
`nolock', it won't lock and will download in parallel.
|
|
Since ImageMagick can take long to build, we are now building it in
parallel. Also, the part where we replace an `_' with `\_' in the software
version at the end of the configure script was removed. It is more
clear/readable that the actual rule that includes such a name deals with
the underline (as is the case for `sip_tpv' which already dealt with it).
Finally, I noticed that the checks at the start of `top-prepare' were
missing new-lines. I had forgot that the Make single-shell variable isn't
activated in this stage yet.
|
|
A special directory is now defined in `initialize.mk' that can be used in
both the preparation and build phases. Also, the contents of prepared
results can now be conditionally read during `./project make'.
|
|
In many real-world scenarios, `./project make' can really benefit from
having some basic information about the data before being run. For example
when quering a server. If we know how many datasets were downloaded and
their general properties, it can greatly optmize the process when we are
designing the solution to be run in `./project make'.
Therefore with this commit, a new phase has been added to the template's
design: `./project prepare'. In the raw template this is empty, because the
simple analysis done in the template doesn't warrant it. But everything is
ready for projects using the template to add preparation phases prior to
the analysis.
|
|
It was some time since these three software were not updated! With this
commit the template now uses the most recent stable release of these
packages.
Also, the hosting server for ImageMagick was moved to my own webpage
because unfortunately ImageMagick removes its tarballs from its own
version.
|
|
Until now we were calling it `Sextractor', but the official way of writing
it is `SExtractor'. With this commit, this has been corrected.
|
|
Until now, we were assuming that the users would just clone the project in
Git. But after submitting arXiv:1909.11230, and trying to build directly
from the arXiv source, I noticed several problems that wouldn't allow users
to build it automatically. So I tried the build step by step and was able
to find a fix for the several issues that came up.
The scripting parts of the fix were primarily related to the fact that the
unpacked arXiv tarball isn't under version control, so some checks had to
be put there. Also, we wanted to make it easy to remove the extra files, so
an extra `--clean-texdit' option was added to `./project'.
Finally, some manual corrections were necessary (prior to running
`./project', which are now described in `README.md'. Most of the later
steps can be automated and we should do it later, I just don't have enough
time now.
|
|
Until now, the pipeline was instructed to only ignore the current temporary
project distribution directory. So if there were directories from previous
builds, they were wrongly included in the current tarball.
With this commit, we don't just ignore the directory of the current
distribution, but generally, all directories starting with `paper-v*'.
|
|
Users that are not familiar with the file structure of the project may
specify the current directory (to-level source directory) as their
build-directory. This will cause a crash right after answering the
questions, where `rm' will complain about `tex/build' not being deleted
because it exists as a directory.
To avoid such confusing situtations, the configure script now checks if the
build directory is actually a sub-directory of the source. If it is, it
will complain with a short message and abort. Also, a `CAUTION' statment
has been put in the initial description, right ontop of the question.
This bug was reported Carlos Allende Prieto and David Valls-Gabaud.
|
|
Until now, when the commit was tagged, `git describe' would just print the
tag and no longer the commit checksum. This is bad because the checksum is
a much more robust way to confirm the point in history.
With this commit the `--long' option has been added to `git describe' to
fix this issue. From now on, when we are on a tag, it will print the tag
followed by a `-0-' and the first characters of the checksum.
|
|
`./project make dist' will package all the LaTeX-specific files (and
analysis source files) into one `tar.gz' file that is ready to upload to
servers like arXiv. However, it wasn't updated for some time, so running it
would complain about not having a `configure' script in the top of the
project.
With this commit, it now works with the new file-structure of the project
and also copies all the BibLaTeX source files and `paper.bbl' into the top
tarball directory, which allows arXiv to build the paper as intended.
The output of `./project make dist' has been uploaded and tested on arXiv
and it is built by arXiv perfectly.
Also, a short description of all the special `make' targets was added to
the output of `./project --help'.
|
|
Until now the SExtractor BibTeX entry has the first-name and family name in
the same field, while the family-name should be in curly braces. Also, it
was missing an ADS URL. So the default ADS generated BibTeX entry for
SExtractor is now used instead of the old one.
|
|
New versions of astropy, bash, cmake, curl, findutils, gawk, gcc,
ghostscript, git, make, gsl had recently come so they are updated with this
commit.
About GNU Findutils and GNU Make: I was bootstrapping (building the tarball
of) these two separately separately because their standard tarball release
had problems on some systems. Both have been updated now so I am no longer
using my own webpage as their main URL.
A special note about GNU Make. I just noticed that during bootstrapping,
GNU Make would use the fixed version string of `4.2.90' for any commit!!!
But fortunately they have officially released their 4.2.90 version, so we
are safely using their own webpage. The only difference is the compression
format. My old bootstrapped build was `tar.lz', but the standard release is
`tar.gz'.
Also, all the basic programs (installed in `.local/bin') in `basic.mk' are
now existance-only dependencies (after a `|'). Because later programs just
use them at a very basic level, so there is no need to rebuild everything
when Bash gets updated for example.
|
|
Until now, OpenMPI would complain about not having `ssh' or `rsh' as a
remote shell feature. However, such features should not be necessary in a
reproducible scenario and they also have major security issues.
With this commit, we are now using OpenMPI's `OMPI_MCA_plm_rsh_agent'
environment variable to disable any remote shell dependency for it (as
suggested by Boud). Therefore, any dependency for OpenSSH has been
removed. But I thought to keep the build instructions incase it may be
useful under some un-foreseen scenario. However, to discourage people from
building it, a notice was added ontop of the build instructions.
This bug was found, tested and solved thanks to Roberto Baena Gallé and
Boud Roukema.
This fixes bug #56724.
|
|
Until now we have followed the convention of using space characters before
comment lines in recepies, not tabs. This has been corrected in one case.
|
|
Until this commit, `fpack' and `funpack' were not installed by default
with the installation of CFITSIO. It is necessary to explicity do a
`make fpack' and `make funpack' to have them installed. With this
commit, these two programs have been added.
|
|
Until this commit, the name of the variable for `beautifulsoup4'
checksum was wrong, and because of that, it was not able to install it.
With this commit, `beautifulsoup-checksum' has been replaced for
`beautifulsoup4-checksum' in the `reproduce/software/make/python.mk'
Makefile, and the problem has been fixed. This was not noticed
previously because this Python package is only installed when some high
level programs are requested to be installed.
With this commit the version of `imagemagick' program has been also
updated because the previous version is not available in the official
website anymore.
|
|
There were no conflicts in this merge.
|
|
When trying to install `libgit2' on my macOS system, it complains about
not finding `_iconv*' functions! But apparently `libgit2' has its own
implementation of `libiconv' that it uses if it can't find `libiconv' on
macOS. So, the solution to this problem was to add `-DUSE_ICONV=OFF' to
the configuration options of `libgit2'.
I reported this issue that now is fixed thanks to the help of Mohammad
Akhlaghi.
|
|
Recently the version of Gnuastro in the template was updated to version
0.10. However, I had forgot to update the `gnuastro.conf' file to fit with
the necessary new features of this version. The new general Gnuastro
configuration file is now added instead.
This bug was reported by Deepak.
|
|
Until now, we weren't explicitly telling libtiff to ignore JBIG-KIT. So on
some systems, it would try to link withit and would thus fail.
With this commit, we have disabled JBIG-KIT support for libtiff. I tried to
import it into the template, but it doesn't have use the standard GNU Build
system. Maybe later we can add a set of build rules for it, I don't have
time now.
Also, this problem with libtiff occurred while building Ghostscript. But
was fixed after adding this option to libtiff. So libtiff was added as a
dependency of Ghostscript.
This bug was reported by Sheeraz Ahmad.
|
|
Until now we weren't setting OpenSSH's `privsep-path' configure option. As
a result, it would try to use its default (`/var/empty'). Therefore, when
the host doesn't have `/var/empty', OpenSSH would crash because of not
having permissions to create this directory.
With this commit, we are now using OpenSSH's `--with-privsep-path'
configure-time option to explicitly use a directory with the project's
build directory.
This bug was found by Sheeraz Ahmad and Amina Aahil.
|
