| Age | Commit message (Collapse) | Author | Lines |
|---|
|
The two modifications to the LaTeX source of an arXiv-downloaded source
weren't rendered properly on Gitlab, so they are corrected to be in the
same line and not have a separate code-block.
|
|
Until now, we were assuming that the users would just clone the project in
Git. But after submitting arXiv:1909.11230, and trying to build directly
from the arXiv source, I noticed several problems that wouldn't allow users
to build it automatically. So I tried the build step by step and was able
to find a fix for the several issues that came up.
The scripting parts of the fix were primarily related to the fact that the
unpacked arXiv tarball isn't under version control, so some checks had to
be put there. Also, we wanted to make it easy to remove the extra files, so
an extra `--clean-texdit' option was added to `./project'.
Finally, some manual corrections were necessary (prior to running
`./project', which are now described in `README.md'. Most of the later
steps can be automated and we should do it later, I just don't have enough
time now.
|
|
The description of arXiv:1909.11230 was slightly modified to be in the same
format as the other papers.
|
|
This paper was published on arXiv today and is a good example for people to
see the application of this system in practice.
|
|
Until now, the pipeline was instructed to only ignore the current temporary
project distribution directory. So if there were directories from previous
builds, they were wrongly included in the current tarball.
With this commit, we don't just ignore the directory of the current
distribution, but generally, all directories starting with `paper-v*'.
|
|
Until now customizing it was a little more detailed, for example the
copyright statement wasn't generic and was about "this template". So the
user would have to correct it.
With this commit, the copyright statment just says "this project", so it
can apply to the raw template and also any customization of it. Also, some
minor edits were made in the various parts of the text to make it more
clear.
|
|
Users that are not familiar with the file structure of the project may
specify the current directory (to-level source directory) as their
build-directory. This will cause a crash right after answering the
questions, where `rm' will complain about `tex/build' not being deleted
because it exists as a directory.
To avoid such confusing situtations, the configure script now checks if the
build directory is actually a sub-directory of the source. If it is, it
will complain with a short message and abort. Also, a `CAUTION' statment
has been put in the initial description, right ontop of the question.
This bug was reported Carlos Allende Prieto and David Valls-Gabaud.
|
|
Until now, when the commit was tagged, `git describe' would just print the
tag and no longer the commit checksum. This is bad because the checksum is
a much more robust way to confirm the point in history.
With this commit the `--long' option has been added to `git describe' to
fix this issue. From now on, when we are on a tag, it will print the tag
followed by a `-0-' and the first characters of the checksum.
|
|
`./project make dist' will package all the LaTeX-specific files (and
analysis source files) into one `tar.gz' file that is ready to upload to
servers like arXiv. However, it wasn't updated for some time, so running it
would complain about not having a `configure' script in the top of the
project.
With this commit, it now works with the new file-structure of the project
and also copies all the BibLaTeX source files and `paper.bbl' into the top
tarball directory, which allows arXiv to build the paper as intended.
The output of `./project make dist' has been uploaded and tested on arXiv
and it is built by arXiv perfectly.
Also, a short description of all the special `make' targets was added to
the output of `./project --help'.
|
|
After a re-read on Gitlab, it has been slightly edited to be more clear.
|
|
When you want to publish your project, it is very convenient to have a
single file that contains the whole history. So a tip is added to
`README-hacking.md' that describes how to do this with `git bundle'.
|
|
Until now, we weren't explicitly asking for BibLaTeX to build the citations
with only initial of the author's given names. Therefore when one BibTeX
entry had a full given name and another had only initials, this would also
be reflected in the paper's bibliography.
With this commit, the `giveninits=true' option is given to BibLaTeX to
always only print the initial character of an author's given name.
|
|
Until now the SExtractor BibTeX entry has the first-name and family name in
the same field, while the family-name should be in curly braces. Also, it
was missing an ADS URL. So the default ADS generated BibTeX entry for
SExtractor is now used instead of the old one.
|
|
In some cases, users of the template may not need the other template
headers, they may only want `preamble-biblatex.tex'. But `xcolor' needs to
be loaded before being able to load the various colors we use in the
references. So to be self-consistent, it is loaded.
Also, the default style was also printing the month of the publication
which is not common. So a line was added to clear the `month' field before
building the Bibliography.
|
|
New versions of astropy, bash, cmake, curl, findutils, gawk, gcc,
ghostscript, git, make, gsl had recently come so they are updated with this
commit.
About GNU Findutils and GNU Make: I was bootstrapping (building the tarball
of) these two separately separately because their standard tarball release
had problems on some systems. Both have been updated now so I am no longer
using my own webpage as their main URL.
A special note about GNU Make. I just noticed that during bootstrapping,
GNU Make would use the fixed version string of `4.2.90' for any commit!!!
But fortunately they have officially released their 4.2.90 version, so we
are safely using their own webpage. The only difference is the compression
format. My old bootstrapped build was `tar.lz', but the standard release is
`tar.gz'.
Also, all the basic programs (installed in `.local/bin') in `basic.mk' are
now existance-only dependencies (after a `|'). Because later programs just
use them at a very basic level, so there is no need to rebuild everything
when Bash gets updated for example.
|
|
The Copyright year is now on a separate line (by adding a backslash), and
the `file-metadata' is now enclosed in two "`" characters to show
differently after rendering.
|
|
Until now, OpenMPI would complain about not having `ssh' or `rsh' as a
remote shell feature. However, such features should not be necessary in a
reproducible scenario and they also have major security issues.
With this commit, we are now using OpenMPI's `OMPI_MCA_plm_rsh_agent'
environment variable to disable any remote shell dependency for it (as
suggested by Boud). Therefore, any dependency for OpenSSH has been
removed. But I thought to keep the build instructions incase it may be
useful under some un-foreseen scenario. However, to discourage people from
building it, a notice was added ontop of the build instructions.
This bug was found, tested and solved thanks to Roberto Baena Gallé and
Boud Roukema.
This fixes bug #56724.
|
|
Until now we have followed the convention of using space characters before
comment lines in recepies, not tabs. This has been corrected in one case.
|
|
Until this commit, `fpack' and `funpack' were not installed by default
with the installation of CFITSIO. It is necessary to explicity do a
`make fpack' and `make funpack' to have them installed. With this
commit, these two programs have been added.
|
|
Until this commit, the name of the variable for `beautifulsoup4'
checksum was wrong, and because of that, it was not able to install it.
With this commit, `beautifulsoup-checksum' has been replaced for
`beautifulsoup4-checksum' in the `reproduce/software/make/python.mk'
Makefile, and the problem has been fixed. This was not noticed
previously because this Python package is only installed when some high
level programs are requested to be installed.
With this commit the version of `imagemagick' program has been also
updated because the previous version is not available in the official
website anymore.
|
|
There were no conflicts in this merge.
|
|
When trying to install `libgit2' on my macOS system, it complains about
not finding `_iconv*' functions! But apparently `libgit2' has its own
implementation of `libiconv' that it uses if it can't find `libiconv' on
macOS. So, the solution to this problem was to add `-DUSE_ICONV=OFF' to
the configuration options of `libgit2'.
I reported this issue that now is fixed thanks to the help of Mohammad
Akhlaghi.
|
|
Until now, the paper's title and author information were set it
`tex/src/preamble-header.tex'. But they are actually shown in the final PDF
paper and a much better place to keep them is the top-level `paper.tex'.
With this commit, the setting of the title and author names has been moved
to `paper.tex', just after importing all the preambles. However, the basic
package importation and low-level settings are still set in
`tex/src/preamble-header.tex', because they are relatively low-level.
This task was suggested by Deepak (Indian Institute of Astrophysics).
|
|
Recently the version of Gnuastro in the template was updated to version
0.10. However, I had forgot to update the `gnuastro.conf' file to fit with
the necessary new features of this version. The new general Gnuastro
configuration file is now added instead.
This bug was reported by Deepak.
|
|
Until now, when describing the sections to remove for customizing a
project, I had mistakenly repeated the `%% Start of main body.'
statement.
With this commit, the second one is changed to `%% End of main body.'
This issue was reported by Deepak.
|
|
After the checklist was applied in the 5th Indo-French Astronomy School, we
found some cases in the checklist that were extra (and thus had to be
removed), or were needed (and thus were added).
Also the non-necessary steps for a first commit were moved to a
separate/new section in the checklist for the people to add after doing
their first commit.
Also, the software part of the paper was moved to an appendix.
|
|
Until now, we weren't explicitly telling libtiff to ignore JBIG-KIT. So on
some systems, it would try to link withit and would thus fail.
With this commit, we have disabled JBIG-KIT support for libtiff. I tried to
import it into the template, but it doesn't have use the standard GNU Build
system. Maybe later we can add a set of build rules for it, I don't have
time now.
Also, this problem with libtiff occurred while building Ghostscript. But
was fixed after adding this option to libtiff. So libtiff was added as a
dependency of Ghostscript.
This bug was reported by Sheeraz Ahmad.
|
|
Until now we weren't setting OpenSSH's `privsep-path' configure option. As
a result, it would try to use its default (`/var/empty'). Therefore, when
the host doesn't have `/var/empty', OpenSSH would crash because of not
having permissions to create this directory.
With this commit, we are now using OpenSSH's `--with-privsep-path'
configure-time option to explicitly use a directory with the project's
build directory.
This bug was found by Sheeraz Ahmad and Amina Aahil.
|
|
This was a bug in WCSLIB 6.3 that has been fixed in WCSLIB 6.4. From
WCSLIB's changelog: "The rule change to the Fortran makefile in v6.3 to add
getwcstab_f.o to the sharable library causes it to depend on CFITSIO to
resolve fits_get_wcstab(). Hence backed out of that change.".
The actual error was like this:
Undefined symbols for architecture x86_64:
"_fits_read_wcstab", referenced from:
_ftwcst_ in getwcstab_f.o
"_gFitsFiles", referenced from:
_ftwcst_ in getwcstab_f.o
ld: symbol(s) not found for architecture x86_64
|
|
These three libraries are dependencies of Biber, so we will need them
later, but since we don't build biber from source now, we can't control
what library it links with. With this commit, we have just added their
versions, checksum, download URL and build rule incase they are useful in
other software.
Later, when we build Biber (and Texlive in general) from source, we'll be
able to use these.
|
|
In the line where we checked if libffi is installed in `lib/' or `lib64/',
there was a typo that lead to an incorrect result: we had forgot to add a
`-f' to the second check. This is corrected with this commit.
|
|
Until now, unlike most other programs, in Gnuastro we would run `make
check', but this causes a crash on some systems, because of its
BuildProgram test: a linking error because the library isn't installed
yet. Which is natural in cases like this (and must be corrected in future
Gnuastro releases).
With this commit, the checks of Gnuastro have been removed.
|
|
Until now, when building PatchELF, we would always require that it be done
statically. However, some systems don't have a static C library available
for linking. This cause a crash in the static building of PatchELF. But a
static PatchELF is necessary for correcting RPATH in GCC's outputs.
With this commit, in the configure script we check if a static C library is
linkable for the compiler. If it isn't then `host_cc' will be set to 1 and
GCC won't be built. We also pass the result of this test to `basic.mk'
(through `good_static_lib'), so if a static C library isn't available, it
builds a dynamically linked PatchELF.
This bug was reported by Elham Saremi.
|
|
Until now, the Fortran compiler check wouldn't delete the files it creates
in the temporary software building directory.
With this commit, the cleaning steps have been added.
|
|
Until now, OpenMPI was being installed without any dependency. This was
fine because it would indeed build. But the moment you tried loading
something that depends on it (for example `mpi4py' through `astropy'), you
would get an error complaining that SSH isn't present.
With this commit, the pipeline now also installs OpenSSH to solve this
problem.
|
|
Until now, we were relying on WCSLIB's internal checking and linking with
CFITSIO. But on one macOS system (not others that had no problem!), we
noticed that it complains with undefined symbol linking errors to CFITSIO
libraries.
With this commit, as a fast/ugly solution, we are explicity adding
`-lcfitsio' to WCSLIB's `LIBS' variable so all binaries are linked with it
automatically. We'll be in touch with the WCSLIB author to see if a better
solution can be found.
|
|
Until now, we weren't explicitly disabling Java in the build of libffi
because there was no Java platform on the systems we tested. But recently
Yahya Sefidbakht report a crash in libffi (reported in bug #56716) because
of Java not being compiled properly.
Unfortunately libffi doesn't have a configure option to disable Java, but
it does have an internal macro (`NO_JAVA_RAW_API'). With this commit, we
we defined this macro in the `make' step, and it solved the problem.
This fixes bug #56716.
|
|
Some of the software tarballs are directly available on their webpage (for
example due to build problems on systems, where we had to clone the
software and build its tarball ourselves until the next release). Until
now, only for these software, these tarballs were hosted on
`http://akhlaghi.org/src'. But now, I have moved a clone of the software
backup repository (including all its software) to
`http://akhlaghi.org/reproduce-software'.
To be more clear and have a single place for the backup software, the URL
of those software has also been updated in the project source.
|
|
A new version of Gnuastro was recently released with many improvments and
bug fixes, so it is updated here too.
|
|
Until now, we were just checking for the existance of a C and Fortran
compiler. But it can happen that even if they exist, they don't operate
properly, for example see some errors that have been reported until now in
P.S. (both on different macOS systems). But finding this source after the
programs have started is frustrating for the user.
With this commit, before we start building anything, we'll check these two
compilers with a simple program and see if they can indeed compile, and if
their compiled program can run. If it doesn't work an elaborate error
message is printed to help the users navigate to a solution.
Also, the building of `flock' within `configure.sh' has been moved just
before calling `basic.mk'. This was done so any warning/error message
is printed before actually building anything.
This fixes bug #56715.
P.S. The error messages:
C compiler
----------
conftest.c:9:19: fatal error: stdio.h: No such file or directory
^
compilation terminated.
----------
Fortran compiler
----------------
dyld: Library not loaded: @rpath/libisl.10.dylib
Referenced from:
/path/to/anaconda2/gcc/libexec/gcc/x86_64-apple-darwin15.5.0/4.9.3/f951
Reason: image not found
gfortran: internal compiler error: Abort trap: 6
----------------
|
|
Until now, in version 3.0.1, mpi4py couldn't be built with the most recent
version of OpenMPI. However, after trying the next version (3.0.2), I
noticed that it builds successfully without a problem.
|
|
Until now, when you needed to completely clean a project (with `./project
make distclean') the Git hooks that are installed during configure time
would cause problems when committing (the `pre-commit' hook in particular
won't allow you to commit anything!).
With this commit, before deleting the software, the template first removes
these Git hooks.
|
|
After making the previous commit, I noticed an extra line (redundantly
defining a wrong `BASH_ENV') that should have been deleted. It has been
corrected.
|
|
Until now the only way to define the environment of the Make recipes was
through the exported Make variables (mostly in `initialize.mk' for the
analysis steps for example). However, there is only so much you can do with
environment variables! In some situations you want slightly more
complicated environment control, like setting an alias or running of
scripts (things that are commonly done in the `~/.bashrc' file of users to
configure their interactive, non-login shells).
With this commit, a `reproduce/software/bash/bashrc.sh' has been defined
for this job (which is currently empty!). Every major Make step of the
project adds this file as the `BASH_ENV' environment variable, so the shell
that is created to execute a recipe first executes this file, then the
recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment
variable that enables users to limit their envirnoment setup based on the
condition it is being setup (in particular in the early phase of
`basic.mk', where the user can't make any assumption about the programs and
has to write a portable shell script).
|
|
After adding the libiconv library to the template, the C++ library uses
three of its functions (`libiconv', `libiconv_open' and
`libiconv_close'). However, it doesn't explicity link with it inside its
shared library!
I tried by exporting `LIBS=-liconv' before the GCC configure script but it
crashed as soon as it went on to the second GCC building stage (because
this environment variable was no longer present there). I also tried adding
the C++ configure option of `--enable-cstdio' to the GCC configure options
(so it doesn't use iconv according to the manual), but it made no change.
With this commit, as a brute-force solution, `patchelf --add-needed' is run
on the installed `libstdc++.so', so `libiconv.so' is explicitly included
inside the `libstdc++' shared library.
This bug was found by Roberto Baena Galle while trying to load Matplotlib
(which needed the C++ library).
This fixes bug #56702.
|
|
Until now, there was no check on the integrity of the contents of the
downloaded/copied software tarballs, we only relied on the tarball
name. This could be bad for reproducibility and security, for example on
one server the name of a tarball may be the same but with different
content.
With this commit, the SHA512 checksums of all the software are stored in
the newly created `checksums.mk' (similar to how the versions are stored in
the `versions.mk'). The resulting variable is then defined for each
software and after downloading/copying the file we check to see if the new
tarball has the same checksum as the stored value. If it doesn't the script
will crash with an error, informing the user of the problem.
The only limitation now is a bootstrapping problem: if the host system
doesn't already an `sha512sum' executable, we will not do any checksum
verification until we install our `sha512sum' (as part of GNU
Coreutils). All the tarballs downloaded after GNU Coreutils are built will
have their checksums validated. By default almost all GNU/Linux systems
will have a usable `sha512sum' (its part of GNU Coreutils after all for a
long time: from the Coreutils Changelog file atleast since 2013).
This completes task #15347.
|
|
Until now there was no clear explanation on `.file-metadata' within the
project. But since it sometimes appears in the Git changed files and its
binary, it was necessary to add a short explanation to inform users.
With this commit a section has been added to the "Project architecture"
section of `README-hacking.md' to give some context on what it is and how
to deal with it.
This was suggested by Hamed Altafi.
|
|
The configuration step (building all the ncessary software) can take some
time. It is natual for the user to want to see how the build is going
(which software is being built at every moment). So far, we have only put a
"Inspecting status" section in `README-hacking.md' that describes a
solution, but some early users may not have read it yet.
With this commit a short tip was added in the initial installation notice
to inform the user of this very useful command.
|
|
We recently moved the system's `rm' program absolute address to a shell
variable that is found during the `./project' script. But I had forgot to
account for the difference between the Make and Bash variable naming
differences. I had also forgot to add a value to the HOME variable.
With this commit both are corrected: the system's `rm' path is now called
`sys_rm' and the HOME variable is set.
|
|
Until now, to work on a project, it was necessary to `./configure' it and
build the software. Then we had to run `.local/bin/make' to run the project
and do the analysis every time. If the project was a shared project between
many users on a large server, it was necessary to call the `./for-group'
script.
This way of managing the project had a major problem: since the user
directly called the lower-level `./configure' or `.local/bin/make' it was
not possible to provide high-level control (for example limiting the
environment variables). This was especially noticed recently with a bug
that was related to environment variables (bug #56682).
With this commit, this problem is solved using a single script called
`project' in the top directory. To configure and build the project, users
can now run these commands:
$ ./project configure
$ ./project make
To work on the project with other users in a group these commands can be
used:
$ ./project configure --group=GROUPNAME
$ ./project make --group=GROUPNAME
The old options to both configure and make the project are still valid. Run
`./project --help' to see a list. For example:
$ ./project configure -e --host-cc
$ ./project make -j8
The old `configure' script has been moved to
`reproduce/software/bash/configure.sh' and is called by the new `./project'
script. The `./project' script now just manages the options, then passes
control to the `configure.sh' script. For the "make" step, it also reads
the options, then calls Make. So in the lower-level nothing has
changed. Only the `./project' script is now the single/direct user
interface of the project.
On a parallel note: as part of bug #56682, we also found out that on some
macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set
to blank. This is no problem because RPATH is automatically set in macOS
and the executables and libraries contain the absolute address of the
libraries they should link with. But having `DYLD_LIBRARY_PATH' can
conflict with some low-level system libraries and cause very hard to debug
linking errors (like that reported in the bug report).
This fixes bug #56682.
|
