| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, the message that we printed just before starting to build
software didn't actually print the current directory, but only `pwd'. With
this commit, this is fixed (it uses the `currentdir' variable that is
already found before).
|
|
Until now, throughout Maneage we were using the old name of "Reproducible
Paper Template". But we have finally decided to use Maneage, so to avoid
confusion, the name has been corrected in `README-hacking.md' and also in
the copyright notices.
Note also that in `README-hacking.md', the main Maneage branch is now
called `maneage', and the main Git remote has been changed to
`https://gitlab.com/maneage/project' (this is a new GitLab Group that I
have setup for all Maneage-related projects). In this repository there is
only one `maneage' branch to avoid complications with the `master' branch
of the projects using Maneage later.
|
|
Of the GCC dynamically linked libraries we need to manually add RPATH to
all and for `libstdc++' we also need to tell it to link with
`libiconv'. Until now, the conditional to check for libstdc++ was not
working and thus libiconv wasn't been added to it.
With this commit the conditional has been corrected and is now
working. Also, to help in reading the logs, an echo statement was added
after every call to PatchELF.
|
|
Until now, when a the raw tarball of some software wasn't usable, I would
put it under my own webpage, or `akhlaghi.org/reproduce-software'. That
same address was also used as a backup server. However, now the project has
a proper name: Maneage. So I changed the directory on my own server to
`akhlaghi.org/maneage-software'.
With this commit, this new address has replaced the old one. But to avoid
crashes in projects that haven't yet merged with the main Maneage branch,
the old `reproduce-software' still works (its actually a symbolic link to
the new directory now).
|
|
In the previous commit, we remove the `-static' flag from building PatchELF
because it wasn't necessary any more. Howver, the comment for the check
still included it and could be confusing. This is corrected with this
commit. Also, we don't need the `good_static_libc' variable (that was only
defined to pass onto PatchELF). This has also been corrected.
|
|
Until a few commits ago, PatchELF was built statically because it was used
to patch `libstdc++' at the end of the GCC building phase, but PatchELF
also depends on `libstdc++', so it would crash. However, recently when
patching the GCC libraries, we don't directly apply Patchelf to the
library, first we copy it to a temporary place, do the patching, then put
it in its proper place. So the problem above won't happen any more.
With this commit, I am thus removing the static flag from patchelf and
letting it built dynamically all the time. The main problem was that some
systems don't have a static C++ library, so PatchELF couldn't be built
statically. Instead of adding more checks, we just fixed the core
foundation of the problem.
|
|
Until now the software configuration parameters were defined under the
`reproduce/software/config/installation/' directory. This was because the
configuration parameters of analysis software (for example Gnuastro's
configurations) were placed under there too. But this was terribly
confusing, because the run-time options of programs falls under the
"analysis" phase of the project.
With this commit, the Gnuastro configuration files have been moved under
the new `reproduce/analysis/config/gnuastro' directory and the software
configuration files are directly under `reproduce/software/config'. A clean
build was done with this change and it didn't crash, but it may cause
crashes in derived projects, so after merging with Maneage, please
re-configure your project to see if anything has been missed. Please let us
know if there is a problem.
|
|
In the previous commit, I removed the year from the basic installation of
TeXLive packages, but I forgot to correct this in the high-level TeXLive
packages! This is corrected with this commit.
|
|
It is this time of year again: TeXLive has transitioned to its 2020 release
and the year is imprinted into the installation directory of TeXLive. Until
now, we have had to manually change this year and it caused complications
and was very annoying.
With this commit, the explicit year has been removed from TeXLive's
installation and we now simply put a `maneage' instead of the year. I tried
this on another system and it worked nicely. Until the time that we can
fully install LaTeX packages from source tarballs, this is the best thing
we could do for now.
|
|
Elham Saremi recently reported the following errors when building Numpy in
numpy/core/src/npysort/radixsort.c.src: "error: 'for' loop initial
declarations are only allowed in C99 or C11 mode". After some searching, I
found Issue 14147[1] on Numpy's main repository about the same problem. As
described there, apparently Numpy needs C99 compiler, but doesn't check for
it or set it manually (for some strange reason, leaving it to the packagers
to check if they want!!!).
Any way, after a check with Elham, we were able to fix it by adding the
`--std=c99' to CFLAGS of Numpy's build and with this commit, it is now
being implemented in the core Maneage to not cause a problem in any other
project.
[1] https://github.com/numpy/numpy/issues/14147
|
|
Until now, Astropy was instructed to build its own internal copy of the
Expat library. However, with the recent commits before, Maneage now
includes an installation of Expat and Astropy can't keep the two (its
internal version and the project's version) separate, so they conflict and
don't let Astropy get built.
With this commit, the problem is fixed by setting the Expat library as an
explicit dependency of Astropy and asking Astropy to ignore its internal
copy.
While doing this, I recognized that it is much easier and elegant to add
steps in various stages of the `pybuild' function through hooks instead of
variables. So the fifth argument of the `pybuild' function was removed and
now it actually checks if hooks are defined as functions and if so, they
will be called.
The `pyhook_after' function was also implemented in the installation of
`pybind11' (which needed it, given that the 5th argument of `pybuild' was
removed) and after doing a test-build, I noticed that two lines were not
ending with a `\' in `boost' (a dependency of `pybind11').
Commit written originally by Mohammad Akghlaghi
|
|
Until now we would simply return the version numbers as they were written
into the separate files and situations can happen where the version numbers
contain an underscore (`_'). However, this character is a methematical
character in LaTeX, causing LaTeX to complain and abort.
With this commit, a step has been added at the end of the configure script
to convert any possible `_' to `\_'. Once it is commented (a backslash is
put behind it), the underscore will be printed as it is in the final PDF.
This commit was originally written by Mohammad Akhlaghi
|
|
Until now, the M4 that was built on macOS had internal problems (as
discussed in #1): it would simply print `Abort trap: 6' in the output and
abort. After looking at the build of Homebrew, I noticed that they apply a
patch (correct one line) to fix this problem. To be able to apply that
patch on macOS systems, I had fully open up the build recipe of M4 and
atleast on the testing system, it was built successfully.
Also, after successfully building M4, and thus Autoconf and thus Minizip,
we were able to build XLSX I/O on a macOS and found out that the internal
library's full address wasn't being put in the libraries and
executables. With this commit, we now use macOS's `install_name_tool' to
correct the positions of the two `libxlsxio_*' libraries in all its
executables.
This commit was originally written by Mohammad Akhlaghi
|
|
Until this commit, only the word `Minizip' was written into the Minizip
installation target (without the version number of Minizip). With this
commit, this minor bug has been fixed by using the appropiate Make
variable: `$(minizip-version)'.
|
|
Minizip is a dependency of XLSX I/O and until now, I was just using the
most recent version I found (2.9.2), but XLSX I/O is written for the
Minizip 1.x series, not 2.x. Somehow it didn't cause a crash on my
computer!!! I think XLSX I/O's CMake is instructed to look into system
directories by default when it doesn't find the directories in the given
places. And because I had installed Minizip on my operating system, it
did't complain.
Upon trying the build on their systems, Yahya, Raul and Zahra reported a
failure in the build of XLSX I/O which was due the to the problem above (we
were installing the wrong version of Minizip!).
With this commit, this has been fixed by installing the 1.x series of
Minizip (whish is actually installed within zlib!).
This commit was original done by Mohammad Akhlaghi.
|
|
With this commit, CMake has been updated to its most recent version.
This upgrade has been done because in the installation of XLSX I/O on
macOS laptop, it crashes complaining about C compiler "not able to
compile a simple test program". After a fast search, I found it could be
possible to just use the most recent version of CMake to solve the
problem. But it didn't work. In any case, it is good to have the most
recent version of CMake included.
|
|
With this commit, multiples tabs in the definition of MissFITS tarball
have been removed. Now they are white spaces.
|
|
Until this commit, PyYAML was not set as prerequisite of Astropy. This
package is an optional dependency of Astropy for some particular
functions. However, we have already included PyYAML into this project so
it is available. With this commit, PyYAML has been set as a prerequisite
of Astropy.
In addition to this, Html5lib and Beautifulsoup4 have been also added as
prerequsites of Astropy (and removed from Astroquery prerequisites). I
noticed that both of them are optional dependencies of Astropy.
|
|
In the last update of Astropy to version 4.0 they removed some things
that the previous version of Astroquery needs. As a consequence, it is
also necessary to update the Astroquery version to be a ble to run with
the Astropy 4.0. With this commit, the update of Astroquery to it most
recent version (0.4) has been done.
|
|
Until now, the preparation phase was always executed before the final build
phase when running `./project make'. But when it becomes necessary, project
preparation can be slow and will un-necessarily slow down the project while
the project is growing (focus is on the analysis that is done after
preparation).
With this commit, preparation will be done automatically the first time
that the project is run (`.build/software/preparation-done.mk' doesn't
exist). However, after preperation is complete once, future runs of
`./project make' won't do preparation any more (by calling
`top-prepare.mk'). They will directly call `top-make.mk' for the analysis.
To manually invoke preparation after the first attempt, the `./project
make' script should be run with the new `--prepare-redo' option.
Also, since the preparation phase is now automatically done before the
analysis phase, the long notice that describes running `./project make' at
the end of the preparation phase has been removed in `top-prepare.mk'. It
now just prints a short line, saying the preparation has been complete.
Finally, when the project has not been run with the proper group
configuration, it ends with an `exit 1' so the main `./project' script
doesn't proceed any further.
|
|
MissFITS is package for manipulating FITS files.
I added it as my first commit to the project for educational
purposes.
|
|
Until now, the final preparation target of the preparation phase depended
on all the `$(makesrc)' files. This caused a problem because we were
telling it to also depend on `prepare.tex' (which is the same file that is
being built).
With this commit, we are applying the same solution we have already done in
`paper.mk' (for `paper.tex'): we are removing `prepare' from the list of
prerequisites.
This bug was found by Zahra Sharbaf.
|
|
Until now, like all software on GNU/Linux systems GCC would be built in RAM
(to speed up the build slightly and also not put too much stress on the
HDDs/SSDs). But some systems don't have enough RAM for building GCC and
will complain and crash.
With this commit, we have added a check on the amount of free space in the
`build_tmp' directory (which will be `/dev/shm' on GNU/Linux systems). If
the amount of free space isn't more than 10GB, then GCC won't be built
there and a temporary directory will be built under the `$(BDIR)/software'
directory for it.
This bug was found by Zahra Sharbaf.
This fixes bug #57853.
|
|
XLSX I/O is a very simple and fast program and library for reading and
writing `.xls' and `.xlsx' files (mainly used by Microsoft Excel) to CSV
files. It has two separate executables that can be called for an Excel file
and will output a CSV plain text file that can then be used within the
pipeline with more standard tools.
|
|
Until now we were mistakenly reporting the version of SED instead of
Texinfo.
With this commit, we corrected it!
This was reported by Raul Infante Sainz.
|
|
Until now, we defined `LIBRARY_PATH' to fix the problem of the `ld' linker
of Binutils needing several `*crt*.o' files to run. However, some software
(for example ImageMagick) over-write `LIBRARY_PATH', therefore there is no
other way than to put a link to these necessary files in our local build
directory.
With this commit, we fixed the problem by putting a link to the system's
relevant files in the local library directory. This fixed the problem with
ImageMagick. Later, when we build the GNU C Library in the project, we
should remove this step.
This bug reported by Raul Castellanos Sanchez.
|
|
Until now, when a Fortran compiler didn't exist on the host operating
system, the configure script would crash with a warning. But some projects
may not need Fortran, so this is just an extra/annoying crash!
With this commit, it will still print the warning, but instead of a crash,
it will just sleep for some seconds, then continue. Later, when if a
software needs Fortran, it's building will crash, but atleast the user was
warned.
In the future, we should add a step to check on the necessary software and
see if Fortran is necessary for the project or not. The project
configuration should indeed crash if Fortran is necessary, but we should
tell the user that software XXXX needs Fortran so we can't continue without
a Fortran compiler.
Also, a small sentence ("Project's configuration will continue in XXXX
seconds.") was added after all the warnings that won't cause a crash, so
user's don't think its a crash.
|
|
Until now, the main download script could only check one server for the
given URL. However, ultimately the actual server that a file is downloaded
from is irrelevant for this project: we actually check its
checksum. Especially in the case of software (which are distributed over
many servers), this can usually be very annoying: the servers may not
properly communicate with the running system and even the 10 trials won't
be enough.
With this commit, the download script
`reproduce/analysis/bash/download-multi-try' can take a new optional
argument (a 5th argument). It assumes this argument is a space-separated
list of server(s) to use as backup for the original URL. When downloading
from the original URL fails, it will look into this list and try
downloading the same file from each given server.
|
|
Until now, Make was just run ordinarily on the two Makefiles of the
software building phase. Therefore when there was a problem with one
software while building in parallel, Make would only complete the running
rules and stop afterwards. But when other rules don't depened on the
crashed rule, its a waste of time to stop the whole thing.
With this commit, both calls to Make in the `configure.sh' script are done
with the `-k' option (or `--keep-going' in GNU Make). With this option, if
a rule crashes, the other rules that don't depend on it will also be
run. Generally, anything that doesn't depend on the crashed rule will be
done. The `-k' option is a POSIX definition in Make, so it is present in
most implemenetations (for the call to `basic.mk').
|
|
Until now the shell scripts in the software building phase were in the
`reproduce/software/bash' directory. But given our recent change to a
POSIX-only start, the `configure.sh' shell script (which is the main
component of this directory) is no longer written with Bash.
With this commit, to fix that problem, that directory's name has been
changed to `reproduce/software/shell'.
|
|
Until now, the project would first ask for the basic directories, then it
would start testing the compiler. But that was problematic because the
build directory can come from a previous setting (with `./project configure
-e'). Also, it could confuse users to first ask for details, then suddently
tell them that you don't have a working C library! We also need to store
the CPATH variable in the `LOCAL.conf' because in some cases, the compiler
won't work without it.
With this commit, the compiler checking has been moved at the start of the
configure script. Instead of putting the test program in the build
directory, we now make a temporary hidden directory in the source directory
and delete that directory as soon as the tests are done.
In the process, I also noticed that the copyright year of the two hidden
files weren't updated and corrected them.
|
|
Rencely the building of GCC was allowed on Debian-based systems that have
their basic C library in architecture-specific directories, like
`/usr/lib/x86_64-linux-gnu'. However, these systems also have their headers
in non-standard locations, for example `/usr/include/x86_64-linux-gnu' and
this caused a crash on a new Ubuntu system.
/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No
such file or directory
27 | #include <bits/libc-header-start.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
The reason it didn't cause problems on other Ubuntus that we tested before
was historic: In the old days, we would ask Ubuntu systems to install
multilib features to have GCC. Because they had installed those features,
this problem didn't show up! But this wasn't mandatory!
With this commit, the `CPATH' environment variable is set (similar to how
`LIBRARY_PATH' was set) and this fixed the problem on a clean Debian
virtual machine.
This bug was reported by Sebastian Luna Valero.
|
|
Until now, the initial project scripts were primarily tested with GNU
Bash. But Bash is not generally available on all systems (it has many
features beyond POSIX). Because of this, effectively we were imposing the
requirement on the user that they must have Bash installed. We recently
started this with setting the shebang of `project' and
`reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul
and Gaspar reported an error on their systems.
To fix the problem, I installed Dash (a minimalist POSIX-compliant shell)
on my computer and temporarily set the shebangs to `/bin/dash', ran the
project configuration step and fixed all issues that came up. With this
commit, it can go all the way to building GCC on my system's Dash. After
this stage (when `high-level.mk' is called), there is no problem, because
we have our own version of GNU Bash and that installed version is used.
Probably some more issues still remain and will hopefully be found in the
future.
While doing this, I also noticed the following two minor issues:
- The `./project configure' option `--input-dir' was not recognized
because it was mistakenly checking `--inputdir'. It has been corrected.
- The test C programs now use the `<<EOF' method instead of `echo'.
- In `basic.mk', the extra space between `syspath' and `:=' was removed
(it was an ancient relic!).
|
|
Until now, the hashbang of these two shell scripts was set to `/bin/bash',
hence assuming that GNU Bash exists on the host system! But this is an
extra requirement on the host operating system and these two scripts should
be written such that they operate on a POSIX shell (the generic `/bin/sh'
which can point to any shell program).
With this commit this has been implemented! We may confront some errors as
the system is run on other systems, but we should fix such errors and work
hard to make these two scripts as POSIX-compatible as possible (runnable on
any shell, so as not to force users to install Bash before running the
project).
This completes Task #15525.
|
|
Until now, the main commands to run the project were these: `./project
configure' (to build the software), `./project prepare' (to possibly
arrange input datasets and build special configuration Makefiles) and
finally `./project make' to run the project.
The main logic behind the "prepare" phase `top-prepare.mk' is to build
configuration files that can be fed into the "make" step and optimize its
operation. For example when the total number of necessary inputs for the
majority of the analysis is not as large as the total number of
inputs. With "prepare" (when necessary), you go through the raw inputs,
select the ones that are necessary for the rest of the project. The output
of `top-prepare.mk' is a configuration file (a Make variable) that keeps
the IDs (numbers, names, etc). That configuration file would then be used
in the `top-make.mk' to identify the lower level targets and allow optimal
project organization and management.
But the last two are both part of the analysis, and while they indeed need
different calls to Make to be executed, many projects don't actually need a
preparation phase: ultimately, its an implementation choice by the project
developers and doesn't concern the project users (or the developers when
they are running it).
To avoid confusing the users, or simply annoying them when a projet doesn't
need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk'
Makefiles are called with the single `./project make' command and
`./project prepare' has been dropped. I noticed this while writing the
paper on this system.
|
|
Until now, the explanation for a missing static C library didn't actually
guide the users to look above and see the error message! So with this
commit, I edited it a little to be more clear (and mention to look
above). Also, I noticed that on Amazon AWS systems, the static C library is
installed as a separate package, so to help the users, I added the
necessary command and some better explanation.
|
|
Until now Perl was built after Coreutils, but I recently noticed that
Coreutils actually uses Perl while creating its manpages. So it is now
built before Coreutils.
Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils
can't build its man page for `md5sum'. The problem was found to be due to
the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our
installed library path in `basic.mk'. Therefore, it would crash because the
server had an older version of OpenSSL than the one that the template's
Coreutils was built with.
In the meantime (while addressing the issues above, because we only had one
thread on the AWS server) I also noticed a few programs that were using a
summarize compilation command (that just prints `CC xxx.c' instead of the
whole command) so I fixed them by adding `V=1'.
This bug was found by Idafen Santana Pérez.
|
|
Until now, the configuration Makefiles (in
`reproduce/software/config/installation' and `reproduce/analysis/config')
had a `.mk' suffix, similar to the workhorse Makefiles. Although they are
indeed Makefiles, but given their nature (to only keep configuration
parameters), it is confusing (especially to early users) for them to also
have a `.mk' (similar to the analysis or software building Makefiles).
To address this issue, with this commit, all the configuration Makefiles
(in those directories) are now given a `.conf' suffix. This is also assumed
for all the files that are loaded.
The configuration (software building) and running of the template have been
checked with this change from scratch, but please report any error that may
not have been noticed.
THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS
FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE
SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES
ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
|
|
GNU Make 4.3 was just announced, so I have updated it here is well. This
was important because until now the installable version was in alpha-mode
(4.2.90), now its a stable version.
|
|
In the previous commmit, I had forgot to add a `\' after the newly added
`sys_library_path' variable to the `high-level.mk' call.
|
|
Until now, GCC wouldn't build properly on Debian-based operating systems
because `ld' needed to link with several necessary C library features like
`crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to
add the directory containing them to `LIBRARY_PATH'. In the previous
commit, I actually searched for these files, but while testing on another
system, I noticed that it can be problematic (other architectures may
exist).
With this commit, we are actually finding the build architecture of the
running GCC (which is the same as the `ld') and using that to fix a fixed
directory to `LIBRARY_PATH'.
|
|
Until now, to see if a working static C library and `sys/cdefs.h' exist, we
were checking absolute locations like `/usr/include/sys/cdefs.h' or
`/usr/lib/libc.a' and `/usr/lib64/libc.a'. But this is not robust because
on different systems, they can be in different locations.
With this commit, we actually use `find' to find the location of `libc.a'
and use that to add elements to CPPFLAGS and LDFLAGS. This should fix the
problem on systems that have them on non-standard locations.
|
|
Until now, when find the versions of the TeXLive packages, we would assume
that `cat-date' is always present (because some packages don't have a
version!). However, apparently an update has been made in the TeXLive
Manager (`tlmgr') and `cat-date' is no longer present! As a result, none of
the TeXLive packages were being printed.
With this commit, it now assumes that `revision' is always present for
every package, but it also attempts to read `cat-date' (for backwards
compatability). When `cat-version' isn't present, it will try printing
`revision' and if that is also not present, it will print the date.
|
|
After a new rebuild of the project, I noticed that we now need to also
build the `mweights' package.
|
|
The checklist descriptions were slightly edited to be more clear. Also,
while following them, I noticed that while removing the "delete-me" parts
on `verify.mk', would cause an error: the `if [ $$m == delete-me ];'
statement we were saying to delete cause an error because `elif' was the
first statement Bash would see. So with this commit, the `download'
conditional (which isn't instructed to be deleted) was set to be the top
(with an `if') and the `delete-me' conditional now has an `elif'.
|
|
Until this commit, the checking of X11 installation done to ensure that
it is already available in the host system was crashing in macOS
systems. The reason is that the place of the X11 libraries use to be
`/opt/X11/lib' in macOS systems. With this commit, this issue has been
fixed by adding this directory to the LDFLAGS.
|
|
Newer versions of Astropy package has been released. With this commit,
it has been updated. It has been increased from v3.2.1 to v4.0
|
|
In markdown, to specify that part of the text should be shown as code we
need to put it between two `. But in `reproduce/software/make/README.md',
the code parts started with ` and finished in ' (like the file name above).
With this commit, this has been corrected and it will display properly on
web browsers.
|
|
Until now, if the file to be verified didn't exist, a different checksum
would be generated, and it would stop, but it wasn't immediately clear if
the differing checksum is because the file doesn't exist at all!
With this commit, before calculating the checksum, we first make sure if
the file exists. If it doesn't exist an explicit error is printed and thus
will help the project editor to find the cause of the problem.
|
|
Until now, the only verification that the template provided was the
published PDF. Users had to manually compare the published and generated
PDFs (numbers, plots, tables) and see if they obtained the same
result. However, this type of manual verification is not good and is prone
to frustration and missing important differences.
With this commit, a new Makefile has been added in the analysis steps:
`verify.mk'. It provides facilities to easily verify the results that go
into the paper. For example tables that go into making the paper's plots,
or the LaTeX macros that blend into the text. See the updated parts in
`README-hacking.md` for a more complete explanation.
This completes task #15497.
|
