| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Since working on the pipeline will evolve along with the projects that use
it, it can be useful for projects to fetch updates in the pipeline. So the
checklist in `README.md' updated to explain how to do this cleanly.
|
|
Until now, because we didn't build the dependencies internally, it was
important for the pipeline to be usable with any version of Make. But
because of the new installation of dependencies (including GNU Make), that
is no longer the case. So we can safely use GNU Make and this needs to be
mentioned in `README.md'.
|
|
Gnuastro is just one of the softwares used in the pipeline, so it is too
much to include its version just under the abstract. However, the version
of the pipeline is very important as well as the link to it. This change
puts more emphasis on these two more important points.
|
|
When there is a problem in creating the final TeX Live installation, the
previous version of the pipeline would not understand and just finish! We
would later have problems in building the paper.
So the following series of steps were taken: to keep the recipes in a
shorter and easier to understand way, the steps to install TeX Live are now
one rule (that produce `.local/bin/texlive-ready-tlmgr' when its
successful), and the steps to install the necessary packages are in another
rule (that produce `.local/bin/texlive-ready' when its successful).
When control comes back inside configure, if `.local/bin/texlive-ready'
isn't there (something failed during the TeX Live installation, or building
packages), then the whole TeX Live installation directory
(`.local/texlive') will be deleted along with the two output files. This
will help ensure that future steps can check the availablility of a working
TeX Live in the pipeline.
|
|
After installing the basic dependencies, we have access to the internally
built `nproc' program (by Coreutils) to find the number of threads we can
use for building the high-level batch of dependencies. Unfortunately, I had
added an extra `./' before `$instdir' when calling `nproc' so the script
failed. This problem is fixed now.
|
|
The system's libraries are no longer used in building the higher-level
dependencies. Also, thanks to Raul Infante Sainz, we found out that Bash's
build script was still removing the extra directory information (not
good!).
|
|
GNU Coreutils are basic programs that can help in the configuration of
higher-level programs. Because of that, it was a dependency of almost all
software built in `dependencies.mk'. To make things more clear, easier to
read and faster (when building in parallel), the building of Coreutils is
now moved to the `dependencies-basic.mk' rules. There, it is built
along-side Bash. Since `dependenceis-basic.mk' is run and completed before
`dependencies.mk', with this, we can be sure that Coreutils is present by
the time we want to build the higher-level programs.
Also, Zlib is now added as a dependency of Git also (it is necessary for
its build).
|
|
Until now, we were building Libtool as a high-level `top-level-programs'
software. But all tools that use the `./configure' script already have a
version of Libtool in them. So ultimately the `libtool' in the PATH is not
used. However, in the case of Gnuastro, we need libtool for running
BulidProgram. So in effect, its a dependency of Gnuastro.
|
|
When the C compiler is not GNU GCC, linking with GNU Binutils is going to
cause problems. So until the time that we can include GCC into this
pipeline, its best to avoid Binutils also.
Also, for building CMake, we were relying on an installed CMake, but now,
we are using its own `./bootstrap' script, so it can be built even if the
host system doesn't have CMake.
Also, for TeX Live, we are now setting a custom file as main target to
avoid complications with symbolic links as targets in Make.
Finally, when the user says they don't want to re-write an existing
configuration file, no extra notices will be printed and the configure
script will immediately start building programs.
|
|
After going through the checklist for starting a new project based on the
pipeline, I noticed some parts that could be modified to be more
clear. They are now applied.
|
|
Until now, we were using a customized `tar.lz' tarball for Gzip. But on
systems that don't have GNU Tar, this will cause a problem (non-GNU Tar
doesn't recognize `.tar.lz'). So to keep things simple, we are using the
customized gzip in `tar.gz' format. After the internal build of GNU Tar and
Lzip, the default method of unpacking (`tar xf XXXXX.tar.XX') will work
nicely on all the standard compression algorithms and we don't have to
modify our commands based on the algorithm (nice feature of GNU Tar).
|
|
To help in debugging on other systems, the building of dependencies is only
done on a single thread.
|
|
The two README files have been updated to explain the new feature of
downloading and building dependencies.
|
|
Since the final product of the pipeline is a LaTeX-created PDF file, it was
necessary to also have LaTeX within the pipeline. With this commit, TeX
Live is also built as part of the configuration and all the necessary
packages to build the PDF are also installed and mentioned in the paper
along with their versions.
|
|
TeX Live is now also downloaded and built by the reproduction
pipeline. Currently on the basic (TeX and LaTeX) source is built but no
extra packages, so the PDF building will fail. We'll add them in the next
commit.
|
|
In the previous commit, for testing the static build, I had added a
`-ljunk' option to the compiler. But I had forgot to remove it! It is
removed now.
|
|
If the Makefile `$(static_build)' variable in
'/reproduce/src/make/dependencies-build-rules.mk' isn't defined (by
mistake), it will default to blank space, then the Shell will complain
about a bad formatted operator (needing two operands). So an `x' was added
before it and before `yes' which will allow us to safely pass such cases
without a terrible crash.
|
|
The default Mac compiler has problems building static libraries. Since we
are not yet building the GNU C Compiler as part of the pipeline, we'll have
to rely on the host system's compiler. Therefore, a check is now added a
the start of the configure script that will build a minimal program with
the `-static' flag and if it fails, it will print a warning. Afterwards,
none of the dependencies will be built with the `-static' flag.
|
|
Some minor corrections have been made in the paper's text to make things
easier to read and be more formal.
|
|
A semi-colon was missed in defining the link for `zlib'. It is now added.
|
|
To have better control over the build, GNU Binutils, Bzip2, GNU Gzip, and
XZ Utils have also been added to the pipeline. Some other minor cleanups
and fixes were also implemented throughout the process.
|
|
Until now, when a package was to be built statically, we were adding the
`--static' option to `CFLAGS'. This was the wrong place to put it! It
should be in the linking step (thus `LDFLAGS'). Also, based on Bash's
configure script, we are now using the more generic form of `-static'
(single dash, not double dash).
On the other hand, the `--disable-shared' option isn't available in many of
the packages and it is highly redundant with the `-static' option, so it
has been removed to avoid an extra warning in such packages.
|
|
To ensure the easy unpacking and building of the programs, Lzip and Tar are
now also build during the initial setup phase.
Some minor corrections were also applied to make things cleaner and
smoother.
|
|
Until now, we used semicolons in Make's Call function definitions to build
the programs with GNU build system or CMake. Therefore, if any step of the
process failed, the rest would be ignorant to it and pass. Now, we use `&&'
to separate the different processing steps. In this way, we can be sure
that if any of them fails (during configuration, or building for example),
the pipeline will also stop and not continue to the next command (in the
same recipe).
Since the two Make Call functions were identical in the two
`dependencies-basic.mk' and `dependencies.mk', they are now in one file to
be imported in both.
This bug was found by Raul Infante Sainz.
|
|
All the used software are now acknowledged in the template paper along with
their versions. This section is also mentioned in the check list, so users
don't delete it by mistake.
|
|
After a test by Raúl Infante Sainz, we found out that the configure script
and the Make script for Bash and Make are making too many assumptions on
more recent versions of both. As a result, it couldn't be built.
Therefore, the `configure' script was modified to not use more recent tools
like `readlink' (to find the absolute address of a relative one). It was
also re-organized to not have to read the configuration parameters from a
text file. The parameters are directly read from the command-line and are
written into the proper file afterwards. This removes the need to opening a
text editor by the user (which also caused problems on Raúl's system).
To fix the Make version issue, the building of Bash and Make are now done
in a new Makefile (`reproduce/src/make/dependencies-basic.mk'). This file
doesn't make many of the assumptions that were made in
`dependencies.mk'. So it should hopefully work on any version of Make.
To help in debugging, for now, the Makefile of configure, are asked to work
on one thread (the `-j' option is commented in the `configure'). But after
checks, we'll fix this.
|
|
All the libraries that define their version string as a macro in their
headers are now also checked in `reproduce/src/make/initialize.mk'.
Also, the CFITSIO tarball now follows the same versioning style as the rest
of the tarballs: a script is added to convert the version string into what
is included in the tarball.
|
|
The version of all programs is now checked in
`reproduce/make/src/initialize.mk' and the pipeline won't complete if any
of the program versions change from those listed in
`reproduce/config/pipeline/dependency-versions.mk'.
Since the pipeline is systematically checking all program versions, we
don't need Gnuastro's `--onlyversion' option any more. So it (and all
references to it) have been removed.
|
|
The system's environment is now removed from the internal processing of the
pipeline, except for LaTeX.
|
|
We were mistakenly using GSL's name for the unpacked tarball.
|
|
During the configuration step several new programs that were necessary for
a more complete controlled environment are now also downloaded and built
statically.
|
|
The host web address of most of the necessary packages was blank (filled
with `WWWWWWWWWWWWWWWW' as a place holder). They now point to the correct
webpages.
|
|
To enable easy/proper reproduction of results, all the high-level
dependencies are now built within the pipeline and installed in a fixed
directory that is added to the PATH of the Makefile. This includes GNU Bash
and GNU Make, which are then used to run the pipeline.
The `./configure' script will first build Bash and Make within itself, then
it will build
All the dependencies are also built to be static. So after they are built,
changing of the system's low-level libraries (like C library) won't change
the tarballs.
Currently the C library and C compiler aren't built within the pipeline,
but we'll hopefully add them to the build process also.
With this change, we now have full control of the shell and Make that will
be used in the pipeline, so we can safely remove some of the generalities
we had before.
|
|
After a full trial of the checklist, some further minor edits were made to
make it more clear.
|
|
Some minor changes were made to be more clear.
|
|
A few minor points were corrected in README.md.
|
|
A step was added close to the top of the checklist to remind people to
check the pipeline before making any changes. Also, the `--origin' option
was removed from the `git clone' command into a separate command to rename
the origin branch. This helps in readability.
|
|
Until now, in the check list of `README.md', we were recommending to delete
the history of the pipeline and start your own history from that. But this
disables users of the pipeline to keep it up to date with new features that
are added to it.
With this commit, the main branch is now called `pipeline' (to allow users
to use `master' for their own research) and in the clone command, the
pipeline's remote is now called `pipeline-origin' (to allow the user to use
`origin' for their own remote).
|
|
While trying the checklist, I noticed that I had forgot to add my name
after the copyright year and that `reproduce/src/make/paper.mk' still had
my own name on it, the copyright notice also said `script' instead of
`Makefile' which is now corrected.
|
|
While testing the reproduction pipeline on a small project, I noticed some
parts of the checklist that were either repetative or needed to be
corrected. This is done with this commit.
|
|
We had previously started the `configure' script with `/bin/bash'. But this
script is meant to check for Bash inside of it. So to be run-able (on a
system which may not have Bash), the `configure' script has to be run by
`/bin/sh'.
|
|
In the previous commit, ` was mistakenly written as '. This was only
noticed after pushing and rendering the Markup on the webpage.
|
|
A minor edit was made in the copyright correction part of the checklist.
|
|
To be more generic (and avoid listing file names in `README.md'), a command
is now used to find the files where the name has to be added to the
copyright notice. Also, I noticed that `reproduce/src/make/paper.mk' lacked
a placeholder name/email, so one was added.
|
|
To be more easy to read/understand, the comments in `.dir-locals.el' were
edited and made a little more elaborate.
|
|
Previously we had tried to corrected the problem that `.mk' files use space
instead of TAB by setting them to be read into makefile-mode at the
start. However, for some reason, this caused a problem in reading the
maximum line length (the value of 75 wasn't read by Emacs).
So after some playing around, I found out that the problem is that the
`nil' setting (for all other files) must be defined in the end, not the
start of the basic settings and this solved this problem.
|
|
Different implementations of AWK may use different random number
generators, so even setting the seed will not ensure a reproducible
result. Because of this, the random plot may be different when the
pipeline runs on different systems and this can confuse early users
(its contrary to the exact reproducibility that is the whole purpose
of this pipeline).
The plot is just a simple X^2 plot, showing the squared value of the X
axis on the Y axis. It is very simple, but atleast it will be
identical on all systems. Also, there may be too many complicated
things in the pipeline already for an early user, and its just a
demonstration, so the easier/simpler, the better.
|
|
While we had set the default Makefile SHELL to be bash, we weren't
actually checking if `bash' is available on the system. With this
commit, it is also checked at configure time.
|
|
On some systems, the default shell `/bin/sh' doesn't point to Bash and
this can cause problems and failures when the designer uses its
features. Bash (and its extra features) make things very easy and it
is very ubiquitous, so it is safe to assume users will have it.
This problem was reported by Alejandro Serrano Borlaff.
|
|
Since version 26.1, apparently Emacs doesn't tream `.mk' files in
Makefile-mode. So thanks to Mosè Giordano, a line was added in the
`.dir-locals.el' file so we can be sure they are always treated as
Makefiles and make things more convenient for Emacs users.
|
