| Age | Commit message (Collapse) | Author | Lines |
|---|
|
A few minor conflicts occurred and were fixed.
|
|
I hadn't updated the abstract since first writing it. With this commit, it
has been updated to be more precise and generically interesting, focusing
more on the principles and usability. I also greatly improved the section
on publishing the workflow.
|
|
There weren't any conflicts in this merge.
|
|
The subdirectories here (and the fact that they may be symbolic links) may
be confusing for some early project users, so a `README.md' file was added
there describing them and when they are links, when directories and when
some may not yet exist.
|
|
Until this commit, the year of this paper was 2019 and the linking url
was the temporal one. However, the final official publication year is
2020. With this commit, the year and the url have been changed to the
final ones.
|
|
With this commit, multiples tabs in the definition of MissFITS tarball
have been removed. Now they are white spaces.
|
|
This section (of sharing a build directory between multiple members of the
project) is also a good features of Maneage.
|
|
The figure was greatly improved, becoming much more clear and descriptive
of some of the main advantages of having version control in a complete
project like Maneage.
|
|
With the main structure of Maneage explained, I have started to explain how
a new project is created, along with a schematic diagram that shows two
scenarios of how Git can help with project management.
|
|
Until now, the introduction had repeated several things and also had a
relatively long list of things to add in its end. Also, it was highly
focused to scientific papers.
With this commit, I effectively re-wrote it, with the starting paragraphs
becoming more industry-friendly, while also focusing on the scientific
cases. Many of the repetative parts were removed and the listed items in
the end were put into the text in a much better context.
Also, now that the name of the system involves "lineage" (and a lot of
focus is put on it in the start) the terms data provenance and lineage were
defined in the definition section.
Some other intersting points that I encountered during the research on
definitions were added to the discussion and final lists, and the DOI of
one reference paper was corrected.
|
|
With this commit a description of these two important parts have been added
to the project, along with several figures showing various parts of the
files that are discussed. I also done some other restructuring of the
figures and files to make things fit better into the the description of the
paper.
|
|
Until this commit, PyYAML was not set as prerequisite of Astropy. This
package is an optional dependency of Astropy for some particular
functions. However, we have already included PyYAML into this project so
it is available. With this commit, PyYAML has been set as a prerequisite
of Astropy.
In addition to this, Html5lib and Beautifulsoup4 have been also added as
prerequsites of Astropy (and removed from Astroquery prerequisites). I
noticed that both of them are optional dependencies of Astropy.
|
|
In the last update of Astropy to version 4.0 they removed some things
that the previous version of Astroquery needs. As a consequence, it is
also necessary to update the Astroquery version to be a ble to run with
the Astropy 4.0. With this commit, the update of Astroquery to it most
recent version (0.4) has been done.
|
|
Until now, I was mistakenly multiplying the fraction of papers in that
journal. This is corrected with this commit.
|
|
In order to make the description more clear and readable, the rules in the
demonstrated Makefile (and their links to the data lineage plot) were made
more clear.
|
|
Until now, there was no explanation on an actual analysis phase, therefore
with this commit an example scenario with a readable Makefile is included.
The Data lineage graph was also simplified to both be more readable, and
also to correspond to this new explanation and subMakefile.
Some random edits/typos were also corrected and some references added for
discussion.
|
|
Until now, the preparation phase was always executed before the final build
phase when running `./project make'. But when it becomes necessary, project
preparation can be slow and will un-necessarily slow down the project while
the project is growing (focus is on the analysis that is done after
preparation).
With this commit, preparation will be done automatically the first time
that the project is run (`.build/software/preparation-done.mk' doesn't
exist). However, after preperation is complete once, future runs of
`./project make' won't do preparation any more (by calling
`top-prepare.mk'). They will directly call `top-make.mk' for the analysis.
To manually invoke preparation after the first attempt, the `./project
make' script should be run with the new `--prepare-redo' option.
Also, since the preparation phase is now automatically done before the
analysis phase, the long notice that describes running `./project make' at
the end of the preparation phase has been removed in `top-prepare.mk'. It
now just prints a short line, saying the preparation has been complete.
Finally, when the project has not been run with the proper group
configuration, it ends with an `exit 1' so the main `./project' script
doesn't proceed any further.
|
|
MissFITS is package for manipulating FITS files.
I added it as my first commit to the project for educational
purposes.
|
|
Until now, the final preparation target of the preparation phase depended
on all the `$(makesrc)' files. This caused a problem because we were
telling it to also depend on `prepare.tex' (which is the same file that is
being built).
With this commit, we are applying the same solution we have already done in
`paper.mk' (for `paper.tex'): we are removing `prepare' from the list of
prerequisites.
This bug was found by Zahra Sharbaf.
|
|
Until now, like all software on GNU/Linux systems GCC would be built in RAM
(to speed up the build slightly and also not put too much stress on the
HDDs/SSDs). But some systems don't have enough RAM for building GCC and
will complain and crash.
With this commit, we have added a check on the amount of free space in the
`build_tmp' directory (which will be `/dev/shm' on GNU/Linux systems). If
the amount of free space isn't more than 10GB, then GCC won't be built
there and a temporary directory will be built under the `$(BDIR)/software'
directory for it.
This bug was found by Zahra Sharbaf.
This fixes bug #57853.
|
|
I had forgot to add a `./' before the call to `project' for the
`--check-config'.
|
|
This was done just to get going with describing the analysis process.
|
|
The main problems with this dataset was the names of the journals (which
sometimes have single quotes or apostrophes in them that is really annoying
for SED)! But ultimately, for the simple study we want to do here, the
journal names are irrelevant, so in the end I just ignored the names. Later
we can set an identifier for the journals if necessary.
But now we have the basic information in a way that is usable in a plot to
show in this paper.
|
|
Until now, there was no easy way to read/write `.xlsx' files (Microsoft
Excel spreadsheets) within the template. But XLSX I/O provides to simple
programs and some libraries to easily convert `.xlsx' files to CSV that can
easily be read by any tool.
This has also been implemented in the core template branch.
|
|
XLSX I/O is a very simple and fast program and library for reading and
writing `.xls' and `.xlsx' files (mainly used by Microsoft Excel) to CSV
files. It has two separate executables that can be called for an Excel file
and will output a CSV plain text file that can then be used within the
pipeline with more standard tools.
|
|
The text was slightly improved/edited and I also recently came up to the
Menke et al. 2020 (DOI:10.1101/2020.01.15.908111) which also has some good
datasets we can use as a demonstration here.
|
|
Until now we were mistakenly reporting the version of SED instead of
Texinfo.
With this commit, we corrected it!
This was reported by Raul Infante Sainz.
|
|
Until now, we defined `LIBRARY_PATH' to fix the problem of the `ld' linker
of Binutils needing several `*crt*.o' files to run. However, some software
(for example ImageMagick) over-write `LIBRARY_PATH', therefore there is no
other way than to put a link to these necessary files in our local build
directory.
With this commit, we fixed the problem by putting a link to the system's
relevant files in the local library directory. This fixed the problem with
ImageMagick. Later, when we build the GNU C Library in the project, we
should remove this step.
This bug reported by Raul Castellanos Sanchez.
|
|
Until now, when a Fortran compiler didn't exist on the host operating
system, the configure script would crash with a warning. But some projects
may not need Fortran, so this is just an extra/annoying crash!
With this commit, it will still print the warning, but instead of a crash,
it will just sleep for some seconds, then continue. Later, when if a
software needs Fortran, it's building will crash, but atleast the user was
warned.
In the future, we should add a step to check on the necessary software and
see if Fortran is necessary for the project or not. The project
configuration should indeed crash if Fortran is necessary, but we should
tell the user that software XXXX needs Fortran so we can't continue without
a Fortran compiler.
Also, a small sentence ("Project's configuration will continue in XXXX
seconds.") was added after all the warnings that won't cause a crash, so
user's don't think its a crash.
|
|
Until now, the main download script could only check one server for the
given URL. However, ultimately the actual server that a file is downloaded
from is irrelevant for this project: we actually check its
checksum. Especially in the case of software (which are distributed over
many servers), this can usually be very annoying: the servers may not
properly communicate with the running system and even the 10 trials won't
be enough.
With this commit, the download script
`reproduce/analysis/bash/download-multi-try' can take a new optional
argument (a 5th argument). It assumes this argument is a space-separated
list of server(s) to use as backup for the original URL. When downloading
from the original URL fails, it will look into this list and try
downloading the same file from each given server.
|
|
While reading over the already written parts (and hopefully complete the
paper), they were edited/corrected to be more clear.
|
|
Some edits were made after rereading of some parts.
|
|
There were no conflicts in this merge.
|
|
Until now, when `./project make' was run after an insuccessful run of
`./project configure', it would just say to run `./project configure'. But
for a first time user, this could be confusing because when the
configuration is done in parallel, the error message can be very high on
the command-line outputs and not seen clearly.
With this commit, the error message is more complete and describes the
problem and what the users should do in which circumstance.
|
|
Until now, Make was just run ordinarily on the two Makefiles of the
software building phase. Therefore when there was a problem with one
software while building in parallel, Make would only complete the running
rules and stop afterwards. But when other rules don't depened on the
crashed rule, its a waste of time to stop the whole thing.
With this commit, both calls to Make in the `configure.sh' script are done
with the `-k' option (or `--keep-going' in GNU Make). With this option, if
a rule crashes, the other rules that don't depend on it will also be
run. Generally, anything that doesn't depend on the crashed rule will be
done. The `-k' option is a POSIX definition in Make, so it is present in
most implemenetations (for the call to `basic.mk').
|
|
Until now the shell scripts in the software building phase were in the
`reproduce/software/bash' directory. But given our recent change to a
POSIX-only start, the `configure.sh' shell script (which is the main
component of this directory) is no longer written with Bash.
With this commit, to fix that problem, that directory's name has been
changed to `reproduce/software/shell'.
|
|
Until now, the project would first ask for the basic directories, then it
would start testing the compiler. But that was problematic because the
build directory can come from a previous setting (with `./project configure
-e'). Also, it could confuse users to first ask for details, then suddently
tell them that you don't have a working C library! We also need to store
the CPATH variable in the `LOCAL.conf' because in some cases, the compiler
won't work without it.
With this commit, the compiler checking has been moved at the start of the
configure script. Instead of putting the test program in the build
directory, we now make a temporary hidden directory in the source directory
and delete that directory as soon as the tests are done.
In the process, I also noticed that the copyright year of the two hidden
files weren't updated and corrected them.
|
|
Rencely the building of GCC was allowed on Debian-based systems that have
their basic C library in architecture-specific directories, like
`/usr/lib/x86_64-linux-gnu'. However, these systems also have their headers
in non-standard locations, for example `/usr/include/x86_64-linux-gnu' and
this caused a crash on a new Ubuntu system.
/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No
such file or directory
27 | #include <bits/libc-header-start.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
The reason it didn't cause problems on other Ubuntus that we tested before
was historic: In the old days, we would ask Ubuntu systems to install
multilib features to have GCC. Because they had installed those features,
this problem didn't show up! But this wasn't mandatory!
With this commit, the `CPATH' environment variable is set (similar to how
`LIBRARY_PATH' was set) and this fixed the problem on a clean Debian
virtual machine.
This bug was reported by Sebastian Luna Valero.
|
|
Until this commit, the small section of `README-hacking.md' in which it
is explained how to do the first configuration of Git was at the
beginning of the section `First custom commit'. However, it is better to
have it just before the item `Your first commit' in that section. With
this commit, this change has been done. Now the reader has the necessary
steps for configuring Git just before it is needed for making the first
commit.
|
|
Until now, the initial project scripts were primarily tested with GNU
Bash. But Bash is not generally available on all systems (it has many
features beyond POSIX). Because of this, effectively we were imposing the
requirement on the user that they must have Bash installed. We recently
started this with setting the shebang of `project' and
`reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul
and Gaspar reported an error on their systems.
To fix the problem, I installed Dash (a minimalist POSIX-compliant shell)
on my computer and temporarily set the shebangs to `/bin/dash', ran the
project configuration step and fixed all issues that came up. With this
commit, it can go all the way to building GCC on my system's Dash. After
this stage (when `high-level.mk' is called), there is no problem, because
we have our own version of GNU Bash and that installed version is used.
Probably some more issues still remain and will hopefully be found in the
future.
While doing this, I also noticed the following two minor issues:
- The `./project configure' option `--input-dir' was not recognized
because it was mistakenly checking `--inputdir'. It has been corrected.
- The test C programs now use the `<<EOF' method instead of `echo'.
- In `basic.mk', the extra space between `syspath' and `:=' was removed
(it was an ancient relic!).
|
|
With this commit, the general outline of the analysis phase is given, as
well as a description of the LaTeX macros and their relation to the paper
and thier verification.
Also, the data-lineage figure was updated to have references.tex also and
some resizing of the folders in file-architecture to be more clear.
|
|
In the last few days I have been writing these two sections in the middle
of other work. But I am making this commit because it has already become a
lot! I am now going onto the description of `./project make'.
|
|
Until now, the hashbang of these two shell scripts was set to `/bin/bash',
hence assuming that GNU Bash exists on the host system! But this is an
extra requirement on the host operating system and these two scripts should
be written such that they operate on a POSIX shell (the generic `/bin/sh'
which can point to any shell program).
With this commit this has been implemented! We may confront some errors as
the system is run on other systems, but we should fix such errors and work
hard to make these two scripts as POSIX-compatible as possible (runnable on
any shell, so as not to force users to install Bash before running the
project).
This completes Task #15525.
|
|
Until now, the main commands to run the project were these: `./project
configure' (to build the software), `./project prepare' (to possibly
arrange input datasets and build special configuration Makefiles) and
finally `./project make' to run the project.
The main logic behind the "prepare" phase `top-prepare.mk' is to build
configuration files that can be fed into the "make" step and optimize its
operation. For example when the total number of necessary inputs for the
majority of the analysis is not as large as the total number of
inputs. With "prepare" (when necessary), you go through the raw inputs,
select the ones that are necessary for the rest of the project. The output
of `top-prepare.mk' is a configuration file (a Make variable) that keeps
the IDs (numbers, names, etc). That configuration file would then be used
in the `top-make.mk' to identify the lower level targets and allow optimal
project organization and management.
But the last two are both part of the analysis, and while they indeed need
different calls to Make to be executed, many projects don't actually need a
preparation phase: ultimately, its an implementation choice by the project
developers and doesn't concern the project users (or the developers when
they are running it).
To avoid confusing the users, or simply annoying them when a projet doesn't
need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk'
Makefiles are called with the single `./project make' command and
`./project prepare' has been dropped. I noticed this while writing the
paper on this system.
|
|
Since I (Raul) did some changes (and I hope to do more :-)) in the
`README-hacking.md', I am adding my information at the beginning of this
file.
|
|
Until this commit, we were asuming that Git was already properly
configured. However, in order to be as complete as possible, it would
be good if the basic commands to configure Git were in the
`README-hacking.md'. With this commit, a small paragraph has been added
in order to have the basic Git configuration commands (i.e. to configure
the name, email, and favorite text editor).
|
|
Until now, the explanation for a missing static C library didn't actually
guide the users to look above and see the error message! So with this
commit, I edited it a little to be more clear (and mention to look
above). Also, I noticed that on Amazon AWS systems, the static C library is
installed as a separate package, so to help the users, I added the
necessary command and some better explanation.
|
|
Until now Perl was built after Coreutils, but I recently noticed that
Coreutils actually uses Perl while creating its manpages. So it is now
built before Coreutils.
Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils
can't build its man page for `md5sum'. The problem was found to be due to
the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our
installed library path in `basic.mk'. Therefore, it would crash because the
server had an older version of OpenSSL than the one that the template's
Coreutils was built with.
In the meantime (while addressing the issues above, because we only had one
thread on the AWS server) I also noticed a few programs that were using a
summarize compilation command (that just prints `CC xxx.c' instead of the
whole command) so I fixed them by adding `V=1'.
This bug was found by Idafen Santana Pérez.
|
|
Until now the file architecture plot at the directories ontop of the
top-level files. This made it hard to visually identify the top-level
files. They are not placed ontop of the sub-directories and some space is
added to highlight the files in the top-level directory and those in the
subdirectories.
Two other changes were made:
- The symbolic links created in the top source directory are also shown.
- The coding of this figure was made much more elegant by defining a
PGFPlots node class and just changing the things that are direrent
between each directory.
|
|
It was a little hard to describe the file structure so instead of using a
standard listing as most papers do, I thought of showing the file and
directory structure as boxes within each other (modeled on the Gnome
disk-utility).
Some other polishing was done throughout the paper also.
|
