| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Until now, when the figures were built directly from EPS
('\newcommand{\makepdf}{}' was commented), they would take the full
line-width becoming a little too large! I noticed this after letting arXiv
build the PDF.
With this commit, the 'includetikz' tool takes a second argument to be a
parameter given to 'includegraphics' (which is scale in this case).
|
|
Everything else regarding the submission to arXiv and Zenodo has been
complete, so I done a final read, making some minor edits to hopefully make
the text easier to read.
|
|
The previous explanation was not too clear and simply following it was
confusing. The issue was that with the tarball you have three scenarios: 1)
only build the PDF using existing figures. 2) only build the PDF, but build
the figures yourself, 3) build the full Maneaged project.
Hopefully this distinction is now more clear from the README.md file.
|
|
Some extra explanation can help the user understand the difference between
a Git-based project and a distributed tarball.
|
|
When the project is being re-built from the tarball (not the Git
repository), the 'tex/build' and 'tex/tikz' addresses are actual
directories, not symbolic links. In this case, when someone runs './project
configure', it will complain about not being able to delete them (it
assumes they are symbolic links!).
So with this commit, we first check if they are deletable without '-r'. If
so, then they are full directories and we rename them to a backup directory
to allow the rest of the project to continue building a link there.
|
|
This paper doesn't use pdflatex or biblatex, so it was necessary to make
some small corrections in the make-dist rule of initialize.mk. Also, while
testing the upload on arXiv, I noticed that it complains about an empty
'verify.tex' file, so that is also corrected.
|
|
All the steps following the to-be-added (in 'README-hacking.md')
publication checklist prior to the final check from new clone have been
added:
- 'README.md' file has been set.
- "Reproducible supplement" was added just above the keywords, pointing to
Zenodo.
- A link to the to-be-uploaded data underlying the plot was added in the
caption of the tools-per-year plot.
- A new meta-data configuration file was added to store basic project
metadata to be used throughout the project. This will later be taken
into Maneage. For examle the project title is now stored here and
written into the paper's LaTeX source and output datasets automatically.
- Verification was activated and plot's data and LaTeX macro files are now
automatically verified.
- A complete metadata was added for the data underlying the plot.
- A generic function was added in 'initialize.mk' that will automatically
write project info and copyright in all plain-text outputs.
|
|
The minor conflict was with 'reproduce/software/make/high-level.mk', and in
particular because we implemented the fix to Maneage's Task #15664 in this
project first. After it was moved to the main Maneage branch some minor
stylistic corrections were done to it, thus causing the conflict. To
resolve the conflict, I simply imported the full Maneage version of the
file with this command:
git checkout maneage -- reproduce/software/make/high-level.mk
The other conflicts were due to the deleted files (that were resolved as
described in 'README-hacking.md') and the LaTeX files that I had told
'.gitattributes' to ignore from the Maneage branch.
|
|
When some files should not be merged, until now we were suggesting to also
add deleted files to the '.gitattributes' file. However, this feature of
Git doesn't work for deleted files and they would still show up in the
'master' branch after a merge.
So with this commit, we have added a simple AWK command to run after a
merge that will automatically detect and delete such files (using the
output of 'git status --porcelain').
Also, two minor typos were corrected in the newly added
'servers-backup.conf' file: the copyright year was wrong and there was no
new-line at the end of the file (a good convention!).
|
|
Following a test merge, I noticed that the '.gitattributes' file is not
doing anything about the deleted files and also that all the files in
'tex/src/*.txt' should be added (they are too project-specific). So now it
only includes the files that aren't deleted.
For the files that are deleted, in the Maneage 'README-hacking.md' file, I
added an AWK command to easily remove them.
|
|
I noticed that we hadn't include the publication of the workflow and the
advantage that Maneage provides in this regard. So it was added at the end
of the proof-of-concept section. However, it was necessary to summarize
some other parts to not increase the wordcount.
|
|
Until now, Maneage would only build Flock before building everything else
using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid
parallel downloads during the building of software (which could cause
network problems). But after recently trying Maneage on FreeBSD (which is
not yet complete, see bug #58465), we noticed that the BSD implemenation of
Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq'
parts) and its shell also had some peculiarities.
It was thus decided to also install our own minimalist shell, Make and
compressor program before calling 'basic.mk'. In this way, 'basic.mk' can
now assume the same GNU Make features that high-level.mk and python.mk
assume. The pre-make building of software is now organized in
'reproduce/software/shell/pre-make-build.sh'.
Another nice feature of this commit is for macOS users: until now the
default macOS Make had problems for parallel building of software, so
'basic.mk' was built in one thread. But now that we can build the core
tools with GNU Make on macOS too, it uses all threads. Furthermore, since
we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have
to finish every line of a long rule with a backslash to keep variables and
such.
Generally, the pre-make software are now organized like this: first we
build Lzip before anything else: it is downloaded as a simple '.tar' file
that is not compressed (only ~400kb). Once Lzip is built, the pre-make
phase continues with building GNU Make, Dash (a minimalist shell) and
Flock. All of their tarballs are in '.tar.lz'. Maneage then enters
'basic.mk' and the first program it builds is GNU Gzip (itself packaged as
'.tar.lz'). Once Gzip is built, we build all the other compression software
(all downloaded as '.tar.gz'). Afterwards, any compression standard for
other software is fine because we have it.
In the process, a bug related to using backup servers was found in
'reproduce/analysis/bash/download-multi-try' for calling outside of
'basic.mk' and removed Bash-specific features. As a result of that bug-fix,
because we now have multiple servers for software tarballs, the backup
servers now have their own configuration file in
'reproduce/software/config/servers-backup.conf'. This makes it much easier
to maintain the backup server list across the multiple places that we need
it.
Some other minor fixes:
- In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'.
- In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD
(which doesn't have GCC).
- We are now using 'uname -s' to specify if we are on a Linux kernel or
not, if not, we are still using the old 'on_mac_os' variable.
- While I was trying to build on FreeBSD, I noticed some further
corrections that could help. For example the 'makelink' Make-function
now takes a third argument which can be a different name compared to the
actual program (used for examle to make a link to '/usr/bin/cc' from
'gcc'.
- Until now we didn't know if the host's Make implementation supports
placing a '@' at the start of the recipe (to avoid printing the actual
commands to standard output). Especially in the tarball download phase,
there are many lines that are printed for each download which was really
annoying. We already used '@' in 'high-level.mk' and 'python.mk' before,
but now that we also know that 'basic.mk' is called with our custom GNU
Make, we can use it at the start for a cleaner stdout.
- Until now, WCSLIB assumed a Fortran compiler, but when the user is on a
system where we can't install GCC (or has activated the '--host-cc'
option), it may not be present and the project shouldn't break because
of this. So with this commit, when a Fortran compiler isn't present,
WCSLIB will be built with the '--disable-fortran' configuration option.
This commit (task #15667) was completed with help/checks by Raul
Infante-Sainz and Boud Roukema.
|
|
These are some corrections that David sent to me by email and I am
committing here.
|
|
Antonio Diaz Diaz (author of the Lzip program/library), has had a very
supportive role in what became Maneage in the last 4 years. For example I
really started to appreciate the value of simplicity and archivability
while reading Lzip's documentation.
Fortunately he also read a recent version of the paper that was again very
supportive. Some of the minor points he raised had already been fixed, but
using 'supplier' instead of 'server' (in the Free Software) criterion was
new so I implemented it here with this commit. With this, I am also
thanking him for all his wonderful support and encouragement in the last 4
years.
|
|
Boud's point about a "random reader" not being a good example case was
correct. But "user" also gives it a software perspective that is ofcourse
not wrong, its can just be confusing. So I thought of changing it to
"interested reader".
In the part about the C-library dependency of high-level software, from
Boud's correction, I found out that it is very hard to convey what I wanted
to say (that separating errors due to C-library implementation and
measurement errors will be easy, because they should be on much different
scales). But I then corrected it to give it a slightly better tone while
mentioning the same thing: that with Maneage we can now accurately measure
the effect of the C library.
|
|
Changes with this commit are mostly minor and obvious. Some worth
commenting on include:
* `technologies develop very fast` - As a general statement, this
is too jargony, since technology is much wider than just
`software`; `some technologies` makes it clear that we're referring
to the specific case of the previous sentence
* `in a functional-like paradigm, enabling exact provenance` -
While `make` is not an imperative programming language, I don't
see how `make` is `like` a functional programming language.
Classifying it as a declarative and a dataflow programming
language and as a metaprogramming language would seem to go in
the right direction [1-3]. I also couldn't see how the language
type relates to tracking exact provenance.
But since we don't want to lengthen the text, my proposal is to
put `and efficient in managing exact provenance` without trying
to explain this in terms of a taxonomy of programming languages.
[1] https://en.wikipedia.org/wiki/Functional_programming
[2] https://en.wikipedia.org/wiki/Comparison_of_multi-paradigm_programming_languages
[3] https://en.wikipedia.org/wiki/Dataflow_programming
* `A random reader` - In the scientific programming context, `random`
has quite specific meanings which we are not using here; a `reader`
has not necessarily tried to reproduce the project. So I've proposed
`A user` here - with the idea that a `user` is more likely to be someone
who has done `./project configure && ./project make`.
* `studying this is another research project` - the present tense `is`
doesn't sound so good; I've put what seems to be about the shortest
natural equivalent.
Pdf word count: 5856
|
|
An "internally" was added to the part about core GNU tools accounting for
the differences between POSIX-compatible systems. One extra word was also
removed in the next sentence.
|
|
Hopefully, it is more to the point with these few word-corrections.
|
|
Konrad raised some very interesting points in particular about the
limitations of POSIX as a fuzzy standard that does not guaratee
reproducibility. A relatively long paragraph was thus added in the
discussion to address this important point.
In order to fit it in, the paragraph on "unwanted competition" was removed
since the POSIX issue was much more relevant for a curious
reader. Throughout the text, some other parts were edited to decrease the
length of the paper while making it easier to read.
|
|
Some of the redundant sentences have been removed and some minor edits
made.
|
|
The changes in this commit are best shown with `git diff --word-diff`
or `git patch --word-diff`. There are about half a dozen changes
of 1-2 words or a comma, the reasons should be obvious.
The sentence with "can not just" seems to be correct formally, but
"can not only" seems to me better to warn the reader that this
is a phrase of the form "can not only do X but can also do Y";
"can not just" sounds a bit like "You cannot just enter the room
without knocking" - it doesn't require a second part.
|
|
One major point was that following Konrad's suggestion the issue of not
being familiar with the Lisp/Scheme framework of GWL is now removed. We
actually mention the main problem we have had with Guix, but also highlight
that their solution was one of the main inspirations for this work.
|
|
Until this commit, when the user have previous TeX tarball already
present, the project crashed when trying to re-configure, if there was a
newer version of TeX. This is because TeX are updated yearly. With this
commit, this bug has been fixed. Now, during the installation of TeX, it
checks if this problem happens. If this is the case, then it moves the
old tarball, download the new one and install it. If not, it will just
install the already present tarball or crash because of any other
reason. This probem was recurrent, and each time TeX was updated, the
previous tarball had to be removed manually. But now, with this commit,
it is done automatically. The detection and fix of this bug has been
possible with the help of Mohammad Akhlaghi, thanks!
|
|
Until this commit, there was only a small description of me. With this
commit, I have added a small paragraph with my biography. I know we are
very restricted because of the word limit so I tried to be very short!
|
|
With this commit, I have corrected several minor typos.
|
|
With this commit, I did some minor changes in these Sections. Main
changes are: define the contraction `OS' from Operating System and use
only `OS' later on, and not use contractions like `isn't'
|
|
Before this commit: Roberto's bio was about 120 words. With this
commit: it is now less than 100 words. A comment about
reproducibility has been added.
|
|
Publishing a paper on reproducible research without making it easy for
readers to read the references would defeat the point. Of course we have to
make some compromises with some journals' reluctance to shift towards the
free world, but to satisfy scientific ethics, we should at least provide
clickable URLs to the references, preferably to the ArXiv version if
available [1], and also to the DOI, again, preferably to an open-access
version of the URL if available.
I was not able to fully get this done in the .bst file, so there's an
sed/tr hack done to the .bbl file in `reproduce/analysis/make/paper.mk` to
tidy up commas and spaces.
This commit also reverts some of the hacks in the Akhlaghi IAU Symposium
`tex/src/references.tex` entry, to match the improved .bst file,
`tex/src/IEEEtran_openaccess.bst`, provided here with a different name to
the original, in order to satisfy the LaTeX licence.
[1] https://cosmo.torun.pl/blog/arXiv_refs
|
|
David and Raul had both reported that because 'pdftotext' wasn't available
on their system, the project failed (even though the PDF was built!). So
with this commit, we first check if the system has 'pdftotext' and call it
only if its is available.
Some minor edits were made, building upon Boud's previous commit.
|
|
This commit provides mostly small changes. There didn't seem much point in
repeating the `lessons learned` jargon and claiming that we draw good
conclusions - insights - from our experience. Better just state what
hypotheses we have generated from the experience rather than give the
misleading impression that our hypotheses are well-established facts. In
the comments, I put a suggested translation of what the `lessons learned`
jargon means. I seem to have first heard this term in the mainstream media
a few years after the US 2003 attack on Iraq, when a US military
representative stated that the US forces had "learned lessons" after having
started a war of aggression against Iraq.
|
|
This commit changes two lines.
(1) Keeping the exact quote with the clerk while having a sentence that
makes sense in plain English cannot be done, it seems to me, without
making the sentence a bit longer. Here's one option that seems about
the best we can do, even though it still sounds a bit funny, because
it's hard to write a future conditional with the present "can". Since
it's a quote, it will probably survive the proofreaders.
(2) Software is an uncountable noun [1], so we say "software is", like
"water is"; "used software" sounds odd; I added "is itself" to
emphasise that we're especially talking about the full chain of
software for running the project. This commit modifies the "When the
..." sentence and hopefully sounds better.
[1] https://en.wiktionary.org/wiki/software#Noun
|
|
To help show the simplicity of 'top-make.mk', it was included as a
listing. I also went over some of Boud's corrections and made small
edits. In particular:
- The '\label' and '\ref' to a section were removed. I done this after
inspecting some of their recent papers and noticing that they generally
have a simple flow, without such redirections.
- In the part about the RDA adoption grant, I moved the "from the
researcher perspective" to the end. Because Austin+2017 is mainly
focused on data-center management, not the researcher's. They do touch
upon researcher solutions that can help data-base managers, but not
directly the researchers. In effect with this grant, they acknowledged
that our researcher-focused solution confirms with their criteria for
data-base management.
|
|
Possibly the least trivial edit in this commit is that the previous text
appeared to state that it's normal to find that a project prepared with
`maneage` may be ... unbuildable. Which would defeat our whole claim of
reproducibility! Obviously, `maneage` is still in a rapid development
stage and might still have significant, not-yet-detected bugs. But the
wording has to explain that this would constitute a bug in `Maneage` (in a
particular version of it), not an expected regular event. :) This commit
aims to fix that and other minor wording issues in IV.
Pdf word count 5855.
|
|
This series of commits aims to edit sections II+III, but first implements
the changes from 7bf5fcd, apart from one that conflicts in the abstract:
this commit has ``Maneage'' without `(managing+lineage)` in the abstract.
From Mohammad: this commit has been rebased after several other parallel
branches, so some things may differ from the message.
|
|
Until this commit, when the user had a previous TeXLive tarball already
present (in their software-tarball directory) compared to the CTAN server,
the project crashed in the configure phase. This was because TeXLive is
updated yearly and we don't yet install TeXLive from source (currently we
use its own package manager, but we plan to fix this in task #15267).
With this commit, we fix the problem by checking the cause of the crash
during the installation of TeX. If the crash is due to this particular
error, we ignore the old tarball and download the new one and install it
(the old one is still kept in '.build/software/tarballs', but will get a
'-OLD' in its name. This probem was recurrent, and every year that TeXLive
is updated, the previous tarball had to be removed manually! But with this
commit, this is done automatically. The detection and fix of this bug has
been possible with the help of Mohammad Akhlaghi, thanks!
|
|
One of the main reasons to building Maneage is to properly
acknowledge/attribute the authors of software in research. So we have
adopted a standard of never referring to the GNU-based operating systems
running the Linux kernel simply as "Linux", we avoid terms like "Open
Sourse" and use Free Software instead (in the same spirit).
With this commit, a few instances of the cases above have been corrected,
they had slipped through our fingers when we initially imported them into
the project. In the special case of the "Journal for Open Source Software",
we simply replaced it with its abbreviation (JOSS). This was done because
in effect we were generally using journal name abbreviations in almost all
the citations already. To avoid any inconsistancies, the names of the three
other journals that weren't abbreviated are also abbreviated.
|
|
Generally they were great, but after looking through them I thought a
hand-full of them slightly changed my original idea so I am correcting them
here. Boud, if you feel the changes aren't good, let's talk about it and
find the best way forward ;-).
They are mostly clear from a '--word-diff', just some notes on the ones
that have changed the meaning:
* On the "a clerk can do it" quotation, since its so short, I think its
better to keep its original form, otherwise a reader may thing there
were paragraphs instead of the "to" and we have changed their
intention.
* In the part where we are saying that the workflow can get "separated"
from the paper, I mostly meant to highlight that the data-centers and
journals (hosts) may diverge in decades, or one of them may go
bankrupt, or etc. Hence loosing the connection. The issue of it
evolving can in theory be addressed through version control, so I think
this is a more fundamental problem.
* In the part about free software, in the list, the original point was
the free software that are used by the project, not the project itself
(after all, the project itself falls under the "Open Science" titles
that is very fashionable these days, but my point here is to those
people who claim to do "Open Science" with closed software (like
Microsoft Excel!).
|
|
This commit makes several small changes to Section III, some of
which are quite significant in terms of meaning.
It was difficult to improve the clarity without extending
the word length. Now we're at 5901 words.
|
|
This commit implements quite a few minor changes in section II.
The aim of most is to clarify the meaning and remove ambiguity.
A few changes are that the reader will normally assume that
successive sentences in a paragraph are closely related in terms
of logical flow. It is superfluous - and considered excessive -
to put too many "Therefore"'s and "Hence"'s in (at least) modern
astronomy style. These are supposed to be used when there is a
strong chain of reasoning.
One change is done in the Introduction, because if we're going to
use "solution(s)" throughout to mean "reproducible workflow
solution(s)", then we have to clearly define this as jargon for
this particular paper. It's probably preferable to RWS - reproducible
workflow solution - or RWI - reproducible workflow implementation.
But we can't just keep saying "solution" because that has many
different meanings in a scientific context.
Pdf word count = 5880
|
|
This series of commits aims to edit sections II+III,
but first implements the changes from 7bf5fcd, apart
from one that conflicts in the abstract: this commit
has ``Maneage'' without `(managing+lineage)` in the
abstract.
|
|
This commit implements most of David's changes from c76727b, but excluding
some, such as the proposal to use 'which' in a restrictive clause in the
abstract. This is allowed, but the Fowler brothers' rule tends to followed
in science writing:
https://ell.stackexchange.com/questions/5/is-there-any-difference-between-which-and-that
A few points on the abstract:
* an immediate solution = singular
* The "immediate, fast short-term" benefits sentence sounded like it was
redundantly superfluously repetitively repeating doubled-up
information. Hopefully this edit is better.
* in the %Conclusion, "solutions" is vague, like people who say
"technology" when they're only talking about software, so this edit
reminds the reader to make the sentence more self-contained and
understandable.
|
|
With this commit, Maneage now includes instructions to build the memory
tracing tool Valgrind and the program 'patch' (to apply corrections/patches
in text files and in particular the sources of programs).
For this version of Valgrind, some patches were necessary for an interface
with OpenMPI 2.x (which is the case now). Also note that this version of
Valgrind's checks can fail with GCC 10.1.x (when using '--host-cc'), and
the failures aren't due to internal problems but due to how the tests are
designed (https://bugs.gentoo.org/707598). So currently if any of
Valgrind's checks fail, Maneage still assumes that Valgrind was built and
installed successfully.
While testing on macOS, we noticed that it needs the macOS-specific 'mig'
program which we can't build in Maneage. DESCRIPTION: The mig command
invokes the Mach Interface Generator to generate Remote Procedure Call
(RPC) code for client-server style Mach IPC from specification files. So a
symbolic link to the system's 'mig' is now added to the project's programs
on macOS systems.
This commit's build of Patch and Valgrind has been tested on two GNU/Linux
distributions (Debian and ArchLinux) as well as macOS.
Work on this commit started by Boud Roukema, but also involved tests and
corrections by Mohammad Akhlaghi and Raul Infante-Sainz.
|
|
David reported this problem, it happened right after importing IEEEtran,
but for some reason, it didn't happen for me.
|
|
When entering the name of the "listings" package, I had forgot to add the
final 's', so it wasn't being installed on a clean system! I didn't have a
problem until now, because it remained from previous builds.
|
|
After a look at the PDFs of the linked papers of the previous commit and a
few 2020 papers, we noticed that the biography format of the webpage and
PDFs are different! So it is now back in its old way (which is how
biographies are presented in the PDF).
A few other minor edits were made in the text.
|
|
It appears from looking at
https://ieeexplore.ieee.org/document/5725236/authors#authors
https://ieeexplore.ieee.org/document/7878935/authors#authors
that the affiliations section needs to start with a one-phrase
definition of the author's main affiliation. In 5725236,
the typesetters/proofreaders swapped van der Walt and Colbert,
so don't be confused by that. It shows that nobody proofread
properly.
With this commit, each author's institute (single hierarchical
level) is written as the first paragraph of the author's
affiliation section. Since 5725236 allows a very-well-known
acronym, I'm guessing that IAC can be defined for Mohammad
and then re-used for the others.
I've added a brief CV for me. If necessary, we could compress
my main research together as "observational cosmology", but let's
see how we go in the word count.
I have not (yet) worked through the main text.
There is also one minor language fix - `Because is complete` was
incomplete.
Pdf word count: 5873
|
|
After one day not looking at the first draft of this new version (commit
7b008dfbb9b2), I went through the text and done some general edits to make
its presentation and logic smoother.
|
|
Before this commit: several typos were present along the text. With this
commit several typos have been corrected (types listed below) and my bio
has been added.
a) double words
b) general typos
c) comas after adverbs at the beginning of a sentence
d) contractions are removed, e.g., don't vs do not
e) three sentences in parenthesis have been removed since I think they
were out of context or unnecessary
f) etc
|
|
In order to correspond to the updated datalineage plot, the name of the
plotted columns was changed to 'columns.txt', but I had forgot to update it
in the LaTeX source and since the old file still remained I hadn't
noticed. This was found by Boud and corrected.
|
|
Until now, two of the software BibTeX sources (Matplolib and Sympy) had an
"abstract" entry that was long, not similar to the rest, and not relevant
in this context, so they are removed with this commit.
|
