| Age | Commit message (Collapse) | Author | Lines |
|---|
|
This was done just to get going with describing the analysis process.
|
|
The text was slightly improved/edited and I also recently came up to the
Menke et al. 2020 (DOI:10.1101/2020.01.15.908111) which also has some good
datasets we can use as a demonstration here.
|
|
While reading over the already written parts (and hopefully complete the
paper), they were edited/corrected to be more clear.
|
|
Some edits were made after rereading of some parts.
|
|
With this commit, the general outline of the analysis phase is given, as
well as a description of the LaTeX macros and their relation to the paper
and thier verification.
Also, the data-lineage figure was updated to have references.tex also and
some resizing of the folders in file-architecture to be more clear.
|
|
In the last few days I have been writing these two sections in the middle
of other work. But I am making this commit because it has already become a
lot! I am now going onto the description of `./project make'.
|
|
Until now the file architecture plot at the directories ontop of the
top-level files. This made it hard to visually identify the top-level
files. They are not placed ontop of the sub-directories and some space is
added to highlight the files in the top-level directory and those in the
subdirectories.
Two other changes were made:
- The symbolic links created in the top source directory are also shown.
- The coding of this figure was made much more elegant by defining a
PGFPlots node class and just changing the things that are direrent
between each directory.
|
|
It was a little hard to describe the file structure so instead of using a
standard listing as most papers do, I thought of showing the file and
directory structure as boxes within each other (modeled on the Gnome
disk-utility).
Some other polishing was done throughout the paper also.
|
|
Until now, I was writing the paper without the template. But we will soon
be adding a tutorial to the template, and I thought it will be good to have
an example demonstration here too. So I just brought the hole project into
the template structure, allowing us to add the template analysis later when
its ready, and also allowing us to easily reproduce this paper ofcourse
(without having to worry about the host's TeXLive installation.
|
|
The unnecessary parts were removed and the project now runs.
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
Since we got the RDA Adoption grant, it was necessary to add it in the
acknowledgements.
|
|
Until now, the paper's title and author information were set it
`tex/src/preamble-header.tex'. But they are actually shown in the final PDF
paper and a much better place to keep them is the top-level `paper.tex'.
With this commit, the setting of the title and author names has been moved
to `paper.tex', just after importing all the preambles. However, the basic
package importation and low-level settings are still set in
`tex/src/preamble-header.tex', because they are relatively low-level.
This task was suggested by Deepak (Indian Institute of Astrophysics).
|
|
After the checklist was applied in the 5th Indo-French Astronomy School, we
found some cases in the checklist that were extra (and thus had to be
removed), or were needed (and thus were added).
Also the non-necessary steps for a first commit were moved to a
separate/new section in the checklist for the people to add after doing
their first commit.
Also, the software part of the paper was moved to an appendix.
|
|
Until now, the `tex/build' symbolic link was put in the clone/source tree
when the build-directory's `tex' directory was being built. Thanks to
Roberto Baena, we just found a bug because of this behavior: when a second
group member is trying to build the pipeline, since the build directory's
`tex' directory already exists, no `tex/build' will be put in their
clone/source directory. As a result, the PDF building will crash.
To fix this (and keep things organized), the two `tex/build' and `tex/tikz'
links (to the build directory) are now built in the configure step while it
is building all the top-level directories. They are no longer built within
the Makefiles.
Also, a comment was added on top of every directory built during the
configuration phase to be clear.
This fixes bug #56362.
|
|
Until now, the software building and analysis steps of the pipeline were
intertwined. However, these steps (of how to build a software, and how to
use it) are logically completely independent.
Therefore with this commit, the pipeline now has a new architecture
(particularly in the `reproduce' directory) to emphasize this distinction:
The `reproduce' directory now has the two `software' and `analysis'
subdirectories and the respective parts of the previous architecture have
been broken up between these two based on their function. There is also no
more `src' directory. The `config' directory for software and analysis is
now mixed with the language-specific directories.
Also, some of the software versions were also updated after some checks
with their webpages.
This new architecture will allow much more focused work on each part of the
pipeline (to install the software and to run them for an analysis).
|
|
All occurances of "pipeline" have been chanaged to "project" or "template"
withint the text (comments, READMEs, and comments) of the template. The
main template branch is now also named `template'.
This was all because `pipeline' is too generic and couldn't be
distinguished from the base, and customized project.
|
|
Until now, the files where the people were meant to change didn't have a
proper copyright notice (for example `Copyright (C) YOUR NAME.'). This was
wrong because the license does not convey copyright ownership. So the name
of the file's original author must always be included and when people
modify it (and add their own copyright-able modifications).
With this commit, the file's original author (and email) are added to the
copyright notice and when more than one person modified a file, both names
have their individual copyright notice.
Based on this, the description for adding a copyright notice in
`README-hacking.md' has also been modified.
|
|
With this commit, we are applying the new style of citing software within
the build rule of Gnuastro.
|
|
Until now, management of the software names and versions in the paper was
done manually (a macro had to be defined in `initialize.mk', then used in
`paper.tex', so they had to be manually set in two places). Managing this
was not easy.
To fix this, with this commit, each software building rule's target is a
text file that contains its human-readable name and its version. In the
end, the configure script sorts them by their name and writes them into a
LaTeX input file that we can easily import as a file into the main paper.
|
|
Until now, these versions were written in each run. This was mainly
inherited from the old days of the pipeline, where we didn't know the
software on the host. But now that we have almost everything under control,
we can just write these LaTeX macros at the end of the configure script and
make `initialize.mk' simpler and also (very slightly!) speed-up/simplify
the processing.
|
|
We were developing the build of Numpy and Scipy on Mac in a parallel thread
and things seems to be working relatively nice now. There were only two
problems:
1) GCC still has some random building issues on Mac.
2) ATLAS shared libraries can't be built on Mac (so we used OpenBLAS to
build Numpy and Scipy on both Mac and GNU/Linux).
But for now, none of these problems are critical. So, we can progress in
one branch.
There were only very minor conflicts in the merge.
|
|
After doing a systematic search for files without a copyright notice, a few
more were found that didn't have a notice. So a notice was added for them.
I used this Bash command to find the files:
for f in $(find ./ -type f); do \
if [[ $f != *.git* ]]; then \
n=$(grep -i copyright $f | wc -l); \
echo "$n $f"; \
fi; \
done | awk '$1==0'
|
|
In order to be more clear, a copyright statement was added to all the LaTeX
and README files.
|
|
In the previous commit I already have included the latex macros.
In this commit we use them in the paper source.
|
|
Conflicts in `gcc' build comments and in mentioning software used in
paper fixed.
|
|
In this commit we add `h5py' Python package.
We also include `setuptools' as a main dependency of Python because with the
previous commit it (as well as `pip') is no longer installed with Python.
Numpy version also has been incremented.
|
|
The LaTeX macro for libgit2 was not properly used in `paper.tex'.
On Mac systems, after browsing the directory, a `.DS_Store' file was
created. So to keep things clean on those systems, it is added to the files
to be ignored by Git.
|
|
Astropy was added and one very important thing is that we have to
use the pypi tarball (https://pypi.org/) (which is bootstrapped)
and not the github tarball.
|
|
In order to collaborate effectively in the project, even project members
that don't necessarily want (or have the capacity) to do the whole analysis
must be able to contribute to the project. Until now, the users of the
distributed tarball could only modify the text and not the figures (built
with PGFPlots) of the paper.
With this commit, the management of TeX source files in the pipeline was
slightly modified to allow this as cleanly as I could think of now! In
short, the hand-written TeX files are now kept in `tex/src' and for the
pipeline's generated TeX files (in particular the old `tex/pipeline.tex'),
we now have a `tex/pipeline' symbolic-link/directory that points to the
`tex' directory under the build directory.
When packaging the project, `tex/pipeline' will be a full directory with a
copy of all the necessary files. Therefore as far as LaTeX is concerned,
having a build-directory is no longer relevant. Many other small changes
were made to do this job cleanly which will just make this commit message
too long!
Also, the old `tarball' and `zip' targets are now `dist' and `dist-zip' (as
in the standard GNU Build system).
|
|
With this commit, it is now possible to package the project into a tarball
or zip file, ready to be distributed to collaborators who only want to
modify the final paper (and not do the analysis technicalities), or for
uploading to sites like arXiv, or online LaTeX sharing pages.
|
|
In this version, too many extra notices (just regarding a change from
branch to branch) are not printed with `-q'. Instead only a one line
statement is printed that it is saved or applied.
|
|
Metastore depends on `bsd/string.h' to work properly (atleast on GNU/Linux
systems). The first system I tried building with had that library, so I
didn't notice! With this commit, we also build `libbsd' as part of the
pipeline.
Also, I couldn't find libbsd's version in any of its installed headers, so
like Libjpeg, we can't actually check and will directly write our internal
version into the paper.
|
|
The pipeline heavily depends on file meta data (and in particular the
modification dates), for example the configuration-Makefiles within the
pipeline are set as prerequisites to the rules of the pipeline.
However, when Git checks out a branch, it doesn't preserve the meta-data of
the files unique to that branch (for example program source files or
configuration-Makefiles). As a result, the rules that depend on them will
be re-done.
This is especially troublesome in the scenario of this reproducible paper
project because we commonly need to switch between branches (for example to
import recent work in the pipeline into the projects). After some
searching, I think the Metastore program is the best solution. Metastore is
now built as part of the pipeline and through two Git hooks, it is called
by Git to store the original meta-data of files into a binary file that is
version controlled (and managed by Metastore).
|
|
Until now, Gnuastro was only mentioned in the first acknowledgments
section, but not in the paragraph with all the program names. But these two
are not mutually exclusive. All the software should be mentioned in the
last paragraph and those that need special mention can be mentioned before
it.
|
|
Readline is a prerequisite of Bash and AWK, while NCURSES is a prerequisite
of Readline. With the recent update of GNU Bash (and thus GNU Readline) on
my host operating system, the pipeline crashed and I noticed this hole in
the pipeline. In particular, AWK (which linked with Readline 7.0) would
complain about not finding it and abort.
|
|
Both Gzip and Gnuastro were being bootstrapped personally from their Git
repository until now. But fortunately a new release of both came out last
week and so to make things standard we are now using their standard
tarballs.
I also noticed that we weren't checking the version of Gzip or mentioning
it in the acknowledgement section. This was also corrected.
|
|
The TeX Live installer needs Wget to operate smoothly, especially on recent
Mac OS systems that don't have Wget pre-installed. Also, it would be good
for the pipeline to have its own downloader. So with this commit, the
pipeline also installs Wget and OpenSSL which is a dependency.
Many other small changes/fixes were done in this process.
|
|
Until now we weren't explicity writing the full path of the dynamic
libraries necessary for linking a program. But now with
`-Wl,-rpath=$(ildir)' we ensure that the linker keeps the address of the
dynamic libraries necessary for linking at linking time, not running
time. Also, `pkg-config' is also built when preparing the basics. Several
other minor corrections were made thanks to the great help of Raúl Infante
Sainz.
|
|
The high-level dependencies are now built without having access to the
system's PATH. To do this, all the necessary software that we aren't
building ourselves are now brought into the installed `bin/' directory
using a symbolic link to the corresponding software on the host. To do
this, it was also necessary to increase the number of basic/low-level
packages that we are building, and add several more (Diffutils and
Findutils).
With this process in place, we now have a list of the exact software
packages that we are not building our selves, enabling easy building of all
such dependencies in the future.
|
|
Until now, we were keeping the input file within the reproduction
pipeline's directories using the same name as the database/server. Now, we
are using a short/summarized filename convention for the input dataset.
|
|
In most analysis situations (except for simulations), an input dataset is
necessary, but that part of the pipeline was just left out and a general
`SURVEY' variable was set and never used. So with this commit, we actually
use a sample FITS file from the FITS standard webpage, show it (as well as
its histogram) and do some basic calculations on it.
This preparation of the input datasets is done in a generic way to enable
easy addition of more datasets if necessary.
|
|
Gnuastro is just one of the softwares used in the pipeline, so it is too
much to include its version just under the abstract. However, the version
of the pipeline is very important as well as the link to it. This change
puts more emphasis on these two more important points.
|
|
When the C compiler is not GNU GCC, linking with GNU Binutils is going to
cause problems. So until the time that we can include GCC into this
pipeline, its best to avoid Binutils also.
Also, for building CMake, we were relying on an installed CMake, but now,
we are using its own `./bootstrap' script, so it can be built even if the
host system doesn't have CMake.
Also, for TeX Live, we are now setting a custom file as main target to
avoid complications with symbolic links as targets in Make.
Finally, when the user says they don't want to re-write an existing
configuration file, no extra notices will be printed and the configure
script will immediately start building programs.
|
|
Since the final product of the pipeline is a LaTeX-created PDF file, it was
necessary to also have LaTeX within the pipeline. With this commit, TeX
Live is also built as part of the configuration and all the necessary
packages to build the PDF are also installed and mentioned in the paper
along with their versions.
|
|
Some minor corrections have been made in the paper's text to make things
easier to read and be more formal.
|
|
To have better control over the build, GNU Binutils, Bzip2, GNU Gzip, and
XZ Utils have also been added to the pipeline. Some other minor cleanups
and fixes were also implemented throughout the process.
|
|
To ensure the easy unpacking and building of the programs, Lzip and Tar are
now also build during the initial setup phase.
Some minor corrections were also applied to make things cleaner and
smoother.
|
|
All the used software are now acknowledged in the template paper along with
their versions. This section is also mentioned in the check list, so users
don't delete it by mistake.
|
|
Different implementations of AWK may use different random number
generators, so even setting the seed will not ensure a reproducible
result. Because of this, the random plot may be different when the
pipeline runs on different systems and this can confuse early users
(its contrary to the exact reproducibility that is the whole purpose
of this pipeline).
The plot is just a simple X^2 plot, showing the squared value of the X
axis on the Y axis. It is very simple, but atleast it will be
identical on all systems. Also, there may be too many complicated
things in the pipeline already for an early user, and its just a
demonstration, so the easier/simpler, the better.
|
