| Age | Commit message (Collapse) | Author | Lines |
|---|
|
SUMMARY: Nothing special is necessary for your existing projects. This
commit just addds two new features (read the commit description for more):
1. To provide a user and password to servers that need authentication
before they allow downloading of proprietary data,
2. To use the FITS Standard's DATASUM for file verification (for cases
where the file is not static on the server, and is generated upon
receiving your download request).
Until now, Maneage didn't have any infrastructure for databases that
require authentication (through a user or password, when calling
'wget'). Furthermore, when the downloaded file is automatically generated
by the server upon request, the server usually adds metadata (like file
date, or query number and etc) in the header. Therefore the simple SHA256
checksum of the file would differ on every download! This made it very hard
to verify if the data (not headers) are unchanged.
With this commit, both these problems have been addressed:
- Server authentication: the 'reproduce/software/config/LOCAL.conf' now
contains three new variables for this purpose. With them, you can give
your username and password, along with the authentication method of the
server. The comments on top of these three variables give a full
description of their usage.
- Verifying only the data in a file (ignoring the headers): The
'reproduce/analysis/config/INPUTS.conf' now accepts two new optional
variables for each input file using the FITS standard's DATASUM
convention: 'INPUT-%-fitsdatasum' and 'INPUT-%-fitshdu'. If the SHA256
isn't specified for a file, Maneage will use these to verify the
file. With the latter, you specify the HDU of the data you want to
verify and with the former you give the DATASUM value for that HDU. As
the name suggests, this is only valid for FITS files. If we find other
formats that support a similar behavior, we can add this feature for
those formats also. This is also thoroughly discussed in the comments of
'reproduce/analysis/config/INPUTS.conf'.
This commit was done with the help of Pedram Ashofte Ardakani, Sepideh
Eskandarlou and Mohammadreza Khellat.
|
|
SUMMARY: This is a software update to make Maneage more portable and up to
date. It does not involve any Maneage infrastructure changes. You should
just re-build your project to make sure the updated software haven't
removed/changed any of their features that you were using. In particular,
for Astrometry.net users, please see the respective note in P.S.2 below.
Until now, there have been many updates in the software that are built
within Maneage. The last software upadte was almost one year ago.
With this commit, the software in the P.S.1 have been updated. A
description of notable changes in the software environment is given in
P.S.2. This software environment has been tested on an Arch GNU/Linux,
Debian, CentOS-7 and macOS.
This commit is the merging of 24 individual commits by Raul Infante-Sainz
(who put a lot of energy on porting the software below for macOS, and
updating citations), Boudewijn Roukema (who helped with memory checking for
GCC, and testing on Debian and CentOS), Sepideh Eskandarlou (who tested the
environment) and myself.
Besides the updates in the core software, the followimg improvements have
also been implemented in this commit:
- When you run './project shell':
- A welcome message is printed that will remind the caller that they
have entered a new environment, it will print the location of 'HOME'
and the location of the shell startup file.
- The 'reproduce/software/shell/bashrc.sh' is loaded as a startup
file. This allows you to customize your interactive Maneage shell. A
default step has already been placed there that will put the git
branch name (in green) within the shell prompt (which was
purple). This greatly helps when dealing with directories under Git
version control. These settings won't bother with Maneage's default
operations: through environment variables we make sure that these
'./project shell' features will not slow-down the calls to the shell
within the non-interactive Make calls.
- The host's 'COLORTERM' is passed to the Maneage environment. It is
used by some programs that can have color outputs on the terminal.
- Updates to citations:
- Numpy and Scipy (as requested on their pages):
https://numpy.org/citing-numpy and https://scipy.org/citing-scipy
- Gnuastro: Added https://arxiv.org/abs/1909.11230 which describes major
updates to Gnuastro after 10 releases.
- When a software's paper is indexed in the SAO/NASA Astrophysics Data
System (ADS), Maneage now use the BibTeX entries provided by ADS. This
helps to give a unified format to most software, and more information
(like ADS+arXiv hyperlinks in the BibLaTeX compilation of the default
bibliography).
- We were able to build this version of Maneage on a Debian system from
2010 (+12 years ago!). Only three downgrades were necessary in the
"basic" software (not affecting the high-level science software!). A
description of the necessary downgrades for such old systems has been
added in 'README.md'.
P.S.1 List of updated software:
Basic software:
cURL 7.79.1 --> 7.84.0
Dash 0.5.11.5 --> 0.5.11-057cd65
File 5.41 --> 5.42
GNU AWK 5.1.0 --> 5.1.1
GNU Bash 5.1.8 --> 5.2-rc2
GNU Binutils 2.37 --> 2.39
GNU Compiler Collection (GCC) 11.2.1 --> 12.1.0
GNU Findutils 4.8.0 --> 4.9.0
GNU Gzip 1.11 --> 1.12
GNU Help2man 1.48.5 --> 1.49.2
GNU Integer Set Library (ISL) 0.18 --> 0.24
GNU Libtool 2.4.6 --> 2.4.7
GNU Nano 6.0 --> 6.4
GNU Readline 8.1.1 --> 8.2-rc2
GNU libiconv 0.16 --> 0.17
Git 2.36.0 --> 2.37.1
OpenSSL 3.0.0 --> 3.0.5
PatchELF 0.13 --> 0.15.0
Perl 5.34.0 --> 5.36.0
High-level software:
Astrometry.net 0.89 --> 0.91
CFITSIO 4.0.0 --> 4.1.0
CMake 3.21.4 --> 3.24.0
GNU Astronomy Utilities (Gnuastro) 0.16.1 --> 0.18
GPL Ghostscript 9.55.0 --> 9.56.1
HDF5 1.10.5 --> 1.13.1
Libjpeg 9d --> 9e
Libtiff 4.3.0 --> 4.4.0
OpenBLAS 0.3.18 --> 0.3.21
PLplot n/a --> 5.15.0
Python 3.10.0 --> 3.10.6
SCAMP 2.6.7 --> 2.10.0
SWarp 2.38.0 --> 2.41.5
Util-Linux 2.37.2 --> 2.38.1
Vim 8.2 --> 9.0
WCSLIB 7.7 --> 7.11
X.org packages (used by graphical software like Ghostscript and LaTeX):
Fontconfig 2.13.94 --> 2.14.0
LibX11 1.7.2 --> 1.8
LibXCB 1.14 --> 1.15
XCB-proto 1.14.1 --> 1.15
Xorg-proto 2021.5 --> 2022.1
Python modules:
Astropy 5.0 --> 5.1
GalSim 2.3.3 --> 2.3.5
P.S.2: Notable points regarding the software environment:
- Two new links from the host's low-level tools are now included in
Maneage's build environment:
- On GNU/Linux systems, the host's 'ldd' is linked inside the custom
environment. This belongs to the GNU C Library (which is not yet
installed in Maneage). But helps in checking the linking status of the
binaries on GNU/Linux systems.
- On macOS: the 'codesign' binary is included, which is used by GNU
Emacs on macOS to sign the built executable.
- GNU Bison has been moved in basic software (necessary for GNU Binutils).
- The Zip and Unzip programs have been moved as high-level software that
have to be manually requested when necessary. This is because they are
not used by any of the basic software anymore. They were just installed
as dependencies of GNU Tar to be close the other compression
programs. Also, in the past we would use the original tarballs, and some
(for example Numpy) were distributed in Zip format. However, by default,
we now use a custom Lzip tarball and don't need Zip or Unzip. This was
suggested by Zahra Sharbaf and Raul Infante-Sainz.
- Some minor edits in 'reproduce/software/shell/tarball-prepare.sh'. In
particular the 'awk' command was effectively just replacing a '_' with
'-', so it just uses a simple SED expression instead.
- Fixed bug 62700 (https://savannah.nongnu.org/bugs/index.php?62700) by
compiling 'xz' with a patched version of the xz source file
'src/liblzma/liblzma.map'.
- Astrometry.net doesn't depend on NetPBM any more. NetPBM (and its
dependencies) were causing many crashes on macOS and it also a very
strange build system that is hard to maintain. Astrometry.net uses it to
take images as input. However, it isn't necessary when you provide
Astrometry.net with a catalog. Therefore, Raul added some instructions
on how to run astrometry from your own custom X-Y catalog. These
instruction can be seen on top of the build rule of Astrometry.net in
'reproduce/software/make/high-level.mk'.
- h5py has been removed as a dependency of Astropy. It is an optional
dependency to write tables into HDF5 format. But since we couldn't get
it to build on macOS it has been removed. None of the current Maneage
users/developers also use this feature of Astropy!
- PLplot is added a new software, but not a default pre-requisite of SCAMP
(which can use it to generate figures), because there were many build
problems on macOS. Instructions have been added on top of SCAMP on how
to add PLplot as a dependency.
- With the aim of being able to install Plplot on macOS, we have wrote
several lines to fix header problems. However, we didn't succeed. In any
case we are leaving these lines in case they are useful in the future.
- The '-Wno-nullability-completeness' compiler flag (which is primarily
necessary for macOS) is now only added for macOS systems. It was causing
many warnings of un-recognized option in GNU/Linux systems.
- The 'mkswap' program of Util-Linux has been disabled because it caused
crashes on older kernels. Generally, its not necessary for a Maneage
project because it needs root permissions to run!
- LibXT (of the x.org software) has been added as a dependency of Cairo.
- ImageMagick and Lzip were using the host's C++ standard library! But on
GNU/Linux we build our own C++ Standard Library with GCC, so with this
commit, they properly link with Maneage's C++ standard library.
- ImageMagick on macOS couldn't properly link with Maneage's Ghostscript
library! This has been fixed using macOS's install_name_tool.
- Necessary RAM to build GCC on GNU/Linux systems changed to ~8GB, see
https://savannah.nongnu.org/task/?16244#comment12
- Pythran is no longer as prerequisite of Scipy. Until now, Pythran was a
prerequisite of Scipy. But we noticed that it is optional and was
causing problems on macOS.
- The URLs of some of the software have been updated in
'reproduce/software/config/urls.conf'. By default, these are all
commented, but they can be useful when searching for new versions or
when a project needs custom software that is not (yet) in Maneage.
|
|
This commit primarily affects the configuration step of Maneage'd projects,
and in particular, updated versions of the many of the software (see
P.S.). So it shouldn't affect your high-level analysis other than the
version bumps of the software you use (and the software's possibly
improve/changed behavior).
The following software (and thus their dependencies) couldn't be updated as
described below:
- Cryptography: isn't building because it depends on a new
setuptools-rust package that has problems
(https://savannah.nongnu.org/bugs/index.php?61731), so it has been
commented in 'versions.conf'.
- SecretStorage: because it depends on Cryptography.
- Keyring: because it depends on SecretStorage.
- Astroquery: because it depends on Keyring.
This is a "squashed" commit after rebasing a development branch of 60
commits corresponding to a roughly two-month time interval. The following
people contributed to this branch.
- Boudewijn Roukema added all the R software infrastructure and the R
packages, as well as greatly helping in fixing many bugs during the
update.
- Raul Infante-Sainz helped in testing and debugging the build.
- Pedram Ashofteh Ardakani found and fixed a bug.
- Zahra Sharbaf helped in testing and found several bugs.
Below a description of the most noteworthy points is given.
- Software tarballs: all updated software now have a unified format
tarball (ustar; if not possible, pax) and unified compression (Lzip) in
Maneage's software repository in Zenodo
(https://doi.org/10.5281/zenodo.3883409). For more on this See
https://savannah.nongnu.org/task/?15699 . This won't affect any extra
software you would like to add; you can use any format recognized by
GNU Tar, and all common compression algorithms. This new requirement is
only for software that get merged to the core Maneage branch.
- Metastore (and thus libbsd and libmd) moved to highlevel: Metastore
(and the packages it depends on) is a high-level product that is only
relevant during the project development (like Emacs!): when the user
wants the file meta data (like dates) to be unchanged after checking
out branches. So it should be considered a high-level software, not
basic. Metastore also usually causes many more headaches and error
messages, so personally, I have stopped using it! Instead I simply
merge my branches in a separate clone, then pull the merge commit: in
this way, the files of my project aren't re-written during the checkout
phase and therefore their dates are untouched (which can conflict with
Make's dates on configuration files).
- The un-official cloned version of Flex (2.6.4-91 until this commit) was
causing problems in the building of Netpbm, so with this commit, it has
been moved back to version 2.6.4.
- Netpbm's official page had version 10.73.38 as the latest stable
tarball that was just released in late 2021. But I couldn't find our
previously-used version 10.86.99 anywhere (to see when it was released
and why we used it! Its at last more than one year old!). So the
official stable version is being used now.
- Improved instructions in 'README.md' for building software environment
in a Docker container (while having project source and output data
products on the local system; including the usage of the host's
'/dev/shm' to speed up temporary operations).
- Until now, the convention in Maneage was to put eight SPACE characters
before the comment lines within recipes. This was done because by
default GNU Emacs (also many other editors) show a TAB as eight
characters. However, in other text editors, online browsers, or even
the Git diff, a TAB can correspond to a different number of
characters. In such cases, the Maneage recipes wouldn't look too
interesting (the comments and the recipe commands would show a
different indentation!).
With this commit, all the comment lines in the Makefiles within the
core Maneage branch have a hash ('#') as their first character and a
TAB as the second. This allows the comment lines in recipes to have the
same indentation as code; making the code much more easier to read in a
general scenario including a 'git diff' (editor agnostic!).
P.S. List of updated software with their old and new versions
- Software with no version update are not mentioned.
- The old version of newly added software are shown with '--'.
Name (Basic) Old version New version
------------ ----------- -----------
Bzip2 1.0.6 1.0.8
CURL 7.71.1 7.79.1
Dash 0.5.10.2 0.5.11.5
File 5.39 5.41
Flock 0.2.3 0.4.0
GNU Bash 5.0.18 5.1.8
GNU Binutils 2.35 2.37
GNU Coreutils 8.32 9.0
GNU GCC 10.2.0 11.2.0
GNU M4 1.4.18 1.4.19
GNU Readline 8.0 8.1.1
GNU Tar 1.32 1.34
GNU Texinfo 6.7 6.8
GNU diffutils 3.7 3.8
GNU findutils 4.7.0 4.8.0
GNU gmp 6.2.0 6.2.1
GNU grep 3.4 3.7
GNU gzip 1.10 1.11
GNU libunistring 0.9.10 1.0
GNU mpc 1.1.0 1.2.1
GNU mpfr 4.0.2 4.1.0
GNU nano 5.2 6.0
GNU ncurses 6.2 6.3
GNU wget 1.20.3 1.21.2
Git 2.28.0 2.34.0
Less 563 590
Libxml2 2.9.9 2.9.12
Lzip 1.22-rc2 1.22
OpenSLL 1.1.1a 3.0.0
Patchelf 0.10 0.13
Perl 5.32.0 5.34.0
Podlators -- 4.14
Name (Highlevel) Old version New version
---------------- ----------- -----------
Apachelog4cxx 0.10.0-603 0.12.1
Astrometry.net 0.80 0.85
Boost 1.73.0 1.77.0
CFITSIO 3.48 4.0.0
Cmake 3.18.1 3.21.4
Eigen 3.3.7 3.4.0
Expat 2.2.9 2.4.1
FFTW 3.3.8 3.3.10
Flex 2.6.4-91 2.6.4
Fontconfig 2.13.1 2.13.94
Freetype 2.10.2 2.11.0
GNU Astronomy Utilities 0.12 0.16.1-e0f1
GNU Autoconf 2.69.200-babc 2.71
GNU Automake 1.16.2 1.16.5
GNU Bison 3.7 3.8.2
GNU Emacs 27.1 27.2
GNU GDB 9.2 11.1
GNU GSL 2.6 2.7
GNU Help2man 1.47.11 1.48.5
Ghostscript 9.52 9.55.0
ICU -- 70.1
ImageMagick 7.0.8-67 7.1.0-13
Libbsd 0.10.0 0.11.3
Libffi 3.2.1 3.4.2
Libgit2 1.0.1 1.3.0
Libidn 1.36 1.38
Libjpeg 9b 9d
Libmd -- 1.0.4
Libtiff 4.0.10 4.3.0
Libx11 1.6.9 1.7.2
Libxt 1.2.0 1.2.1
Netpbm 10.86.99 10.73.38
OpenBLAS 0.3.10 0.3.18
OpenMPI 4.0.4 4.1.1
Pixman 0.38.0 0.40.0
Python 3.8.5 3.10.0
R 4.0.2 4.1.2
SWIG 3.0.12 4.0.2
Util-linux 2.35 2.37.2
Util-macros 1.19.2 1.19.3
Valgrind 3.15.0 3.18.1
WCSLIB 7.3 7.7
Xcb-proto 1.14 1.14.1
Xorgproto 2020.1 2021.5
Name (Python) Old version New version
------------- ----------- -----------
Astropy 4.0 5.0
Beautifulsoup4 4.7.1 4.10.0
Beniget -- 0.4.1
Cffi 1.12.2 1.15.0
Cryptography 2.6.1 36.0.1
Cycler 0.10.0 0.11.0+}
Cython 0.29.21 0.29.24
Esutil 0.6.4 0.6.9
Extension-helpers -- 0.1
Galsim 2.2.1 2.3.3
Gast -- 0.5.3
Jinja2 -- 3.0.3
MPI4py 3.0.3 3.1.3
Markupsafe -- 2.0.1
Numpy 1.19.1 1.21.3
Packaging -- 21.3
Pillow -- 8.4.0
Ply -- 3.11
Pyerfa -- 2.0.0.1
Pyparsing 2.3.1 3.0.4
Pythran -- 0.11.0
Scipy 1.5.2 1.7.3
Setuptools 41.6.0 58.3.0
Six 1.12.0 1.16.0
Uncertainties 3.1.2 3.1.6
Wheel -- 0.37.0
Name (R) Old version New version
-------- ----------- -----------
Cli -- 2.5.0
Colorspace -- 2.0-1
Cowplot -- 1.1.1
Crayon -- 1.4.1
Digest -- 0.6.27
Ellipsis -- 0.3.2
Fansi -- 0.5.0
Farver -- 2.1.0
Ggplot2 -- 3.3.4
Glue -- 1.4.2
GridExtra -- 2.3
Gtable -- 0.3.0
Isoband -- 0.2.4
Labeling -- 0.4.2
Lifecycle -- 1.0.0
Magrittr -- 2.0.1
MASS -- 7.3-54
Mgcv -- 1.8-36
Munsell -- 0.5.0
Pillar -- 1.6.1
R-Pkgconfig -- 2.0.3
R6 -- 2.5.0
RColorBrewer -- 1.1-2
Rlang -- 0.4.11
Scales -- 1.1.1
Tibble -- 3.1.2
Utf8 -- 1.2.1
Vctrs -- 0.3.8
ViridisLite -- 0.4.0
Withr -- 2.4.2
|
|
Until now, while the series of steps mentioned in 'README.md' were
complete, they had some implicit thing in them that made it a little hard
to run as a checklist (the commands to do some basic things weren't
included). Also, it was recommending to run a long 'docker run ...'
command, which wasn't too user friendly.
With this commit, the series of steps is now a complete checklist,
containing every step. Also, the checklist now recommends putting the long
'docker run' command inside a script called 'docker-run' that will also do
a 'sudo' internally (thus making things very easy for a first-time user).
Also, since the 'docker-run' script contains host OS-specific directory
names, it should not be under control, so it has been added to the
'.gitignore' file in case users decide to keep this same name (which is
recommended).
|
|
Until now, the build directory contained a 'software/' directory (that
hosted all the built software), a 'tex/' subdirectory for the final
building of the paper, and many other directories containing
intermediate/final data of the specific project. But this mixing of built
software and data is against our modularity and minimal complexity
principles: built software and built data are separate things and keeping
them separate will enable many optimizations.
With this commit, the build directory of the core Maneage branch will only
contain two sub-directories: 'software/' and 'analysis/'. The 'software/'
directory has the same contents as before and is not touched in this
commit. However, the 'analysis/' directory is new and everything created in
the './project make' phase of the project will be created inside of this
directory.
To facilitate easy access to these top-level built directories, two new
variables are defined at the top of 'initialize.mk': 'badir', which is
short for "built-analysis directory" and 'bsdir', which is short for
"built-software directory".
HOW TO IMPLEMENT THIS CHANGE IN YOUR PROJECT. It is easy: simply replace
all occurances of '$(BDIR)' in your project's subMakefiles (except the ones
below) to '$(badir)'. To confirm if everything is fine before building your
project from scratch after merging, you can run the following command to
see where 'BDIR' is used and confirm the only remaning cases.
$ grep -r BDIR reproduce/analysis/*
--> make/verify.mk: innobdir=$$(echo $$infile | sed -e's|$(BDIR)/||g'); \
--> make/initialize.mk:badir=$(BDIR)/analysis
--> make/initialize.mk:bsdir=$(BDIR)/software
--> make/initialize.mk: $$sys_rm -rf $(BDIR)
--> make/top-prepare.mk:all: $(BDIR)/software/preparation-done.mk
'BDIR' should only be present in lines of the files above. If you see
'$(BDIR)' used anywhere else, simply change it to '$(badir)'. Ofcourse, if
your project assumes BDIR in other contexts, feel free to keep it, it will
not conflict. If anything un-expected happens, please post a comment on the
link below (you need to be registered on Savannah to post a comment):
https://savannah.nongnu.org/task/?15855
One consequence of this change is that the 'analysis/' subdirectory can be
optionally mounted on a separate partition. The need for this actually came
up for some new users of Maneage in a Docker image. Docker can fix
portability problems on systems that we haven't yet supported (even
Windows!), or had a chance to fix low-level issues on. However, Docker
doesn't have a GUI interface. So to see the built PDF or intermediate data,
it was necessary to copy the built data to the host system after every
change, which is annoying during working on a project. It would also need
two copies of the source: one in the host, one in the container. All these
frustrations can be fixed with this new feature.
To describe this scenario, README.md now has a new section titled "Only
software environment in the Docker image". It explains step-by-step how you
can make a Docker image to only host the built software environment. While
your project's source, software tarballs and 'BDIR/analysis' directories
are on your host operating system. It has been tested before this commit
and works very nicely.
|
|
Until now, the description in 'README.md' to build the Dockerfile in
'README.md' had one item per line, thoroughly describing the reason behind
that line. But in many cases, the user is already familiar with Docker (or
has already read through the items) and just wants to have the Dockerfile
ready fast. In these cases, all those extra explanations are annoying.
With this commit, an item '0' has been added at the start of the item list
for summary. It only contains the necessary Dockerfile contents with no
extra explanation.
|
|
Having entered 2021, it was necessary to update the copyright years at the
top of the source files. We recommend that you do this for all your
project-specific source files also.
|
|
Until now we had described the basic commands on how to create and use
Docker images, but we hadn't mentioned how you can delete them.
With this commit the commands necessary for deleting Docker images have
also been added at the bottom of the section on Docker.
|
|
With the previous commit, we now build Nano by default within Maneage, and
project authors can ask to install Emacs and Vim within 'TARGETS.conf'. So
in the instructions to build within a Docker image have been removed.
|
|
When building Maneage inside a Docker container, in the end the users want
to extract the final outputs from the container into their host operating
system to inspect more comfortably. So with this commit, a short
examplanation has been added on how to do this.
We also noticed that it is much better if the 'Dockerfile' is stored and
run in an empty directory, otherwise, it will start parsing the full
directory and its subdirectories as the docker image's environment.
|
|
Docker is a "container" technology that allows an almost independent
operating system run on the host. It is useful when the host OS doesn't
support some features or has internal problems (for example its C library
or C compiler have problems). Fortunately a Maneaged project can easily be
built within a Docker image and a minimal image operating system.
With this commit, a section has been added to 'README.md' to describe this
process. Each step of the Dockerfile is explined, to help users that may
not be too familiar with Docker, or help Docker user who are not familiar
with Maneage.
|
|
Possible semantic conflicts (that may not show up as Git conflicts but may
cause a crash in your project after the merge):
1) The project title (and other basic metadata) should be set in
'reproduce/analysis/conf/metadata.conf'. Please include this file in
your merge (if it is ignored because of '.gitattributes'!).
2) Consider importing the changes in 'initialize.mk' and 'verify.mk' (if
you have added all analysis Makefiles to the '.gitattributes' file
(thus not merging any change in them with your branch). For example
with this command:
git diff master...maneage -- reproduce/analysis/make/initialize.mk
3) The old 'verify-txt-no-comments-leading-space' function has been
replaced by 'verify-txt-no-comments-no-space'. The new function will
also remove all white-space characters between the columns (not just
white space characters at the start of the line). Thus the resulting
check won't involve spacing between columns.
A common set of steps are always necessary to prepare a project for
publication. Until now, we would simply look at previous submissions and
try to follow them, but that was prone to errors and could cause
confusion. The internal infrastructure also didn't have some useful
features to make good publication possible. Now that the submission of a
paper fully devoted to the founding criteria of Maneage is complete
(arXiv:2006.03018), it was time to formalize the necessary steps for easier
submission of a project using Maneage and implement some low-level features
that can make things easier.
With this commit a first draft of the publication checklist has been added
to 'README-hacking.md', it was tested in the submission of arXiv:2006.03018
and zenodo.3872248. To help guide users on implementing the good practices
for output datasets, the outputs of the default project shown in the paper
now use the new features). After reading the checklist, please inspect
these.
Some other relevant changes in this commit:
- The publication involves a copy of the necessary software
tarballs. Hence a new target ('dist-software') was also added to
package all the project's software tarballs in one tarball for easy
distribution.
- A new 'dist-lzip' target has been defined for those who want to
distribute an Lzip-compressed tarball.
- The '\includetikz' LaTeX macro now has a second argument to allow
configuring the '\includegraphics' call when the plot should not be
built, but just imported.
|
|
In time, some of the copyright license description had been mistakenly
shortened to two paragraphs instead of the original three that is
recommended in the GPL. With this commit, they are corrected to be exactly
in the same three paragraph format suggested by GPL.
The following files also didn't have a copyright notice, so one was added
for them:
reproduce/software/make/README.md
reproduce/software/bibtex/healpix.tex
reproduce/analysis/config/delete-me-num.conf
reproduce/analysis/config/verify-outputs.conf
|
|
Until now, the primary Maneage URLs were under GitLab, but since we now
have a dedicated URL and Git repository, its better to transfer to this as
soon as possible. Therefore with this commit, throughout Maneage, any place
that Maneage was referenced through GitLab has been corrected.
Please correct your project's remote to point to the new repository at
`git.maneage.org/project.git', and please make sure it follows the
`maneage' branch. There is no more `master' branch on Maneage.
|
|
Until now, throughout Maneage we were using the old name of "Reproducible
Paper Template". But we have finally decided to use Maneage, so to avoid
confusion, the name has been corrected in `README-hacking.md' and also in
the copyright notices.
Note also that in `README-hacking.md', the main Maneage branch is now
called `maneage', and the main Git remote has been changed to
`https://gitlab.com/maneage/project' (this is a new GitLab Group that I
have setup for all Maneage-related projects). In this repository there is
only one `maneage' branch to avoid complications with the `master' branch
of the projects using Maneage later.
|
|
Until now, the main commands to run the project were these: `./project
configure' (to build the software), `./project prepare' (to possibly
arrange input datasets and build special configuration Makefiles) and
finally `./project make' to run the project.
The main logic behind the "prepare" phase `top-prepare.mk' is to build
configuration files that can be fed into the "make" step and optimize its
operation. For example when the total number of necessary inputs for the
majority of the analysis is not as large as the total number of
inputs. With "prepare" (when necessary), you go through the raw inputs,
select the ones that are necessary for the rest of the project. The output
of `top-prepare.mk' is a configuration file (a Make variable) that keeps
the IDs (numbers, names, etc). That configuration file would then be used
in the `top-make.mk' to identify the lower level targets and allow optimal
project organization and management.
But the last two are both part of the analysis, and while they indeed need
different calls to Make to be executed, many projects don't actually need a
preparation phase: ultimately, its an implementation choice by the project
developers and doesn't concern the project users (or the developers when
they are running it).
To avoid confusing the users, or simply annoying them when a projet doesn't
need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk'
Makefiles are called with the single `./project make' command and
`./project prepare' has been dropped. I noticed this while writing the
paper on this system.
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
In many real-world scenarios, `./project make' can really benefit from
having some basic information about the data before being run. For example
when quering a server. If we know how many datasets were downloaded and
their general properties, it can greatly optmize the process when we are
designing the solution to be run in `./project make'.
Therefore with this commit, a new phase has been added to the template's
design: `./project prepare'. In the raw template this is empty, because the
simple analysis done in the template doesn't warrant it. But everything is
ready for projects using the template to add preparation phases prior to
the analysis.
|
|
Until now, when the project's source was downloaded from something like
arXiv, in `README.md', we were instructing them to set the executable flags
of all the files that need it. But except for `./project', the reader
shouldn't have to worry about the project internals! Once its executable,
`./project' can easily fix the executable flags of all the files that need
it automatically.
With this commit, in `README.md', we just instruct the reader to set the
executable flag of `./project' and any other file that needs an executable
flag is given one at the start of the set of commands for `./project
configure'. In customized projects, if an author needs executable flags on
any other files, they can easily add it there without involving the user.
|
|
Konrad Hinsen pointed out that this part was missing from the instructions
in `README.md' after cloning. So it is added.
|
|
The two modifications to the LaTeX source of an arXiv-downloaded source
weren't rendered properly on Gitlab, so they are corrected to be in the
same line and not have a separate code-block.
|
|
Until now, we were assuming that the users would just clone the project in
Git. But after submitting arXiv:1909.11230, and trying to build directly
from the arXiv source, I noticed several problems that wouldn't allow users
to build it automatically. So I tried the build step by step and was able
to find a fix for the several issues that came up.
The scripting parts of the fix were primarily related to the fact that the
unpacked arXiv tarball isn't under version control, so some checks had to
be put there. Also, we wanted to make it easy to remove the extra files, so
an extra `--clean-texdit' option was added to `./project'.
Finally, some manual corrections were necessary (prior to running
`./project', which are now described in `README.md'. Most of the later
steps can be automated and we should do it later, I just don't have enough
time now.
|
|
Until now customizing it was a little more detailed, for example the
copyright statement wasn't generic and was about "this template". So the
user would have to correct it.
With this commit, the copyright statment just says "this project", so it
can apply to the raw template and also any customization of it. Also, some
minor edits were made in the various parts of the text to make it more
clear.
|
|
The Copyright year is now on a separate line (by adding a backslash), and
the `file-metadata' is now enclosed in two "`" characters to show
differently after rendering.
|
|
Until now, to work on a project, it was necessary to `./configure' it and
build the software. Then we had to run `.local/bin/make' to run the project
and do the analysis every time. If the project was a shared project between
many users on a large server, it was necessary to call the `./for-group'
script.
This way of managing the project had a major problem: since the user
directly called the lower-level `./configure' or `.local/bin/make' it was
not possible to provide high-level control (for example limiting the
environment variables). This was especially noticed recently with a bug
that was related to environment variables (bug #56682).
With this commit, this problem is solved using a single script called
`project' in the top directory. To configure and build the project, users
can now run these commands:
$ ./project configure
$ ./project make
To work on the project with other users in a group these commands can be
used:
$ ./project configure --group=GROUPNAME
$ ./project make --group=GROUPNAME
The old options to both configure and make the project are still valid. Run
`./project --help' to see a list. For example:
$ ./project configure -e --host-cc
$ ./project make -j8
The old `configure' script has been moved to
`reproduce/software/bash/configure.sh' and is called by the new `./project'
script. The `./project' script now just manages the options, then passes
control to the `configure.sh' script. For the "make" step, it also reads
the options, then calls Make. So in the lower-level nothing has
changed. Only the `./project' script is now the single/direct user
interface of the project.
On a parallel note: as part of bug #56682, we also found out that on some
macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set
to blank. This is no problem because RPATH is automatically set in macOS
and the executables and libraries contain the absolute address of the
libraries they should link with. But having `DYLD_LIBRARY_PATH' can
conflict with some low-level system libraries and cause very hard to debug
linking errors (like that reported in the bug report).
This fixes bug #56682.
|
|
All occurances of "pipeline" have been chanaged to "project" or "template"
withint the text (comments, READMEs, and comments) of the template. The
main template branch is now also named `template'.
This was all because `pipeline' is too generic and couldn't be
distinguished from the base, and customized project.
|
|
Since `.file-metadata' is a binary file, we can't include a copyright
inside of it so we have to use `README.md' to mention its copyright and
license notice. However, this was not done clearly and is now corrected.
|
|
Until now, the files where the people were meant to change didn't have a
proper copyright notice (for example `Copyright (C) YOUR NAME.'). This was
wrong because the license does not convey copyright ownership. So the name
of the file's original author must always be included and when people
modify it (and add their own copyright-able modifications).
With this commit, the file's original author (and email) are added to the
copyright notice and when more than one person modified a file, both names
have their individual copyright notice.
Based on this, the description for adding a copyright notice in
`README-hacking.md' has also been modified.
|
|
Since `.file-metadata' is a binary file and we couldn't put a copyright
notice within it, it has been mentioned in `README.md' to have the same
copyright.
Also, the copyright modification step in `README-hacking.md' was brought to
a later step to be more clear that it should always be done (on new files
or files that are changed).
|
|
Until now, for short files, we only had a license notice, not an actual
copyright notice. With this commit, a copyright notice has also been
added. We use this new command to find these files, suggested by
`ineiev@gnu.org'.
|
|
In order to be more clear, a copyright statement was added to all the LaTeX
and README files.
|
|
To be more generic and recognizable, the `README-pipeline.md' script was
renamed to `README-hacking.md'. In essence, it is just that: to hack the
existing pipeline for your own project. We follow a similar naming
convention in many GNU software.
|
|
Until now, there was no reference to `README-pipeline.md' within the
`README.md' file. Since `README.md' is the first file that someone reads
and the basic perpose and structure of the pipeline is described in
`README-pipeline.md', it was necessary to bring it up there.
|
|
To help and be more clear a link to this pipeline's dependency repository
has been added to `README.md'.
|
|
The README.md file was updated to reflect recent changes in the pipeline
(especially regarding the downloader).
|
|
A spellcheck was run on the two README files.
|
|
The note to the pipeline designers was corrected to display properly on
Gitlab.
|
|
A placeholder link is now used in `README.md' to encourage the pipeline
designers to keep a backup of all the dependencies they use.
|
|
Until now, were were advising the users to rename the two README files
after cloning the project. This was because online Git browsers usually
display the `README.md' file, so we wanted the description of the pipeline
to be visible in the pipeline, and later when a project adopts it, they can
have their own `README.md'. But the problem is that any change in
`REAME.md' will later cause conflicts with a project's `README.md'. So we
are now using the same naming convention as the papers that use the
pipeline.
|
|
In the checklist, we are now defining the remote host of the repository at
an early stage. This is because we will need it in the `README.md' file
(which now has a placeholder `XXXXXXX' instead of a valid URL).
|
|
Until now, in the instructions, we were suggesting to run
`./.local/bin/make', but the `./' part is extra: this is already a
directory and so the shell will be able to find it. So to make things more
clear and easy to read/write, we removed the `./' part from the calls to
our custom Make installation.
|
|
When you point to this project, the `README.md' file is the default file
that opens on GitLab and other online git repositories. Since a
reproduction pipeline project is different from the actual pipeline, its
best for the default text that opens to describe the paper, not the
pipeline. The old `README.md' is also kept, but its now called
`REAME-pipeline.md'.
|
|
In the previous commit, we were recommending to fetch the work from this
pipeline. But since we have a separate `pipeline' branch, we can simply
checkout to that branch and pull all the recent changes. So with this
commit, the steps to get recent updates to the pipeline are updated.
|
|
Since working on the pipeline will evolve along with the projects that use
it, it can be useful for projects to fetch updates in the pipeline. So the
checklist in `README.md' updated to explain how to do this cleanly.
|
|
Until now, because we didn't build the dependencies internally, it was
important for the pipeline to be usable with any version of Make. But
because of the new installation of dependencies (including GNU Make), that
is no longer the case. So we can safely use GNU Make and this needs to be
mentioned in `README.md'.
|
|
GNU Coreutils are basic programs that can help in the configuration of
higher-level programs. Because of that, it was a dependency of almost all
software built in `dependencies.mk'. To make things more clear, easier to
read and faster (when building in parallel), the building of Coreutils is
now moved to the `dependencies-basic.mk' rules. There, it is built
along-side Bash. Since `dependenceis-basic.mk' is run and completed before
`dependencies.mk', with this, we can be sure that Coreutils is present by
the time we want to build the higher-level programs.
Also, Zlib is now added as a dependency of Git also (it is necessary for
its build).
|
|
After going through the checklist for starting a new project based on the
pipeline, I noticed some parts that could be modified to be more
clear. They are now applied.
|
|
Until now, we were using a customized `tar.lz' tarball for Gzip. But on
systems that don't have GNU Tar, this will cause a problem (non-GNU Tar
doesn't recognize `.tar.lz'). So to keep things simple, we are using the
customized gzip in `tar.gz' format. After the internal build of GNU Tar and
Lzip, the default method of unpacking (`tar xf XXXXX.tar.XX') will work
nicely on all the standard compression algorithms and we don't have to
modify our commands based on the algorithm (nice feature of GNU Tar).
|
|
The two README files have been updated to explain the new feature of
downloading and building dependencies.
|
|
All the used software are now acknowledged in the template paper along with
their versions. This section is also mentioned in the check list, so users
don't delete it by mistake.
|
