diff options
29 files changed, 672 insertions, 228 deletions
diff --git a/.dir-locals.el b/.dir-locals.el index ae7e990..3ffb548 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -8,6 +8,11 @@ ;; or the number of spaces to be used for indentation. ;; ;; For more information see (info "(emacs) Directory Variables") +;; +;; Copying and distribution of this file, with or without modification, +;; are permitted in any medium without royalty provided the copyright +;; notice and this notice are preserved. This file is offered as-is, +;; without any warranty. ( ;; C files. diff --git a/.file-metadata b/.file-metadata Binary files differindex 27bfed5..eb42414 100644 --- a/.file-metadata +++ b/.file-metadata @@ -1,3 +1,19 @@ +# Files to ignore in version control (Git) -*- conf-unix -*- +# ========================================================== +# +# The files satisfying the patterns here (one per line) are ignored by +# the version control using Git, they are all outputs of the various +# tools used to built Gnuastro, so we don't need to keep their +# history. +# +# To help readability, please add new files based on the length of the +# line you wish to add. +# +# Copying and distribution of this file, with or without modification, +# are permitted in any medium without royalty provided the copyright +# notice and this notice are preserved. This file is offered as-is, +# without any warranty. + *~ *.txt *.aux @@ -14,6 +30,7 @@ mmap_* .tex build .local +.build Makefile tex/tikz .gnuastro @@ -22,7 +39,6 @@ tex/tikz tex/pipeline LOCAL_tmp.mk LOCAL_old.mk -reproduce/build gnuastro-local.conf reproduce/config/pipeline/LOCAL.mk diff --git a/README-hacking.md b/README-hacking.md index e7a3f44..ed9f153 100644 --- a/README-hacking.md +++ b/README-hacking.md @@ -1,3 +1,6 @@ +Copyright (C) 2018-2019 Mohammad Akhlaghi. +See the end of the file for license conditions. + Reproducible paper template =========================== @@ -251,29 +254,38 @@ will use). Therefore, the pipeline builds its own dependencies during the `reproduce/src/make/dependencies-basic.mk` and `reproduce/src/make/dependencies.mk`. These Makefiles are called by the `./configure` script and not used afterwards. The first is intended for -downloading and building the most basic tools like GNU Bash, GNU Make, and -GNU Tar. Therefore it must only contain very basic and portable Make and -shell features. The second is called after the first, thus enabling usage -of the modern and advanced features of GNU Bash and GNU Make, similar to -the rest of the pipeline. Later, if you add a new program/library for your -research, you will need to include a rule on how to download and build it -(in `reproduce/src/make/dependencies.mk`). - -After it finishes, `./configure` will create a `Makefile` in the top -directory (a symbolic link to `reproduce/src/make/top.mk`) and a `.local` -directory (a link for easy access to the custom built software -packages). The `.local/bin/make` command will then use our custom version -of GNU Make to do the analysis. The first file that is read by Make is the -top-level `Makefile`. Therefore, we'll start our navigation/discussion with -this file. This file is relatively short and heavily commented so hopefully -the descriptions in each comment will be enough to understand the general -details. As you read this section, please also look at the contents of the -mentioned files and directories to fully understand what is going on. +downloading and building the most basic tools like GNU Tar, GNU Bash, GNU +Make, and GNU Compiler Collection (GCC). Therefore it must only contain +very basic and portable Make and shell features. The second is called after +the first, thus enabling usage of the modern and advanced features of GNU +Bash, GNU Make and other low-level GNU tools, similar to the rest of the +pipeline. Later, if you add a new program/library for your research, you +will need to include a rule on how to download and build it (in +`reproduce/src/make/dependencies.mk`). + +After it finishes, `./configure` will create the following symbolic links +in the project's top source directory: 1) `Makefile` in the top directory +which points to `reproduce/src/make/top.mk`. 2) `.build' which points to +the top build directory, and 3) `.local` for easy access to the custom +built software packages installation directory. The first is for practical +necessity (so you can run `make' from the top source directory), but the +latter are just for convenience (fast access to the built products and +software). + +Therefore, by running `.local/bin/make` we will build the project with the +project's custom version of GNU Make, not the host system's Make. The first +file that is read by Make (the template's starting point) is the top-level +`Makefile` (also created by `./configure`). Therefore, we'll start +describing the template's architecture with this file. This file is +relatively short and heavily commented so hopefully the descriptions in +each comment will be enough to understand the general details. As you read +this section, please also look at the contents of the mentioned files and +directories to fully understand what is going on. Before starting to look into the top `Makefile`, it is important to recall that Make defines dependencies by files. Therefore, the input/prerequisite and output of every step/rule must be a file. Also recall that Make will -use the modification date of the prerequisite and target files to see if +use the modification date of the prerequisite(s) and target files to see if the target must be re-built or not. Therefore during the processing, _many_ intermediate files will be created (see the tips section below on a good strategy to deal with large/huge files). @@ -284,29 +296,31 @@ intermediate files (it was defined in `./configure`). This directory doesn't need to be version controlled or even synchronized, or backed-up in other servers: its contents are all products of the pipeline, and can be easily re-created any time. As you define targets for your new rules, it is -thus important to place them all under sub-directories of `$(BDIR)`. +thus important to place them all under sub-directories of `$(BDIR)`. As +mentioned above, you always have fast access to this "build"-directory with +the `.build` symbolic link. In this architecture, we have two types of Makefiles that are loaded into the top `Makefile`: _configuration-Makefiles_ (only independent variables/configurations) and _workhorse-Makefiles_ (Makefiles that -actually contain rules). +actually contain analysis/processing rules). The configuration-Makefiles are those that satisfy this wildcard: `reproduce/config/pipeline/*.mk`. These Makefiles don't actually have any rules, they just have values for various free parameters throughout the -analysis/processing. Open a few of them to see for your self. These +analysis/processing. Open a few of them to see for yourself. These Makefiles must only contain raw Make variables (pipeline -configurations). By raw we mean that the Make variables in these files must -not depend on variables in any other configuration-Makefile. This is +configurations). By "raw" we mean that the Make variables in these files +must not depend on variables in any other configuration-Makefile. This is because we don't want to assume any order in reading them. It is also very -important to *not* define any rule, or other Make construct in any of these +important to *not* define any rule, or other Make construct, in these configuration-Makefiles. -These conditions will enable you to set these configure-Makefiles as a -prerequisite to any target that depends on their variable -values. Therefore, if you change any of their values, all targets that -depend on those values will be re-built. This is very convenient as your -project scales up and gets more complex. +This enables you to set these configure-Makefiles as a prerequisite to any +target that depends on their variable values. Therefore, if you change any +of their values, all targets that depend on those values will be +re-built. This is very convenient as your project scales up and gets more +complex. The workhorse-Makefiles are those satisfying this wildcard `reproduce/src/make/*.mk`. They contain the details of the processing steps @@ -317,17 +331,16 @@ other rules that will be defined prior to them (not a fixed name like higher-level ones. All processing steps are assumed to ultimately (usually after many rules) -end up in some number, image, figure, or table that are to be included in -the paper. The writing of these results into the final report/paper is -managed through separate LaTeX files that only contain macros (a name given -to a number/string to be used in the LaTeX source, which will be replaced -when compiling it to the final PDF). So the last target in a -workhorse-Makefile is a `.tex` file (with the same base-name as the -Makefile, but in `$(BDIR)/tex/macros`). As a result, if the targets in a -workhorse-Makefile aren't directly a prerequisite of other -workhorse-Makefile targets, they can be a pre-requisite of that -intermediate LaTeX macro file and thus be called when necessary. Otherwise, -they will be ignored by Make. +end up in some number, image, figure, or table that will be included in the +paper. The writing of these results into the final report/paper is managed +through separate LaTeX files that only contain macros (a name given to a +number/string to be used in the LaTeX source, which will be replaced when +compiling it to the final PDF). So the last target in a workhorse-Makefile +is a `.tex` file (with the same base-name as the Makefile, but in +`$(BDIR)/tex/macros`). As a result, if the targets in a workhorse-Makefile +aren't directly a prerequisite of other workhorse-Makefile targets, they +can be a pre-requisite of that intermediate LaTeX macro file and thus be +called when necessary. Otherwise, they will be ignored by Make. This pipeline also has a mode to share the build directory between several users of a Unix group (when working on large computer clusters). In this @@ -336,16 +349,15 @@ the large built files between each other. To do this, it is necessary for all built files to give full permission to group members while not allowing any other users access to the contents. Therefore the `./configure` and Make steps must be called with special conditions which are managed in the -`for-group` file. +`for-group` script. -Let's see how this design is implemented. When the `./configure` finishes, -it a `Makefile` will be placed in the top directory. This `Makefile` is -just a symbolic link to `reproduce/src/make/top.mk`. Please open and -inspect it as we go along here. The first step (un-commented line) is to -import the local configuration (answers to the questions `./configure` -asked you). They are defined in the configuration-Makefile -`reproduce/config/pipeline/LOCAL.mk` which was also built by `./configure` -(based on the `LOCAL.mk.in` template). +Let's see how this design is implemented. When `./configure` finishes: By +creating a `Makefile` in the top directory, it allows us to start "making" +the project. Please open and inspect it as we go along here. The first step +(un-commented line) is to import the local configuration (answers to the +questions `./configure` asked you). They are defined in the +configuration-Makefile `reproduce/config/pipeline/LOCAL.mk` which was also +built by `./configure` (based on the `LOCAL.mk.in` template). The next non-commented set of lines define the ultimate target of the whole pipeline (`paper.pdf`). But to avoid mistakes, a sanity check is necessary @@ -355,14 +367,8 @@ the `./for-group` script, but Make isn't). Therefore we use a Make conditional to define the `all` target based on the group permissions being consistent between the initial configuration and the current run. -If there is a problem `all` will not depend on anything and will just print -a warning to inform you of the problem. When the group conditions are fine, -`all` will depend on `paper.pdf` (which is defined in -`reproduce/src/make/paper.mk` and will be imported into this top Makefile -later). - Having defined the top target, our next step is to include all the other -necessary Makefiles. But order matters in the importing of +necessary Makefiles. However, order matters in the importing of workhorse-Makefiles and each must also have a TeX macro file with the same base name (without a suffix). Therefore, the next step in the top-level Makefile is to define a `makesrc` variable to keep the base names (without @@ -386,15 +392,16 @@ your rules into as many logically-similar but independent steps as possible. The `reproduce/src/make/paper.mk` Makefile must be the final Makefile that -is included. It ends with the rule to build `paper.pdf` (final target of -the whole reproduction pipeline). If look in it, you will notice that it -starts with a rule to create `$(mtexdir)/pipeline.tex` (`mtexdir` is just a -shorthand name for `$(BDIR)/tex/macros` mentioned before). -`$(mtexdir)/pipeline.tex` is the connection between the processing/analysis -steps of the pipeline, and the steps to build the final PDF. As you see, -`$(mtexdir)/pipeline.tex` only instruct LaTeX to import the LaTeX macros of -each high-level processing step during the analysis (the separate -work-horse Makefiles that you defined and included). +is included. This workhorse Makefile ends with the rule to build +`paper.pdf` (final target of the whole reproduction pipeline). If you look +in it, you will notice that it starts with a rule to create +`$(mtexdir)/pipeline.tex` (`mtexdir` is just a shorthand name for +`$(BDIR)/tex/macros` mentioned before). `$(mtexdir)/pipeline.tex` is the +connection between the processing/analysis steps of the pipeline, and the +steps to build the final PDF. As you see, `$(mtexdir)/pipeline.tex` only +instructs LaTeX to import the LaTeX macros of each high-level processing +step during the analysis (the separate work-horse Makefiles that you +defined and included). During the research, it often happens that you want to test a step that is not a prerequisite of any higher-level operation. In such cases, you can @@ -405,16 +412,6 @@ your research, set it as prerequisites to other rules and remove it from the list of prerequisites for TeX macro file. In fact, this is how a project is designed to grow in this framework. -When working within a group, more than one person may want to work with the -pipeline outputs (in the build directory). For example each person is -developing part of the higher-level steps of the pipeline in their own Git -branch of the pipeline, but using the same build directory. Therefore, the -lower-level parts of the built outputs, can be shared between them. In such -scenarios, this pipeline comes with a `for-group` script (in the top -directory) which is just a simple wrapper to run the configure and building -steps. You can specify a group name within this file. Therefore, when you -use it (fully described in the comments at the start of the file), it will -ensure that all group members have write access to the created files. @@ -1011,3 +1008,25 @@ that is not reproducible (due to incomplete information by the author) is not scientific: the readers have to have faith in the subjective experience of the authors in the very important choice of configuration values and order of operations: this is contrary to the scientific spirit. + + + + + +Copyright information +--------------------- +This file is part of the reproducible paper template + https://gitlab.com/makhlaghi/reproducible-paper + +This template is free software: you can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) +any later version. + +This template is distributed in the hope that it will be useful, but +WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY +or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along +with Template. If not, see <https://www.gnu.org/licenses/>.
\ No newline at end of file @@ -1,3 +1,6 @@ +Copyright (C) 2018-2019 Mohammad Akhlaghi. +See the end of the file for license conditions. + Reproduction pipeline for paper XXXXXXX ======================================= @@ -78,3 +81,25 @@ This pipeline was designed to have as few dependencies as possible. ```shell $ .local/bin/make -j8 ``` + + + + + +Copyright information +--------------------- +This file is part of the reproducible paper template + https://gitlab.com/makhlaghi/reproducible-paper + +This template is free software: you can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) +any later version. + +This template is distributed in the hope that it will be useful, but +WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY +or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along +with Template. If not, see <https://www.gnu.org/licenses/>.
\ No newline at end of file @@ -22,9 +22,6 @@ # <http://www.gnu.org/licenses/>. - - - # Script settings # --------------- # Stop the script if there are any errors. @@ -34,13 +31,138 @@ set -e -# Important internal locations -# ---------------------------- +# Default option values +jobs=0 +build_dir= +input_dir= +software_dir= +existing_conf=0 +minmapsize=10000000000 + + + + + +# Output of --help +# ---------------- +me=$0 # Executable file name. +print_help() { + # Print the output. + cat <<EOF +Usage: $me [OPTION]... + +Configure the reproducible paper template for this system (set local +settings for this system). The local settings can be given on the +command-line through the options below. If not, the configure script will +interactively ask for a value to each one (with basic necessary background +information printed before them). Alternatively, if you have already +configured this script for your system, you can use the '--existing-conf' +to use its values directly. + +RECOMMENDATION: If this is the first time you are running this pipeline, +please don't use the options and let the script explain each parameter in +full detail by simply running './configure'. + +The only mandatory value for this script is the local build directory. This +is where all the pipeline's outputs will be stored. Optionally, you can +also provide directories that host input data, or software source codes. If +the necessary files don't exist there, the template will automatically +download them. + +With the options below you can modify the default behavior. Just note that +you should not put an '=' sign between an option name and its value. + +Configure options: + Top-level directory settings: + -b, --build-dir=STR Top directory to build the project in. + -i, --input-dir=STR Directory containing input datasets (optional). + -s, --software-dir=STR Directory containing necessary software tarballs. + + Operating mode options: + -m, --minmapsize=INT (Gnuastro) Minimum number of bytes to use RAM. + -j, --jobs=INT Number of threads to build the software. + -e, --existing-conf Use (possibly existing) local configuration. + -h, --help Print this help list. + +Mandatory or optional arguments to long options are also mandatory or optional +for any corresponding short options. + +Reproducible paper template: https://gitlab.com/makhlaghi/reproducible-paper + +Report bugs to mohammad@akhlaghi.org +EOF +} + + + + + +# Functions to check option values and complain if necessary. +function on_off_option_error() { + cat <<EOF +$scriptname: '$1' doesn't take any values. +EOF + exit 1 +} + +function check_v() { + if [ x"$2" = x ]; then + echo "$scriptname: option '$1' requires an argument." + echo "Try '$scriptname --help' for more information." + exit 1; + fi +} + + + + + +# Separate command-line arguments from options. Then put the option +# value into the respective variable. +# +# Each option has two lines because we want to process both these formats: +# `--name=value' and `--name value'. The former (with `=') is a single +# command-line argument, so we just need to shift the counter by one. The +# latter (without `=') is two arguments, so we'll need two shifts. +while [[ $# -gt 0 ]] +do + case $1 in + # Input parameters. + -b=*|--build-dir=*) build_dir="${1#*=}"; check_v $1 "$build_dir"; shift;; + -b|--builddir) build_dir="$2"; check_v $1 "$build_dir"; shift;shift;; + -i=*|--inputdir=*) input_dir="${1#*=}"; check_v $1 "$input_dir"; shift;; + -i|--inputdir) input_dir="$2"; check_v $1 "$input_dir"; shift;shift;; + -s=*|--software-dir=*) software_dir="${1#*=}"; check_v $1 "$software_dir"; shift;; + -s|--software-dir) software_dir="$2"; check_v $1 "$software_dir"; shift;shift;; + -m=*|--minmapsize=*) minmapsize="${1#*=}"; check_v $1 "$minmapsize"; shift;; + -m|--minmapsize) minmapsize="$2"; check_v $1 "$minmapsize"; shift;shift;; + + # Operating mode options. + -j=*|--jobs=*) jobs="${1#*=}"; check_v $1 "$jobs"; shift;; + -j|--jobs) jobs="$2"; check_v $1 "$jobs"; shift;shift;; + -e=*|--existing-conf=*) existing_conf="${1#*=}"; check_v $1 "$existing_conf"; shift;; + -e|--existing-conf) existing_conf="$2"; check_v $1 "$existing_conf"; shift;shift;; + -?|--help) print_help; exit 0;; + + # Unrecognized option: + -*) echo "$scriptname: unknown option '$1'"; exit 1;; + + # Not an option, an argument. + *) echo "The configure script doesn't accept arguments."; exit 1;; + esac +done + + + + + +# Internal directories +# -------------------- # # These are defined to help make this script more readable. topdir=$(pwd) +lbdir=.build installedlink=.local -lbdir=reproduce/build cdir=reproduce/config optionaldir="/optional/path" @@ -141,22 +263,11 @@ printnotice=yes rewritepconfig=yes rewritegconfig=yes if [ -f $pconf ] || [ -f $glconf ]; then - - # If it already exits, see what the user wants to do. - echo "Atleast one local configuration file already exists." - echo - while [ "$userread" != "y" -a "$userread" != "n" ] - do - read -p"Re-write existing configuration file(s) (y/n)? " userread - done - - # Set `rewriteconfig'. - if [ $userread = "n" ]; then + if [ $existing_conf = 1 ]; then printnotice=no if [ -f $pconf ]; then rewritepconfig=no; fi if [ -f $glconf ]; then rewritegconfig=no; fi fi - echo fi @@ -220,10 +331,10 @@ if [ $rewritepconfig = yes ]; then !!!!!!!!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -Couldn't find GNU Wget. It is used for downloading necessary programs and -data if they aren't already present in the specified directories. Therefore -the pipeline will crash if the necessary files are not already present on -the system. +Couldn't find GNU Wget, or cURL on this system. These programs are used for +downloading necessary programs and data if they aren't already present (in +directories that you can specify with this configure script). Therefore if +the necessary files are not present, the pipeline will crash. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! @@ -245,12 +356,13 @@ if [ $rewritepconfig = yes ]; then Build directory =============== -The "source" (this directory) and "build" directories are treated +The project's "source" (this directory) and "build" directories are treated separately. This greatly helps in managing the many intermediate files that are created during the build. The intermediate build files don't need to be -archived or backed up: you can always re-build them with this reproduction -pipeline. The build directory also needs a relatively large amount of free -space (atleast serveral Giga-bytes). +archived or backed up: you can always re-build them with the contents of +the source directory. The build directory also needs a relatively large +amount of free space (atleast serveral Giga-bytes), while the source +directory (all plain text) will usually be a mega-byte or less. '$lbdir' (a symbolic link to the build directory) will also be created during this configuration. It can help encourage you to set the actual @@ -258,30 +370,36 @@ build directory in a very different address from this one (one that can be deleted and has large volume), while having easy access to it from here. EOF - bdir="" + bdir= junkname=pure-junk-974adfkj38 while [ x$bdir == x ] do - # Ask the user. - read -p"Please enter the top build directory: " inbdir + # Ask the user (if not already set on the command-line). + if [ x"$build_dir" = x ]; then + read -p"Please enter the top build directory: " build_dir + fi # If it exists, see if we can write in it. If not, try making it. - if [ -d $inbdir ]; then - if mkdir $inbdir/$junkname 2> /dev/null; then - bdir=$(absolute_dir $inbdir) + if [ -d $build_dir ]; then + if mkdir $build_dir/$junkname 2> /dev/null; then + bdir=$(absolute_dir $build_dir) echo " -- Build directory: '$bdir'" - rm -rf $inbdir/$junkname + rm -rf $build_dir/$junkname else - echo " -- Can't write in '$inbdir'" + echo " -- Can't write in '$build_dir'" fi else - if mkdir $inbdir 2> /dev/null; then - bdir=$(absolute_dir $inbdir) + if mkdir $build_dir 2> /dev/null; then + bdir=$(absolute_dir $build_dir) echo " -- Build directory set to (the newly created): '$bdir'" else - echo " -- Can't create '$inbdir'" + echo " -- Can't create '$build_dir'" fi fi + + # Reset `build_dir' to blank, so it continues asking when the + # previous value wasn't usable. + build_dir= done fi @@ -291,12 +409,16 @@ fi # Input directory # --------------- -indir=$optionaldir +if [ x"$input_dir" = x ]; then + indir=$optionaldir +else + indir=$input_dir +fi wfpc2name=$(awk '!/^#/ && $1=="WFPC2IMAGE" {print $3}' $pdir/INPUTS.mk) wfpc2md5=$(awk '!/^#/ && $1=="WFPC2MD5" {print $3}' $pdir/INPUTS.mk) wfpc2size=$(awk '!/^#/ && $1=="WFPC2SIZE" {print $3}' $pdir/INPUTS.mk) wfpc2url=$(awk '!/^#/ && $1=="WFPC2URL" {print $3}' $pdir/INPUTS.mk) -if [ $rewritepconfig = yes ]; then +if [ $rewritepconfig = yes ] && [ x"$input_dir" = x ]; then cat <<EOF ---------------------------------- @@ -338,12 +460,16 @@ fi # Dependency tarball directory # ---------------------------- -if [ $rewritepconfig = yes ]; then +if [ x"$software_dir" = x ]; then ddir=$optionaldir +else + ddir=$software_dir +fi +if [ $rewritepconfig = yes ] && [ x"$software_dir" = x ]; then cat <<EOF --------------------------------------- -(OPTIONAL) Dependency tarball directory +(OPTIONAL) Software tarball directory --------------------------------------- To ensure an identical build environment, the pipeline will use its own @@ -367,42 +493,6 @@ fi -# Memory mapping minimum size -# --------------------------- -if [ $rewritegconfig = yes ]; then - defaultminmapsize=10000000000 - minmapsize=$defaultminmapsize - cat <<EOF - ---------------------------- -Minimum memory mapping size ---------------------------- - -Some programs (for example Gnuastro) can deal with cases where the local -system doesn't have enough memory (RAM) to keep large files. For example, -they will create memory-mapped (mmap) files on the HDD or SSD and -read/write to/from them instead of RAM. This will ofcourse, slow down the -processing, but atleast the program won't crash. - -Since the memory requirements of different systems are different and it has -no effect on the software's final result, the minimum size of an allocated -array to warrant a mapping to HDD/SSD instead of RAM must also be defined -here. This value will be used in the programs that support this feature. - -EOF - - read -p"Minimum memory mapping size in bytes (default: $minmapsize): " \ - tmpminmapsize - if [ x"$tmpminmapsize" != x ]; then - minmapsize=$tmpminmapsize - echo " -- Using '$minmapsize'" - fi -fi - - - - - # Write the parameters into the local configuration file. if [ $rewritepconfig = yes ]; then @@ -418,8 +508,12 @@ if [ $rewritepconfig = yes ]; then $pconf.in >> $pconf else # Read the values from existing configuration file. - inbdir=$(awk '$1=="BDIR" {print $3}' $pconf) - downloader=$(awk '$1=="DOWNLOADER" {print $3}' $pconf) + inbdir=$(awk '$1=="BDIR" {print $3}' $pconf) + + # The downloader command may contain multiple elements, so we'll just + # change the (in memory) first and second tokens to empty space and + # write the full line (the original file is unchanged). + downloader=$(awk '$1=="DOWNLOADER" {$1=""; $2=""; print $0}' $pconf) # Make sure all necessary variables have a value err=0 @@ -676,16 +770,79 @@ fi +# Build `flock' as first program +# ------------------------------ +# +# Flock (or file-lock) is a unique program in the pipeline that is +# necessary to serialize the (generally parallel) processing of make when +# necessary. GNU/Linux machines have it as part of their `util-linux' +# programs. But to be consistent, we will be using our own build. +# +# The reason its sepecial is that we need it to serialize the download +# process of the dependency tarballs. +flockversion=$(awk '/flock-version/{print $3}' \ + reproduce/config/pipeline/dependency-versions.mk) +flocktar=flock-$flockversion.tar.gz +flockurl=http://github.com/discoteq/flock/releases/download/v$flockversion/ + +# Prepare/download the tarball. +if ! [ -f $tardir/$flocktar ]; then + if [ -f $ddir/$flocktar ]; then + cp $ddir/$flocktar $tardir/$flocktar + else + if ! $downloader $tardir/$flocktar $flockurl/$flocktar; then + rm -f $tardir/$flocktar; + echo + echo "DOWNLOAD ERROR: Couldn't download the 'flock' tarball:" + echo " $flockurl" + echo + echo "You can manually place it in '$ddir' to avoid downloading." + exit 1 + fi + fi +fi + +# If the tarball is newer than the (possibly existing) program, then delete +# the program. +if [ -f .local/bin/flock ]; then + if [ $tardir/$flocktar -nt .local/bin/flock ]; then + rm .local/bin/flock + fi +fi + +# Build `flock' if necessary. +if ! [ -f .local/bin/flock ]; then + cd $depdir + tar xf $tardir/$flocktar + cd flock-$flockversion + ./configure --prefix=$instdir + make; make install + cd $topdir + rm -rf $depdir/flock-$flockversion +fi + + + + + # Build Basic dependencies # ------------------------ # -# Since the system might not have GNU Coreutils at this stage, we'll just -# default to 4 threads if the actual number isn't found. This is because -# some versions of Make complain about not having enough 'pipe' (memory) on -# some systems. After some searching, I found out its because of too many -# threads. -if which nproc > /dev/null 2>/dev/null; then numthreads=$(nproc --all); -else numthreads=1; +# Since the system might not have GNU Make at this stage, and other Make +# implementations can't deal with parallel build properly, we'll just +# default to 1 thread. This is because some versions of Make complain about +# not having enough 'pipe' (memory) on some systems. After some searching, +# I found out its because of too many threads. GNU Make will be present on +# GNU systems (that have `nproc', part of GNU Coreutils). So to simplify +# the test for GNU Make, we'll just try running `nproc'. +if which nproc > /dev/null 2>/dev/null; then + if [ $jobs = 0 ]; then + numthreads=$(nproc --all); + else + numthreads=$jobs + fi +else + numthreads=1; fi make -f reproduce/src/make/dependencies-basic.mk \ rpath_command=$rpath_command \ @@ -1,3 +1,5 @@ +%% Copyright (C) 2018-2019 Mohammad Akhlaghi. +%% See the end of the file for license conditions. \documentclass[10pt, twocolumn]{article} %% This is a convenience variable if you are using PGFPlots to build plots @@ -251,3 +253,19 @@ without them. %% Finish LaTeX \end{document} + +%% This file is part of the reproducible paper template +%% https://gitlab.com/makhlaghi/reproducible-paper +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with Template. If not, see <https://www.gnu.org/licenses/>. diff --git a/reproduce/config/gnuastro/astbuildprog.conf b/reproduce/config/gnuastro/astbuildprog.conf index 8413e07..a6da3c4 100644 --- a/reproduce/config/gnuastro/astbuildprog.conf +++ b/reproduce/config/gnuastro/astbuildprog.conf @@ -1,6 +1,11 @@ # BuildProgram's configuration file depends on the build and is installed # along with the rest of Gnuastro. So we'll set any specific configuration # we want here, then import those. +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice and +# this notice are preserved. This file is offered as-is, without any +# warranty. # To keep the program's compiled excutable file, comment this line. deletecompiled 1 diff --git a/reproduce/config/gnuastro/gnuastro.conf b/reproduce/config/gnuastro/gnuastro.conf index 561df9f..20ba4a9 100644 --- a/reproduce/config/gnuastro/gnuastro.conf +++ b/reproduce/config/gnuastro/gnuastro.conf @@ -9,6 +9,11 @@ # The rest of this configuration file in this template reproduction # pipeline is taken from the default Gnuastro configuration from its # source (`bin/gnuastro.conf'). +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice and +# this notice are preserved. This file is offered as-is, without any +# warranty. # Reproduction pipeline (`config' has to be before `lastconfig'). config .gnuastro/gnuastro-local.conf diff --git a/reproduce/config/pipeline/INPUTS.mk b/reproduce/config/pipeline/INPUTS.mk index 3522ecc..6d08d9f 100644 --- a/reproduce/config/pipeline/INPUTS.mk +++ b/reproduce/config/pipeline/INPUTS.mk @@ -1,7 +1,11 @@ # Input files necessary for this pipeline. # # This file is read by the configure script and running Makefiles. - +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice and +# this notice are preserved. This file is offered as-is, without any +# warranty. WFPC2IMAGE = WFPC2ASSNu5780205bx.fits WFPC2MD5 = a4791e42cd1045892f9c41f11b50bad8 diff --git a/reproduce/config/pipeline/LOCAL.mk.in b/reproduce/config/pipeline/LOCAL.mk.in index 41298ee..00b907a 100644 --- a/reproduce/config/pipeline/LOCAL.mk.in +++ b/reproduce/config/pipeline/LOCAL.mk.in @@ -2,6 +2,11 @@ # # This is just a template for the `./configure' script to fill in. Please # don't make any change to this file. +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice and +# this notice are preserved. This file is offered as-is, without any +# warranty. BDIR = @bdir@ INDIR = @indir@ DEPENDENCIES-DIR = @ddir@ diff --git a/reproduce/config/pipeline/dependency-texlive.mk b/reproduce/config/pipeline/dependency-texlive.mk index 9d7365f..71c0648 100644 --- a/reproduce/config/pipeline/dependency-texlive.mk +++ b/reproduce/config/pipeline/dependency-texlive.mk @@ -4,6 +4,12 @@ # name to this variable (you can check in `ctan.org' to find the official # name). # +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice and +# this notice are preserved. This file is offered as-is, without any +# warranty. + + # Note on `tex' and `fancyhdr': These two packages are installed along with # the basic installation scheme that we used to install tlmgr, they will be # ignored in the `tlmgr install' command, but will be used later when we diff --git a/reproduce/config/pipeline/dependency-versions.mk b/reproduce/config/pipeline/dependency-versions.mk index 4cb8a71..2e74a45 100644 --- a/reproduce/config/pipeline/dependency-versions.mk +++ b/reproduce/config/pipeline/dependency-versions.mk @@ -1,4 +1,23 @@ -# Versions of the various dependnecies +# Versions of the various dependencies +# +# Original author: +# Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Contributing author(s): +# Raul Infante-Sainz <infantesainz@gmail.com> +# Copyright (C) 2018-2019, Your Name. +# +# This Makefile is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# This Makefile is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# A copy of the GNU General Public License is available at +# <http://www.gnu.org/licenses/>. # Programs bash-version = 5.0 diff --git a/reproduce/config/pipeline/pdf-build.mk b/reproduce/config/pipeline/pdf-build.mk index 504d0bd..ee421ab 100644 --- a/reproduce/config/pipeline/pdf-build.mk +++ b/reproduce/config/pipeline/pdf-build.mk @@ -1,14 +1,19 @@ # Make the final PDF? # ------------------- # -# During the testing a pipeline, it is usually not necessary to build -# the PDF file (which makes a lot of output lines on the command-line -# and can make it hard to find the commands and possible errors (and -# their outputs). Also, in some cases, only the produced results may -# be of interest and not the final PDF, so LaTeX (and its necessary -# packages) may not be installed. +# During the testing a pipeline, it is usually not necessary to build the +# PDF file (which makes a lot of output lines on the command-line and can +# make it hard to find the commands and possible errors (and their +# outputs). Also, in some cases, only the produced results may be of +# interest and not the final PDF, so LaTeX (and its necessary packages) may +# not be installed. # # If this variable is given any string, a PDF will be made with -# LaTeX. Otherwise, a notice will just printed that for now, no PDF -# will be created. +# LaTeX. Otherwise, a notice will just printed that for now, no PDF will be +# created. +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice and +# this notice are preserved. This file is offered as-is, without any +# warranty. pdf-build-final = yes diff --git a/reproduce/config/pipeline/texlive.conf b/reproduce/config/pipeline/texlive.conf index 3aa5db5..1f515c1 100644 --- a/reproduce/config/pipeline/texlive.conf +++ b/reproduce/config/pipeline/texlive.conf @@ -2,6 +2,11 @@ # # installdir: Install directory # topdir: Top pipeline directory +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice and +# this notice are preserved. This file is offered as-is, without any +# warranty. selected_scheme scheme-basic TEXDIR @installdir@/texlive/2018 TEXMFCONFIG @topdir@/.texlive2018/texmf-config diff --git a/reproduce/src/bash/download-multi-try b/reproduce/src/bash/download-multi-try index 1be2ffa..cdeca8e 100755 --- a/reproduce/src/bash/download-multi-try +++ b/reproduce/src/bash/download-multi-try @@ -1,11 +1,13 @@ -#!.local/bin/bash -# # Attempt downloading multiple times before crashing whole pipeline. From # the top project directory (for the shebang above), this script must be # run like this: # # $ /path/to/download-multi-try downloader lockfile input-url downloaded-name # +# NOTE: The `downloader' must contain the option to specify the output name +# in its end. For example "wget -O". Any other option can also be placed in +# the middle. +# # Due to temporary network problems, a download may fail suddenly, but # succeed in a second try a few seconds later. Without this script that # temporary glitch in the network will permanently crash the pipeline and @@ -70,7 +72,7 @@ fi -# Try downloading multiple times before crashing +# Try downloading multiple times before crashing. counter=0 maxcounter=10 while [ ! -f "$outname" ]; do @@ -97,9 +99,12 @@ while [ ! -f "$outname" ]; do sleep $tstep fi - # Attempt downloading the file (one-at-a-time). + # Attempt downloading the file (one-at-a-time). Note that the + # `downloader' ends with the respective option to specify the output + # name. For example "wget -O" (so `outname', that comes after it) will + # be the name of the downloaded file. flock "$lockfile" bash -c \ - "if ! $downloader -O$outname $inurl; then rm -f $outname; fi" + "if ! $downloader $outname $inurl; then rm -f $outname; fi" done diff --git a/reproduce/src/bash/git-post-checkout b/reproduce/src/bash/git-post-checkout index 153b277..a56c21f 100644 --- a/reproduce/src/bash/git-post-checkout +++ b/reproduce/src/bash/git-post-checkout @@ -4,10 +4,10 @@ # controlled files (with each commit) using the `metastore' program. # # This script is taken from the `examples/hooks/pre-commit' file of the -# `metastore' package (installed within the pipeline, with an MIT -# license). We have just changed the name of the `MSFILE' and also set -# special characters for the installation location of meta-store so our own -# installation is found by Git. +# `metastore' package (installed within the pipeline, with an MIT license +# for copyright). We have just changed the name of the `MSFILE' and also +# set special characters for the installation location of meta-store so our +# own installation is found by Git. # File containig the metadata and metastore executable. MSFILE=".file-metadata" diff --git a/reproduce/src/bash/git-pre-commit b/reproduce/src/bash/git-pre-commit index 0fa6c52..cd06c25 100644 --- a/reproduce/src/bash/git-pre-commit +++ b/reproduce/src/bash/git-pre-commit @@ -4,10 +4,10 @@ # controlled files (with each commit) using the `metastore' program. # # This script is taken from the `examples/hooks/pre-commit' file of the -# `metastore' package (installed within the pipeline, with an MIT -# license). We have just changed the name of the `MSFILE' and also set -# special characters for the installation location of meta-store so our own -# installation is found by Git. +# `metastore' package (installed within the pipeline, with an MIT license +# for copyright). We have just changed the name of the `MSFILE' and also +# set special characters for the installation location of meta-store so our +# own installation is found by Git. # # WARNING: # diff --git a/reproduce/src/make/dependencies-basic.mk b/reproduce/src/make/dependencies-basic.mk index 74a04ed..52a41d9 100644 --- a/reproduce/src/make/dependencies-basic.mk +++ b/reproduce/src/make/dependencies-basic.mk @@ -43,12 +43,13 @@ include reproduce/config/pipeline/LOCAL.mk include reproduce/src/make/dependencies-build-rules.mk include reproduce/config/pipeline/dependency-versions.mk -ddir = $(BDIR)/dependencies -tdir = $(BDIR)/dependencies/tarballs -idir = $(BDIR)/dependencies/installed -ibdir = $(BDIR)/dependencies/installed/bin -ildir = $(BDIR)/dependencies/installed/lib -ilidir = $(BDIR)/dependencies/installed/lib/built +lockdir = $(BDIR)/locks +ddir = $(BDIR)/dependencies +tdir = $(BDIR)/dependencies/tarballs +idir = $(BDIR)/dependencies/installed +ibdir = $(BDIR)/dependencies/installed/bin +ildir = $(BDIR)/dependencies/installed/lib +ilidir = $(BDIR)/dependencies/installed/lib/built # We'll need the system's PATH for making links to low-level programs we # won't be building ourselves. @@ -94,6 +95,8 @@ all: $(foreach p, $(top-level-programs), $(ibdir)/$(p)) # # However, downloading from this link is slow (because its just a link). So # its easier to just keep a with the others. +$(lockdir): | $(BDIR); mkdir $@ +downloadwrapper = ./reproduce/src/bash/download-multi-try tarballs = $(foreach t, bash-$(bash-version).tar.gz \ binutils-$(binutils-version).tar.lz \ bzip2-$(bzip2-version).tar.gz \ @@ -123,7 +126,7 @@ tarballs = $(foreach t, bash-$(bash-version).tar.gz \ xz-$(xz-version).tar.gz \ zlib-$(zlib-version).tar.gz \ , $(tdir)/$(t) ) -$(tarballs): $(tdir)/%: +$(tarballs): $(tdir)/%: $(lockdir) if [ -f $(DEPENDENCIES-DIR)/$* ]; then \ cp $(DEPENDENCIES-DIR)/$* $@; \ else \ @@ -132,32 +135,32 @@ $(tarballs): $(tdir)/%: | awk '{print $$1}' ); \ \ mergenames=1; \ - if [ $$n = bash ]; then w=http://ftpmirror.gnu.org/gnu/bash; \ - elif [ $$n = binutils ]; then w=http://ftpmirror.gnu.org/gnu/binutils; \ + if [ $$n = bash ]; then w=http://ftp.gnu.org/gnu/bash; \ + elif [ $$n = binutils ]; then w=http://ftp.gnu.org/gnu/binutils; \ elif [ $$n = bzip ]; then w=http://akhlaghi.org/src; \ elif [ $$n = cert ]; then w=http://akhlaghi.org/src; \ - elif [ $$n = coreutils ]; then w=http://ftpmirror.gnu.org/gnu/coreutils;\ - elif [ $$n = diffutils ]; then w=http://ftpmirror.gnu.org/gnu/diffutils;\ + elif [ $$n = coreutils ]; then w=http://ftp.gnu.org/gnu/coreutils;\ + elif [ $$n = diffutils ]; then w=http://ftp.gnu.org/gnu/diffutils;\ elif [ $$n = findutils ]; then w=http://akhlaghi.org/src; \ - elif [ $$n = gawk ]; then w=http://ftpmirror.gnu.org/gnu/gawk; \ + elif [ $$n = gawk ]; then w=http://ftp.gnu.org/gnu/gawk; \ elif [ $$n = gcc ]; then w=http://ftp.gnu.org/gnu/gcc/gcc-$(gcc-version); \ elif [ $$n = gmp ]; then w=https://gmplib.org/download/gmp; \ - elif [ $$n = grep ]; then w=http://ftpmirror.gnu.org/gnu/grep; \ - elif [ $$n = gzip ]; then w=http://ftpmirror.gnu.org/gnu/gzip; \ + elif [ $$n = grep ]; then w=http://ftp.gnu.org/gnu/grep; \ + elif [ $$n = gzip ]; then w=http://ftp.gnu.org/gnu/gzip; \ elif [ $$n = isl ]; then w=ftp://gcc.gnu.org/pub/gcc/infrastructure; \ elif [ $$n = lzip ]; then w=http://download.savannah.gnu.org/releases/lzip; \ elif [ $$n = make ]; then w=http://akhlaghi.org/src; \ elif [ $$n = mpfr ]; then w=http://www.mpfr.org/mpfr-current;\ - elif [ $$n = mpc ]; then w=http://ftpmirror.gnu.org/gnu/mpc;\ - elif [ $$n = ncurses ]; then w=http://ftpmirror.gnu.org/gnu/ncurses;\ + elif [ $$n = mpc ]; then w=http://ftp.gnu.org/gnu/mpc; \ + elif [ $$n = ncurses ]; then w=http://ftp.gnu.org/gnu/ncurses; \ elif [ $$n = openssl ]; then w=http://www.openssl.org/source; \ elif [ $$n = patchelf ]; then w=http://nixos.org/releases/patchelf/patchelf-$(patchelf-version); \ elif [ $$n = pkg ]; then w=http://pkg-config.freedesktop.org/releases; \ - elif [ $$n = readline ]; then w=http://ftpmirror.gnu.org/gnu/readline; \ - elif [ $$n = sed ]; then w=http://ftpmirror.gnu.org/gnu/sed;\ - elif [ $$n = tar ]; then w=http://ftpmirror.gnu.org/gnu/tar;\ - elif [ $$n = wget ]; then w=http://ftpmirror.gnu.org/gnu/wget;\ - elif [ $$n = which ]; then w=http://ftpmirror.gnu.org/gnu/which;\ + elif [ $$n = readline ]; then w=http://ftp.gnu.org/gnu/readline; \ + elif [ $$n = sed ]; then w=http://ftp.gnu.org/gnu/sed; \ + elif [ $$n = tar ]; then w=http://ftp.gnu.org/gnu/tar; \ + elif [ $$n = wget ]; then w=http://ftp.gnu.org/gnu/wget; \ + elif [ $$n = which ]; then w=http://ftp.gnu.org/gnu/which; \ elif [ $$n = xz ]; then w=http://tukaani.org/xz; \ elif [ $$n = zlib ]; then w=http://www.zlib.net; \ else \ @@ -178,10 +181,9 @@ $(tarballs): $(tdir)/%: downloader="$(DOWNLOADER)"; \ fi; \ \ - if ! $$downloader $@ $$tarballurl; then \ - rm -f $@; \ - echo; echo "DOWNLOAD FAILED: $$tarballurl"; echo; exit 1; \ - fi; \ + touch $(lockdir)/download; \ + $(downloadwrapper) "$$downloader" $(lockdir)/download \ + $$tarballurl $@; \ fi diff --git a/reproduce/src/make/dependencies.mk b/reproduce/src/make/dependencies.mk index fe29d1b..fdbfd7f 100644 --- a/reproduce/src/make/dependencies.mk +++ b/reproduce/src/make/dependencies.mk @@ -103,7 +103,6 @@ tarballs = $(foreach t, cfitsio-$(cfitsio-version).tar.gz \ atlas-$(atlas-version).tar.bz2 \ cmake-$(cmake-version).tar.gz \ curl-$(curl-version).tar.gz \ - flock-$(flock-version).tar.xz \ freetype-$(freetype-version).tar.gz \ fftw-$(fftw-version).tar.gz \ ghostscript-$(ghostscript-version).tar.gz \ @@ -151,7 +150,6 @@ $(tarballs): $(tdir)/%: w=https://sourceforge.net/projects/math-atlas/files/Stable/$(atlas-version)/atlas$(atlas-version).tar.bz2/download elif [ $$n = cmake ]; then w=https://cmake.org/files/v3.12 elif [ $$n = curl ]; then w=https://curl.haxx.se/download - elif [ $$n = flock ]; then w=https://github.com/discoteq/flock/releases/download/v$(flock-version) elif [ $$n = fftw ]; then w=ftp://ftp.fftw.org/pub/fftw elif [ $$n = freetype ]; then w=https://download.savannah.gnu.org/releases/freetype elif [ $$n = hdf ]; then @@ -160,14 +158,14 @@ $(tarballs): $(tdir)/%: w=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-$$majorver/hdf5-$(hdf5-version)/src/$* elif [ $$n = ghostscript ]; then w=https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs926 elif [ $$n = git ]; then w=http://mirrors.edge.kernel.org/pub/software/scm/git - elif [ $$n = gnuastro ]; then w=http://ftpmirror.gnu.org/gnu/gnuastro - elif [ $$n = gsl ]; then w=http://ftpmirror.gnu.org/gnu/gsl + elif [ $$n = gnuastro ]; then w=http://ftp.gnu.org/gnu/gnuastro + elif [ $$n = gsl ]; then w=http://ftp.gnu.org/gnu/gsl elif [ $$n = install ]; then w=http://mirror.ctan.org/systems/texlive/tlnet elif [ $$n = jpegsrc ]; then w=http://ijg.org/files elif [ $$n = lapack ]; then w=http://www.netlib.org/lapack elif [ $$n = libbsd ]; then w=http://libbsd.freedesktop.org/releases elif [ $$n = libpng ]; then w=https://download.sourceforge.net/libpng - elif [ $$n = libtool ]; then w=http://ftpmirror.gnu.org/gnu/libtool + elif [ $$n = libtool ]; then w=http://ftp.gnu.org/gnu/libtool elif [ $$n = libgit ]; then mergenames=0 w=https://github.com/libgit2/libgit2/archive/v$(libgit2-version).tar.gz @@ -511,9 +509,6 @@ $(ibdir)/glibtool: $(tdir)/libtool-$(libtool-version).tar.xz $(ibdir)/gs: $(tdir)/ghostscript-$(ghostscript-version).tar.gz $(call gbuild, $<, ghostscript-$(ghostscript-version)) -$(ibdir)/flock: $(tdir)/flock-$(flock-version).tar.xz - $(call gbuild, $<, flock-$(flock-version), static) - $(ibdir)/git: $(tdir)/git-$(git-version).tar.xz \ $(ibdir)/curl $(call gbuild, $<, git-$(git-version), static, \ diff --git a/reproduce/src/make/download.mk b/reproduce/src/make/download.mk index a1efde7..036332b 100644 --- a/reproduce/src/make/download.mk +++ b/reproduce/src/make/download.mk @@ -69,7 +69,8 @@ $(inputdatasets): $(indir)/%.fits: | $(indir) $(lockdir) ln -s $(INDIR)/$$origname $@ else touch $(lockdir)/download - $(downloadwrapper) wget $(lockdir)/download $$url/$$origname $@ + $(downloadwrapper) "wget --no-use-server-timestamps -O" \ + $(lockdir)/download $$url/$$origname $@ fi # Check the md5 sum to see if this is the proper dataset. diff --git a/tex/src/delete-me-demo.tex b/tex/src/delete-me-demo.tex index 8174bc8..2575e61 100644 --- a/tex/src/delete-me-demo.tex +++ b/tex/src/delete-me-demo.tex @@ -1,3 +1,23 @@ +%% Plot the demonstration image and its histogram. +% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Copyright (C) 2019, Mohammad Akhlaghi. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with this template. If not, see <http://www.gnu.org/licenses/>. + \begin{tikzpicture} %% The displayed WFPC2 image. diff --git a/tex/src/delete-me.tex b/tex/src/delete-me.tex index fc1a1f9..1382b66 100644 --- a/tex/src/delete-me.tex +++ b/tex/src/delete-me.tex @@ -1,4 +1,22 @@ %% PGFPlots code to plot a random set of numbers as demo +%% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Copyright (C) 2019, Mohammad Akhlaghi. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with this template. If not, see <http://www.gnu.org/licenses/>. \begin{tikzpicture} diff --git a/tex/src/preamble-biblatex.tex b/tex/src/preamble-biblatex.tex index bd99502..6358fb1 100644 --- a/tex/src/preamble-biblatex.tex +++ b/tex/src/preamble-biblatex.tex @@ -11,7 +11,24 @@ %% reference. The file suffix doesn't have to be `.bib'. This naming %% helps in clearly identifying the files and avoiding places that %% complain about `.bib' files. - +% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Copyright (C) 2018-2019, Mohammad Akhlaghi. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with this template. If not, see <http://www.gnu.org/licenses/>. diff --git a/tex/src/preamble-header.tex b/tex/src/preamble-header.tex index eb83c27..b2ca364 100644 --- a/tex/src/preamble-header.tex +++ b/tex/src/preamble-header.tex @@ -1,5 +1,23 @@ %% The headers: title, authors, top of pages and section title formatting %% of the final LaTeX file are configured here. +% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Copyright (C) 2018-2019, Mohammad Akhlaghi. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with this template. If not, see <http://www.gnu.org/licenses/>. diff --git a/tex/src/preamble-necessary.tex b/tex/src/preamble-necessary.tex index e6ac18e..cdb6137 100644 --- a/tex/src/preamble-necessary.tex +++ b/tex/src/preamble-necessary.tex @@ -3,6 +3,24 @@ %% These are a set of packages that have been commonly necessary in most %% LaTeX usages. However, if any are not needed in your work, please feel %% free to remove them. +% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Copyright (C) 2018-2019, Mohammad Akhlaghi. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with this template. If not, see <http://www.gnu.org/licenses/>. diff --git a/tex/src/preamble-pgfplots.tex b/tex/src/preamble-pgfplots.tex index 4999656..cd68658 100644 --- a/tex/src/preamble-pgfplots.tex +++ b/tex/src/preamble-pgfplots.tex @@ -1,11 +1,11 @@ %% PGFPlots settings %% ----------------- -%% +% %% PGFPLOTS is a package in (La)TeX for making plots internally. It fits %% nicely with the purpose of a reproduction pipeline. But it isn't %% mandatory. Therefore if you don't need it, just comment/delete the line %% that includes this file in the top LaTeX source (`paper.tex'). -%% +% %% However, TiKZ and PGFPlots are the recommended way to include figures %% and plots in your paper. There are two main reasons: 1) it follows the %% same LaTeX settings as the text of the paper, so the figures will be in @@ -17,21 +17,21 @@ %% but it requires Python and Numpy. The latter is not easy to build from %% source, so after a few years, installing the required version can be %% very frustrating. -%% +% %% Keeping all BibLaTeX settings in a separate preamble was done in the %% spirit of modularity to 1) easily managable, 2) If a similar BibLaTeX %% configuration is necessary in another LaTeX compilation, this file can %% just be copied there and used. -%% +% %% PGFPlots uses the (La)TeX TiKZ package to build plots. So we will first %% do the settings that are necessary in TiKZ, and then go onto the actual %% PGFPlots package. %% %% USAGE: -%% +% %% - All plots are made within a `tikz' directory (that must already be %% present in the location LaTeX is run). -%% +% %% - Use `\includetikz{XXXX}' to make/use the figure. If a `makepdf' LaTeX %% macro is not defined, then \includetikz will assume a `XXXX.pdf' file %% exists in `tex/tikz' and simply import it. If `makepdf' is defined, @@ -39,7 +39,24 @@ %% on `tex/XXXX.tex'. Note that if the contents of `tex/src/XXXX.tex' %% hasn't changed since the last build. TiKZ/PGFPlots won't rebuild the %% plot. - +% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Copyright (C) 2018-2019, Mohammad Akhlaghi. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with this template. If not, see <http://www.gnu.org/licenses/>. diff --git a/tex/src/preamble-style.tex b/tex/src/preamble-style.tex index fb08df7..54f3598 100644 --- a/tex/src/preamble-style.tex +++ b/tex/src/preamble-style.tex @@ -5,6 +5,24 @@ %% initial/internal phase or part of a larger effort, so it has a basic %% style defined here as a preamble. To ignore it, uncomment or delete the %% respective line in `paper.tex'. +%% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Copyright (C) 2019, Mohammad Akhlaghi. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with this template. If not, see <http://www.gnu.org/licenses/>. diff --git a/tex/src/references.tex b/tex/src/references.tex index 785ab21..c1c4084 100644 --- a/tex/src/references.tex +++ b/tex/src/references.tex @@ -1,3 +1,24 @@ +%% References used in the template paper. +% +%% Original author: +%% Mohammad Akhlaghi <mohammad@akhlaghi.org> +%% Contributing author(s): +%% Your name <your@email.address> +%% Copyright (C) 2018-2019, Your Name. +% +%% This template is free software: you can redistribute it and/or modify it +%% under the terms of the GNU General Public License as published by the +%% Free Software Foundation, either version 3 of the License, or (at your +%% option) any later version. +% +%% This template is distributed in the hope that it will be useful, but +%% WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +%% General Public License for more details. +% +%% You should have received a copy of the GNU General Public License along +%% with Template. If not, see <https://www.gnu.org/licenses/>. + @ARTICLE{astropy2018, author = {{Astropy Collaboration} and {Price-Whelan}, A.~M. and {Sip{\H o}cz}, B.~M. and {G{\"u}nther}, H.~M. and {Lim}, P.~L. and {Crawford}, S.~M. and |