diff options
| -rw-r--r-- | README-pipeline.md | 170 | ||||
| -rwxr-xr-x | configure | 2 | ||||
| -rw-r--r-- | reproduce/src/make/Top-Makefile | 122 | ||||
| -rw-r--r-- | reproduce/src/make/paper.mk | 56 | ||||
| -rw-r--r-- | reproduce/src/make/top.mk | 87 |
5 files changed, 231 insertions, 206 deletions
diff --git a/README-pipeline.md b/README-pipeline.md index 9ca0cd2..e4feb19 100644 --- a/README-pipeline.md +++ b/README-pipeline.md @@ -286,78 +286,81 @@ other servers: its contents are all products of the pipeline, and can be easily re-created any time. As you define targets for your new rules, it is thus important to place them all under sub-directories of `$(BDIR)`. -Let's start reviewing the processing with the top Makefile. Please open and -inspect it as we go along here. The first step (un-commented line) defines -the ultimate target (`paper.pdf`). You shouldn't modify this line. The rule -to build `paper.pdf` is in another Makefile that will be imported into this -top Makefile later. Don't forget that Make first scans the Makefile(s) once -completely (to define dependencies and etc) and starts its execution after -that. So it is fine to define the rule to build `paper.pdf` at a later -stage (this is one beauty of Make!). - -Having defined the top target, our next step is to include all the other -necessary Makefiles. First we include all Makefiles that satisfy this -wildcard: `reproduce/config/pipeline/*.mk`. These Makefiles don't actually -have any rules, they just have values for various free parameters -throughout the pipeline. Open a few of them to see for your self. These +In this architecture, we have two types of Makefiles that are loaded into +one: _configuration-Makefiles_ (only independent variables/configurations) +and _workhorse-Makefiles_ (Makefiles that actually contain rules). + +The configuration-Makefiles are those that satisfy this wildcard: +`reproduce/config/pipeline/*.mk`. These Makefiles don't actually have any +rules, they just have values for various free parameters throughout the +analysis/processing. Open a few of them to see for your self. These Makefiles must only contain raw Make variables (pipeline configurations). By raw we mean that the Make variables in these files must -not depend on variables in any other Makefile. This is because we don't -want to assume any order in reading them. It is very important to *not* -define any rule or other Make construct in any of these -_configuration-Makefiles_ (see the next paragraph for Makefiles with -rules). This will enable you to set the respective Makefiles in this -directory as a prerequisite to any target that depends on their variable -values. Therefore, if you change any of their values, all targets that -depend on those values will be re-built. - -Once all the raw variables have been imported into the top Makefile, we are -ready to import the Makefiles containing the details of the processing -steps (Makefiles containing rules, let's call these -_workhorse-Makefiles_). But in this phase *order is important*, because the -prerequisites of most rules will be other rules that will be defined at a -lower level (not a fixed name like `paper.pdf`). The lower-level rules must -be imported into Make before the higher-level ones. Hence, we can't use a -simple wildcard like when we imported configuration-Makefiles above. All -these Makefiles are defined in `reproduce/src/make`, therefore, the top -Makefile uses the `foreach` function to read them in a specific order. - -The main body of this pipeline is thus going to be managed within the -workhorse-Makefiles that are in `reproduce/src/make`. If you set -clear-to-understand names for these workhorse-Makefiles and follow the -convention of the top Makefile that you only include one workhorse-Makefile -per line, the `foreach` loop of the top Makefile that imports them will -become very easy to read and understand by eye. This will let you know -generally which step you are taking before or after another. Projects will -scale up very fast. Thus if you don't start and continue with a clean and -robust convention like this, in the end it will become very dirty and hard -to manage/understand (even for yourself). As a general rule of thumb, break -your rules into as many logically-similar but independent steps as -possible. +not depend on variables in any other configuration-Makefile. This is +because we don't want to assume any order in reading them. It is very +important to *not* define any rule or other Make construct in any of these +configuration-Makefiles. This will enable you to set the respective +Makefiles in this directory as a prerequisite to any target that depends on +their variable values. Therefore, if you change any of their values, all +targets that depend on those values will be re-built. + +The workhorse-Makefiles are those within the `reproduce/src/make` +directory. They contain the details of the processing steps (Makefiles +containing rules). But in this phase *order is important*, because the +prerequisites of most rules will be the targets of other rules that will be +defined prior to them (not a fixed name like `paper.pdf`). The lower-level +rules must be imported into Make before the higher-level ones. Hence, we +can't use a simple wildcard like when we imported configuration-Makefiles +above. All processing steps are assumed to ultimately (usually after many rules) end up in some number, image, figure, or table that are to be included in -the paper. The writing of the values into the final report is managed -through separate LaTeX files that only contain macros (a name given to a -number/string to be used in the LaTeX source, which will be replaced when -compiling it to the final PDF). So usually the last target in a Makefile is -a `.tex` file (with the same base-name as the Makefile, but in -`$(BDIR)/tex/macros`). This intermediate TeX file rule will only contain -commands to fill the TeX file up with values/names that were done in that -Makefile. As a result, if the targets in a workhorse-Makefile aren't -directly a prerequisite of other workhorse-Makefile targets, they should be -a pre-requisite of that intermediate LaTeX macro file. - -In `reproduce/src/make/paper.mk` contains the rule to build `paper.pdf` -(final target of the whole reproduction pipeline). If look in it, you will -notice that it depends on `tex/pipeline.tex`. Therefore, last part of the -top-level `Makefile` is the rule to build -`tex/pipeline.tex`. `tex/pipeline.tex` is the connection between the -processing steps of the pipeline, and the creation of the final -PDF. Therefore, to keep the over-all management clean, the rule to create -this bridge between the two phases is defined in the top-level `Makefile`. - -As you see in the top-level `Makefile`, `tex/pipeline.tex` is only a +the paper. The writing of these results into the final report/paper is +managed through separate LaTeX files that only contain macros (a name given +to a number/string to be used in the LaTeX source, which will be replaced +when compiling it to the final PDF). So usually the last target in a +workhorse-Makefile is a `.tex` file (with the same base-name as the +Makefile, but in `$(BDIR)/tex/macros`). As a result, if the targets in a +workhorse-Makefile aren't directly a prerequisite of other +workhorse-Makefile targets, they should be a pre-requisite of that +intermediate LaTeX macro file. Otherwise, they will be ignored by Make. + +Let's see how this design is implemented. When the `./configure` finishes, +it makes a `Makefile` in the top directory. This Makefile is just a +symbolic link to `reproduce/src/make/top.mk`. Please open and inspect it as +we go along here. The first step (un-commented line) defines the ultimate +target (`paper.pdf`). You shouldn't modify this line. The rule to build +`paper.pdf` is in `reproduce/src/make/paper.mk` that will be imported into +this top Makefile later. + +Having defined the top target, our next step is to include all the other +necessary Makefiles. But order matters in the importing of +workhorse-Makefiles and each must also have a TeX macro file with the same +base name (without a suffix). Therefore, the next step in the top-level +Makefile is to define a `makesrc` variable to keep the base names (without +a `.mk` suffix) of the workhorse-Makefiles that must be imported, in the +proper order. Having defined `makesrc`, in the next step, we'll just import +all the configuration-Makefiles with a wildcard and all workhorse-Makefiles +using a Make `foreach` loop to preserve the order. This finishes the +general view of the pipeline's implementation. + +In short, to keep things modular, readable and managable, follow these +recommendations: 1) Set clear-to-understand names for the +configuration-Makefiles, and workhorse-Makefiles, 2) Only import other +Makefiles from top Makefile. These will let you know/remember generally +which step you are taking before or after another. Projects will scale up +very fast. Thus if you don't start and continue with a clean and robust +convention like this, in the end it will become very dirty and hard to +manage/understand (even for yourself). As a general rule of thumb, break +your rules into as many logically-similar but independent steps as +possible. + +The `reproduce/src/make/paper.mk` Makefile must be the final Makefile that +is included. It ends with the rule to build `paper.pdf` (final target of +the whole reproduction pipeline). If look in it, you will notice that it +starts with a rule to create `tex/pipeline.tex`. `tex/pipeline.tex` is the +connection between the processing/analysis steps of the pipeline, and the +steps to build the final PDF. As you see, `tex/pipeline.tex` is only a merging/concatenation of LaTeX macros defined as the output of each high-level processing step (the separate work-horse Makefiles that you included). @@ -372,11 +375,12 @@ through the `\bdir` macro. During the research, it often happens that you want to test a step that is not a prerequisite of any higher-level operation. In such cases, you can -(temporarily) define the target of that rule as a prerequisite of -`tex/pipeline.tex`. If your test gives a promising result and you want to -include it in your research, set it as prerequisites to other rules and -remove it from the list of prerequisites for `tex/pipeline.tex`. In fact, -this is how a project is designed to grow in this framework. +(temporarily) define that processing as a rule in the most relevant +workhorse-Makefile and set its target as a prerequisite of its TeX +macro. If your test gives a promising result and you want to include it in +your research, set it as prerequisites to other rules and remove it from +the list of prerequisites for TeX macro file. In fact, this is how a +project is designed to grow in this framework. @@ -391,8 +395,8 @@ mind are listed below. - Define new `reproduce/src/make/XXXXXX.mk` workhorse-Makefile(s) with good and human-friendly name(s) replacing `XXXXXX`. - - Add `XXXXXX`, as a new line, to the loop which includes the - workhorse-Makefiles in the top-level `Makefile`. + - Add `XXXXXX`, as a new line, to the values in `makesrc` of the top-level + `Makefile`. - Do not use any constant numbers (or important names like filter names) in the workhorse-Makefiles or paper's LaTeX source. Define such @@ -402,9 +406,9 @@ mind are listed below. the variable defined in it. - Through any number of intermediate prerequisites, all processing steps - should end in (be a prerequisite of) - `tex/pipeline.tex`. `tex/pipeline.tex` is the bridge between the - processing steps and PDF-building steps. + should end in (be a prerequisite of) `tex/pipeline.tex` (defined in + `reproduce/src/make/paper.mk`). `tex/pipeline.tex` is the bridge between + the processing steps and PDF-building steps. @@ -498,12 +502,12 @@ advanced in later stages of your work. - **Title**, **short description** and **author** in source files: In this raw skeleton, the title or short description of your project should be - added in the following two files: `reproduce/src/make/Top-Makefile` - (the first line), and `tex/preamble-header.tex`. In both cases, the - texts you should replace are all in capital letters to make them - easier to identify. Of course, if you use a different LaTeX method of - managing the title and authors, please feel free to use your own - methods after finishing this checklist and doing your first commit. + added in the following two files: `reproduce/src/make/top.mk` (the + first line), and `tex/preamble-header.tex`. In both cases, the texts + you should replace are all in capital letters to make them easier to + identify. Of course, if you use a different LaTeX method of managing + the title and authors, please feel free to use your own methods after + finishing this checklist and doing your first commit. - **Gnuastro**: GNU Astronomy Utilities (Gnuastro) is currently a dependency of the pipeline which will be built and used. The main @@ -634,7 +634,7 @@ fi # # To see why this is the last step of the configuration, see above (when we # delete the top-level Makefile at the start of this script). -ln -s $(pwd)/reproduce/src/make/Top-Makefile Makefile +ln -s $(pwd)/reproduce/src/make/top.mk Makefile diff --git a/reproduce/src/make/Top-Makefile b/reproduce/src/make/Top-Makefile deleted file mode 100644 index 4c547be..0000000 --- a/reproduce/src/make/Top-Makefile +++ /dev/null @@ -1,122 +0,0 @@ -# A ONE-LINE DESCRIPTION OF THE WHOLE PIPELINE -# -# Original author: -# Mohammad Akhlaghi <mohammad@akhlaghi.org> -# Contributing author(s): -# Your name <your@email.address> -# Copyright (C) 2018, Your Name. -# -# This Makefile is free software: you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the -# Free Software Foundation, either version 3 of the License, or (at your -# option) any later version. -# -# This Makefile is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General -# Public License for more details. -# -# A copy of the GNU General Public License is available at -# <http://www.gnu.org/licenses/>. - - - - - -# Ultimate target of this pipeline -# -------------------------------- -# -# The final paper (in PDF format) is the main target of this whole -# reproduction pipeline. So as defined in the Make paradigm, we are -# defining it here. -# -# Note that if you don't have LaTeX to build the PDF, or generally are just -# interested in the processing, you can skip create the final PDF creation -# with `pdf-build-final' of `reproduce/config/pipeline/pdf-build.mk'. -all: paper.pdf - - - - - -# Include specific Makefiles -# -------------------------- -# -# To keep things clean, managable and readable, each set of operations is -# (and must be) classified (modularized) by context into separate -# Makefiles: the more the better. They are included in this top-level -# Makefile through the command below. -# -# To further help in readability, it is best to avoid including Makefiles -# within any other Makefile. So in short, it is best that the `foreach' -# loop below contains all the `reproduce/src/make/*.mk' files. -# -# IMPORTANT NOTE: order matters in the inclusion of the processing -# Makefiles. As the pipeline grows, some Makefiles will probably define -# variables/dependencies that others need. Therefore unlike the -# `reproduce/config/pipeline/*.mk' Makefiles which only define low-level -# variables (not dependent on other variables and contain no rules), the -# high-level processing Makefiles are included through the `foreach' loop -# below by explicitly requesting them in a specific order here. -include reproduce/config/pipeline/*.mk -include $(foreach f, initialize \ - download \ - delete-me \ - paper \ - , reproduce/src/make/$(f).mk) - - - - - -# LaTeX macros for paper -# ---------------------- -# -# The final report's PDF (final target of this reproduction pipeline) takes -# variable strings from the pipeline. Those variables are defined as LaTeX -# macros in `tex/pipeline.tex'. This file is thus the interface between the -# pipeline scripts and the final PDF. -# -# Each of the pipeline steps will save their macros into their own `.tex' -# file in the `$(mtexdir)' directory. Those individual macros are the -# pre-requisite to `tex/pipeline.txt'. `tex/pipeline.tex' is thus a -# high-level output and is defined in this top-most Makefile (and not -# `reproduce/src/make/paper.mk'). This enables a clear demonstration of the -# top-level dependencies clearly. -# -# Note that if you don't want the final PDF and just want the processing -# and file outputs, you can remove the value of the `pdf-build-final' -# variable in `reproduce/config/pdf-build.mk'. -tex/pipeline.tex: $(foreach f, initialize \ - download \ - delete-me \ - , $(mtexdir)/$(f).tex) - - # If no PDF is requested, or if LaTeX isn't available, don't - # continue to building the final PDF. Otherwise, merge all the TeX - # macros into one for building the PDF. - @if [ -f .local/bin/pdflatex ] && [ x"$(pdf-build-final)" != x ]; then - cat $(mtexdir)/*.tex > $@ - else - echo - echo "-----" - echo "The processing has COMPLETED SUCCESSFULLY! But the final " - echo "LaTeX-built PDF paper will not be built." - echo - if [ x$(more-on-building-pdf) = x1 ]; then - echo "To do so, make sure you have LaTeX within the pipeline (you" - echo "can check by running './.local/bin/latex --version'), _AND_" - echo "make sure that the 'pdf-build-final' variable has a value." - echo "'pdf-build-final' is defined in: " - echo "'reproduce/config/pipeline/pdf-build.mk'." - echo - echo "If you don't have LaTeX within the pipeline, please re-run" - echo "'./configure' when you have internet access. To speed it up," - echo "you can keep the previous configuration files (answer 'n'" - echo "when it asks about re-writing previous configuration files)." - else - echo "For more, run './.local/bin/make more-on-building-pdf=1'" - fi - echo - echo "" > $@ - fi diff --git a/reproduce/src/make/paper.mk b/reproduce/src/make/paper.mk index d50c59a..8e024ee 100644 --- a/reproduce/src/make/paper.mk +++ b/reproduce/src/make/paper.mk @@ -22,6 +22,62 @@ +# LaTeX macros for paper +# ---------------------- +# +# To report the input settings and results, the final report's PDF +# (final target of this reproduction pipeline) uses macros generated +# from various steps of the pipeline. All these macros are defined in +# `tex/pipeline.tex'. +# +# `tex/pipeline.tex' is actually just a combination of separate files +# that keep the LaTeX macros related to each workhorse Makefile (in +# `reproduce/src/make/*.mk'). Those individual macros are +# pre-requisites to `tex/pipeline.tex'. The only workhorse Makefile +# that doesn't need to produce LaTeX macros is this Makefile +# (`reproduce/src/make/paper.mk'). +# +# This file is thus the interface between the pipeline scripts and the +# final PDF: when we get to this point, all the processing has been +# completed. +# +# Note that if you don't want the final PDF and just want the +# processing and file outputs, you can remove the value of +# `pdf-build-final' in `reproduce/config/pipeline/pdf-build.mk'. +tex/pipeline.tex: $(foreach s, $(subst paper,,$(makesrc)), $(mtexdir)/$(s).tex) + + # If no PDF is requested, or if LaTeX isn't available, don't + # continue to building the final PDF. Otherwise, merge all the TeX + # macros into one for building the PDF. + @if [ -f .local/bin/pdflatex ] && [ x"$(pdf-build-final)" != x ]; then + cat $(mtexdir)/*.tex > $@ + else + echo + echo "-----" + echo "The processing has COMPLETED SUCCESSFULLY! But the final " + echo "LaTeX-built PDF paper will not be built." + echo + if [ x$(more-on-building-pdf) = x1 ]; then + echo "To do so, make sure you have LaTeX within the pipeline (you" + echo "can check by running './.local/bin/latex --version'), _AND_" + echo "make sure that the 'pdf-build-final' variable has a value." + echo "'pdf-build-final' is defined in: " + echo "'reproduce/config/pipeline/pdf-build.mk'." + echo + echo "If you don't have LaTeX within the pipeline, please re-run" + echo "'./configure' when you have internet access. To speed it up," + echo "you can keep the previous configuration files (answer 'n'" + echo "when it asks about re-writing previous configuration files)." + else + echo "For more, run './.local/bin/make more-on-building-pdf=1'" + fi + echo + echo "" > $@ + fi + + + + # The bibliography # ---------------- diff --git a/reproduce/src/make/top.mk b/reproduce/src/make/top.mk new file mode 100644 index 0000000..c002bf7 --- /dev/null +++ b/reproduce/src/make/top.mk @@ -0,0 +1,87 @@ +# A ONE-LINE DESCRIPTION OF THE WHOLE PIPELINE +# +# Original author: +# Mohammad Akhlaghi <mohammad@akhlaghi.org> +# Contributing author(s): +# Your name <your@email.address> +# Copyright (C) 2018, Your Name. +# +# This Makefile is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# This Makefile is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# A copy of the GNU General Public License is available at +# <http://www.gnu.org/licenses/>. + + + + + +# Ultimate target of this pipeline +# -------------------------------- +# +# The final paper (in PDF format) is the main target of this whole +# reproduction pipeline. So as defined in the Make paradigm, we are +# defining it here. +# +# Note that if you don't have LaTeX to build the PDF, or generally are just +# interested in the processing, you can skip create the final PDF creation +# with `pdf-build-final' of `reproduce/config/pipeline/pdf-build.mk'. +all: paper.pdf + + + + + +# Define source Makefiles +# ----------------------- +# +# To keep things clean, managable and readable, each set of operations +# is (and must be) classified (modularized) by context into separate +# Makefiles: the more the better. These modular steps are then +# included in this top-level Makefile through the `include' command of +# the next step. Each Makefile should also produce a LaTeX macro file +# with the same fixed name (used to keep all the parameters and +# relevant outputs of the steps in it for the final paper). +# +# In the rare case that no special LaTeX macros are necessary in a +# workhorse Makefile, you can simply make an empty file with `touch +# $@'. This will not add any lines to the final combined LaTeX macros +# file, but will create the file that is a prerequisite to the final +# paper generation. +# +# To (significantly) help in readability, this top-level Makefile should be +# the only one in charge of including Makefiles. So if you care about easy +# maintainence and understandability (even for your self, in one year! It +# is VERY IMPORTANT and as a scientist, you MUST care about it!), do not +# include Makefiles from any other Makefile. +# +# IMPORTANT NOTE: order matters in the inclusion of the processing +# Makefiles. As the pipeline grows, some Makefiles will define +# variables/dependencies that later Makefiles need. Therefore we are using +# a `foreach' loop in the next step to explicitly request loading them in +# the same order that they are defined here (we aren't just using a +# wild-card like the configuration Makefiles). +makesrc = initialize \ + download \ + delete-me \ + paper + + + + + +# Include necessary Makefiles +# --------------------------- +# +# First, we'll include all the configuration-Makefiles (only defining +# variables with no rules or order), then the workhorse Makefiles which +# contain rules and order matters for them. +include reproduce/config/pipeline/*.mk +include $(foreach s,$(makesrc), reproduce/src/make/$(s).mk) |