Rule of tex/pipeline.tex now defined in paper.mk not top Makefile

To avoid redundant steps in the the top-level Makefile and make it simpler
and easier to follow, we now define the base names of all the Makefiles in
the `makesrc' variable of the top-level Makefile. `makesrc' is then used to
define the Makefiles to include and the necessary TeX macros at the same
time. This is much more clear and obvious than the previous case were we
had to list the Makefiles and TeX macro files separately in the top level
Makefile.

1 files changed, 87 insertions, 83 deletions

diff --git a/README-pipeline.md b/README-pipeline.md
index 9ca0cd2..e4feb19 100644
--- a/README-pipeline.md
+++ b/README-pipeline.md
@@ -286,78 +286,81 @@ other servers: its contents are all products of the pipeline, and can be
 easily re-created any time. As you define targets for your new rules, it is
 thus important to place them all under sub-directories of `$(BDIR)`.
 
-Let's start reviewing the processing with the top Makefile. Please open and
-inspect it as we go along here. The first step (un-commented line) defines
-the ultimate target (`paper.pdf`). You shouldn't modify this line. The rule
-to build `paper.pdf` is in another Makefile that will be imported into this
-top Makefile later. Don't forget that Make first scans the Makefile(s) once
-completely (to define dependencies and etc) and starts its execution after
-that. So it is fine to define the rule to build `paper.pdf` at a later
-stage (this is one beauty of Make!).
-
-Having defined the top target, our next step is to include all the other
-necessary Makefiles. First we include all Makefiles that satisfy this
-wildcard: `reproduce/config/pipeline/*.mk`. These Makefiles don't actually
-have any rules, they just have values for various free parameters
-throughout the pipeline. Open a few of them to see for your self. These
+In this architecture, we have two types of Makefiles that are loaded into
+one: _configuration-Makefiles_ (only independent variables/configurations)
+and _workhorse-Makefiles_ (Makefiles that actually contain rules).
+
+The configuration-Makefiles are those that satisfy this wildcard:
+`reproduce/config/pipeline/*.mk`. These Makefiles don't actually have any
+rules, they just have values for various free parameters throughout the
+analysis/processing. Open a few of them to see for your self. These
 Makefiles must only contain raw Make variables (pipeline
 configurations). By raw we mean that the Make variables in these files must
-not depend on variables in any other Makefile. This is because we don't
-want to assume any order in reading them. It is very important to *not*
-define any rule or other Make construct in any of these
-_configuration-Makefiles_ (see the next paragraph for Makefiles with
-rules). This will enable you to set the respective Makefiles in this
-directory as a prerequisite to any target that depends on their variable
-values. Therefore, if you change any of their values, all targets that
-depend on those values will be re-built.
-
-Once all the raw variables have been imported into the top Makefile, we are
-ready to import the Makefiles containing the details of the processing
-steps (Makefiles containing rules, let's call these
-_workhorse-Makefiles_). But in this phase *order is important*, because the
-prerequisites of most rules will be other rules that will be defined at a
-lower level (not a fixed name like `paper.pdf`). The lower-level rules must
-be imported into Make before the higher-level ones. Hence, we can't use a
-simple wildcard like when we imported configuration-Makefiles above. All
-these Makefiles are defined in `reproduce/src/make`, therefore, the top
-Makefile uses the `foreach` function to read them in a specific order.
-
-The main body of this pipeline is thus going to be managed within the
-workhorse-Makefiles that are in `reproduce/src/make`. If you set
-clear-to-understand names for these workhorse-Makefiles and follow the
-convention of the top Makefile that you only include one workhorse-Makefile
-per line, the `foreach` loop of the top Makefile that imports them will
-become very easy to read and understand by eye. This will let you know
-generally which step you are taking before or after another. Projects will
-scale up very fast. Thus if you don't start and continue with a clean and
-robust convention like this, in the end it will become very dirty and hard
-to manage/understand (even for yourself). As a general rule of thumb, break
-your rules into as many logically-similar but independent steps as
-possible.
+not depend on variables in any other configuration-Makefile. This is
+because we don't want to assume any order in reading them. It is very
+important to *not* define any rule or other Make construct in any of these
+configuration-Makefiles. This will enable you to set the respective
+Makefiles in this directory as a prerequisite to any target that depends on
+their variable values. Therefore, if you change any of their values, all
+targets that depend on those values will be re-built.
+
+The workhorse-Makefiles are those within the `reproduce/src/make`
+directory. They contain the details of the processing steps (Makefiles
+containing rules). But in this phase *order is important*, because the
+prerequisites of most rules will be the targets of other rules that will be
+defined prior to them (not a fixed name like `paper.pdf`). The lower-level
+rules must be imported into Make before the higher-level ones. Hence, we
+can't use a simple wildcard like when we imported configuration-Makefiles
+above.
 
 All processing steps are assumed to ultimately (usually after many rules)
 end up in some number, image, figure, or table that are to be included in
-the paper. The writing of the values into the final report is managed
-through separate LaTeX files that only contain macros (a name given to a
-number/string to be used in the LaTeX source, which will be replaced when
-compiling it to the final PDF). So usually the last target in a Makefile is
-a `.tex` file (with the same base-name as the Makefile, but in
-`$(BDIR)/tex/macros`). This intermediate TeX file rule will only contain
-commands to fill the TeX file up with values/names that were done in that
-Makefile. As a result, if the targets in a workhorse-Makefile aren't
-directly a prerequisite of other workhorse-Makefile targets, they should be
-a pre-requisite of that intermediate LaTeX macro file.
-
-In `reproduce/src/make/paper.mk` contains the rule to build `paper.pdf`
-(final target of the whole reproduction pipeline). If look in it, you will
-notice that it depends on `tex/pipeline.tex`. Therefore, last part of the
-top-level `Makefile` is the rule to build
-`tex/pipeline.tex`. `tex/pipeline.tex` is the connection between the
-processing steps of the pipeline, and the creation of the final
-PDF. Therefore, to keep the over-all management clean, the rule to create
-this bridge between the two phases is defined in the top-level `Makefile`.
-
-As you see in the top-level `Makefile`, `tex/pipeline.tex` is only a
+the paper. The writing of these results into the final report/paper is
+managed through separate LaTeX files that only contain macros (a name given
+to a number/string to be used in the LaTeX source, which will be replaced
+when compiling it to the final PDF). So usually the last target in a
+workhorse-Makefile is a `.tex` file (with the same base-name as the
+Makefile, but in `$(BDIR)/tex/macros`). As a result, if the targets in a
+workhorse-Makefile aren't directly a prerequisite of other
+workhorse-Makefile targets, they should be a pre-requisite of that
+intermediate LaTeX macro file. Otherwise, they will be ignored by Make.
+
+Let's see how this design is implemented. When the `./configure` finishes,
+it makes a `Makefile` in the top directory. This Makefile is just a
+symbolic link to `reproduce/src/make/top.mk`. Please open and inspect it as
+we go along here. The first step (un-commented line) defines the ultimate
+target (`paper.pdf`). You shouldn't modify this line. The rule to build
+`paper.pdf` is in `reproduce/src/make/paper.mk` that will be imported into
+this top Makefile later.
+
+Having defined the top target, our next step is to include all the other
+necessary Makefiles. But order matters in the importing of
+workhorse-Makefiles and each must also have a TeX macro file with the same
+base name (without a suffix). Therefore, the next step in the top-level
+Makefile is to define a `makesrc` variable to keep the base names (without
+a `.mk` suffix) of the workhorse-Makefiles that must be imported, in the
+proper order. Having defined `makesrc`, in the next step, we'll just import
+all the configuration-Makefiles with a wildcard and all workhorse-Makefiles
+using a Make `foreach` loop to preserve the order. This finishes the
+general view of the pipeline's implementation.
+
+In short, to keep things modular, readable and managable, follow these
+recommendations: 1) Set clear-to-understand names for the
+configuration-Makefiles, and workhorse-Makefiles, 2) Only import other
+Makefiles from top Makefile. These will let you know/remember generally
+which step you are taking before or after another. Projects will scale up
+very fast. Thus if you don't start and continue with a clean and robust
+convention like this, in the end it will become very dirty and hard to
+manage/understand (even for yourself). As a general rule of thumb, break
+your rules into as many logically-similar but independent steps as
+possible.
+
+The `reproduce/src/make/paper.mk` Makefile must be the final Makefile that
+is included. It ends with the rule to build `paper.pdf` (final target of
+the whole reproduction pipeline). If look in it, you will notice that it
+starts with a rule to create `tex/pipeline.tex`. `tex/pipeline.tex` is the
+connection between the processing/analysis steps of the pipeline, and the
+steps to build the final PDF. As you see, `tex/pipeline.tex` is only a
 merging/concatenation of LaTeX macros defined as the output of each
 high-level processing step (the separate work-horse Makefiles that you
 included).
@@ -372,11 +375,12 @@ through the `\bdir` macro.
 
 During the research, it often happens that you want to test a step that is
 not a prerequisite of any higher-level operation. In such cases, you can
-(temporarily) define the target of that rule as a prerequisite of
-`tex/pipeline.tex`. If your test gives a promising result and you want to
-include it in your research, set it as prerequisites to other rules and
-remove it from the list of prerequisites for `tex/pipeline.tex`. In fact,
-this is how a project is designed to grow in this framework.
+(temporarily) define that processing as a rule in the most relevant
+workhorse-Makefile and set its target as a prerequisite of its TeX
+macro. If your test gives a promising result and you want to include it in
+your research, set it as prerequisites to other rules and remove it from
+the list of prerequisites for TeX macro file. In fact, this is how a
+project is designed to grow in this framework.
 
 
 
@@ -391,8 +395,8 @@ mind are listed below.
  - Define new `reproduce/src/make/XXXXXX.mk` workhorse-Makefile(s) with
    good and human-friendly name(s) replacing `XXXXXX`.
 
- - Add `XXXXXX`, as a new line, to the loop which includes the
-   workhorse-Makefiles in the top-level `Makefile`.
+ - Add `XXXXXX`, as a new line, to the values in `makesrc` of the top-level
+   `Makefile`.
 
  - Do not use any constant numbers (or important names like filter names)
    in the workhorse-Makefiles or paper's LaTeX source. Define such
@@ -402,9 +406,9 @@ mind are listed below.
    the variable defined in it.
 
  - Through any number of intermediate prerequisites, all processing steps
-   should end in (be a prerequisite of)
-   `tex/pipeline.tex`. `tex/pipeline.tex` is the bridge between the
-   processing steps and PDF-building steps.
+   should end in (be a prerequisite of) `tex/pipeline.tex` (defined in
+   `reproduce/src/make/paper.mk`). `tex/pipeline.tex` is the bridge between
+   the processing steps and PDF-building steps.
 
 
 
@@ -498,12 +502,12 @@ advanced in later stages of your work.
 
  - **Title**, **short description** and **author** in source files: In this
      raw skeleton, the title or short description of your project should be
-     added in the following two files: `reproduce/src/make/Top-Makefile`
-     (the first line), and `tex/preamble-header.tex`. In both cases, the
-     texts you should replace are all in capital letters to make them
-     easier to identify. Of course, if you use a different LaTeX method of
-     managing the title and authors, please feel free to use your own
-     methods after finishing this checklist and doing your first commit.
+     added in the following two files: `reproduce/src/make/top.mk` (the
+     first line), and `tex/preamble-header.tex`. In both cases, the texts
+     you should replace are all in capital letters to make them easier to
+     identify. Of course, if you use a different LaTeX method of managing
+     the title and authors, please feel free to use your own methods after
+     finishing this checklist and doing your first commit.
 
  - **Gnuastro**: GNU Astronomy Utilities (Gnuastro) is currently a
      dependency of the pipeline which will be built and used. The main