Rule of tex/pipeline.tex now defined in paper.mk not top Makefile

To avoid redundant steps in the the top-level Makefile and make it simpler
and easier to follow, we now define the base names of all the Makefiles in
the `makesrc' variable of the top-level Makefile. `makesrc' is then used to
define the Makefiles to include and the necessary TeX macros at the same
time. This is much more clear and obvious than the previous case were we
had to list the Makefiles and TeX macro files separately in the top level
Makefile.

5 files changed, 231 insertions, 206 deletions

diff --git a/README-pipeline.md b/README-pipeline.md
index 9ca0cd2..e4feb19 100644
--- a/README-pipeline.md
+++ b/README-pipeline.md
@@ -286,78 +286,81 @@ other servers: its contents are all products of the pipeline, and can be
 easily re-created any time. As you define targets for your new rules, it is
 thus important to place them all under sub-directories of `$(BDIR)`.
 
-Let's start reviewing the processing with the top Makefile. Please open and
-inspect it as we go along here. The first step (un-commented line) defines
-the ultimate target (`paper.pdf`). You shouldn't modify this line. The rule
-to build `paper.pdf` is in another Makefile that will be imported into this
-top Makefile later. Don't forget that Make first scans the Makefile(s) once
-completely (to define dependencies and etc) and starts its execution after
-that. So it is fine to define the rule to build `paper.pdf` at a later
-stage (this is one beauty of Make!).
-
-Having defined the top target, our next step is to include all the other
-necessary Makefiles. First we include all Makefiles that satisfy this
-wildcard: `reproduce/config/pipeline/*.mk`. These Makefiles don't actually
-have any rules, they just have values for various free parameters
-throughout the pipeline. Open a few of them to see for your self. These
+In this architecture, we have two types of Makefiles that are loaded into
+one: _configuration-Makefiles_ (only independent variables/configurations)
+and _workhorse-Makefiles_ (Makefiles that actually contain rules).
+
+The configuration-Makefiles are those that satisfy this wildcard:
+`reproduce/config/pipeline/*.mk`. These Makefiles don't actually have any
+rules, they just have values for various free parameters throughout the
+analysis/processing. Open a few of them to see for your self. These
 Makefiles must only contain raw Make variables (pipeline
 configurations). By raw we mean that the Make variables in these files must
-not depend on variables in any other Makefile. This is because we don't
-want to assume any order in reading them. It is very important to *not*
-define any rule or other Make construct in any of these
-_configuration-Makefiles_ (see the next paragraph for Makefiles with
-rules). This will enable you to set the respective Makefiles in this
-directory as a prerequisite to any target that depends on their variable
-values. Therefore, if you change any of their values, all targets that
-depend on those values will be re-built.
-
-Once all the raw variables have been imported into the top Makefile, we are
-ready to import the Makefiles containing the details of the processing
-steps (Makefiles containing rules, let's call these
-_workhorse-Makefiles_). But in this phase *order is important*, because the
-prerequisites of most rules will be other rules that will be defined at a
-lower level (not a fixed name like `paper.pdf`). The lower-level rules must
-be imported into Make before the higher-level ones. Hence, we can't use a
-simple wildcard like when we imported configuration-Makefiles above. All
-these Makefiles are defined in `reproduce/src/make`, therefore, the top
-Makefile uses the `foreach` function to read them in a specific order.
-
-The main body of this pipeline is thus going to be managed within the
-workhorse-Makefiles that are in `reproduce/src/make`. If you set
-clear-to-understand names for these workhorse-Makefiles and follow the
-convention of the top Makefile that you only include one workhorse-Makefile
-per line, the `foreach` loop of the top Makefile that imports them will
-become very easy to read and understand by eye. This will let you know
-generally which step you are taking before or after another. Projects will
-scale up very fast. Thus if you don't start and continue with a clean and
-robust convention like this, in the end it will become very dirty and hard
-to manage/understand (even for yourself). As a general rule of thumb, break
-your rules into as many logically-similar but independent steps as
-possible.
+not depend on variables in any other configuration-Makefile. This is
+because we don't want to assume any order in reading them. It is very
+important to *not* define any rule or other Make construct in any of these
+configuration-Makefiles. This will enable you to set the respective
+Makefiles in this directory as a prerequisite to any target that depends on
+their variable values. Therefore, if you change any of their values, all
+targets that depend on those values will be re-built.
+
+The workhorse-Makefiles are those within the `reproduce/src/make`
+directory. They contain the details of the processing steps (Makefiles
+containing rules). But in this phase *order is important*, because the
+prerequisites of most rules will be the targets of other rules that will be
+defined prior to them (not a fixed name like `paper.pdf`). The lower-level
+rules must be imported into Make before the higher-level ones. Hence, we
+can't use a simple wildcard like when we imported configuration-Makefiles
+above.
 
 All processing steps are assumed to ultimately (usually after many rules)
 end up in some number, image, figure, or table that are to be included in
-the paper. The writing of the values into the final report is managed
-through separate LaTeX files that only contain macros (a name given to a
-number/string to be used in the LaTeX source, which will be replaced when
-compiling it to the final PDF). So usually the last target in a Makefile is
-a `.tex` file (with the same base-name as the Makefile, but in
-`$(BDIR)/tex/macros`). This intermediate TeX file rule will only contain
-commands to fill the TeX file up with values/names that were done in that
-Makefile. As a result, if the targets in a workhorse-Makefile aren't
-directly a prerequisite of other workhorse-Makefile targets, they should be
-a pre-requisite of that intermediate LaTeX macro file.
-
-In `reproduce/src/make/paper.mk` contains the rule to build `paper.pdf`
-(final target of the whole reproduction pipeline). If look in it, you will
-notice that it depends on `tex/pipeline.tex`. Therefore, last part of the
-top-level `Makefile` is the rule to build
-`tex/pipeline.tex`. `tex/pipeline.tex` is the connection between the
-processing steps of the pipeline, and the creation of the final
-PDF. Therefore, to keep the over-all management clean, the rule to create
-this bridge between the two phases is defined in the top-level `Makefile`.
-
-As you see in the top-level `Makefile`, `tex/pipeline.tex` is only a
+the paper. The writing of these results into the final report/paper is
+managed through separate LaTeX files that only contain macros (a name given
+to a number/string to be used in the LaTeX source, which will be replaced
+when compiling it to the final PDF). So usually the last target in a
+workhorse-Makefile is a `.tex` file (with the same base-name as the
+Makefile, but in `$(BDIR)/tex/macros`). As a result, if the targets in a
+workhorse-Makefile aren't directly a prerequisite of other
+workhorse-Makefile targets, they should be a pre-requisite of that
+intermediate LaTeX macro file. Otherwise, they will be ignored by Make.
+
+Let's see how this design is implemented. When the `./configure` finishes,
+it makes a `Makefile` in the top directory. This Makefile is just a
+symbolic link to `reproduce/src/make/top.mk`. Please open and inspect it as
+we go along here. The first step (un-commented line) defines the ultimate
+target (`paper.pdf`). You shouldn't modify this line. The rule to build
+`paper.pdf` is in `reproduce/src/make/paper.mk` that will be imported into
+this top Makefile later.
+
+Having defined the top target, our next step is to include all the other
+necessary Makefiles. But order matters in the importing of
+workhorse-Makefiles and each must also have a TeX macro file with the same
+base name (without a suffix). Therefore, the next step in the top-level
+Makefile is to define a `makesrc` variable to keep the base names (without
+a `.mk` suffix) of the workhorse-Makefiles that must be imported, in the
+proper order. Having defined `makesrc`, in the next step, we'll just import
+all the configuration-Makefiles with a wildcard and all workhorse-Makefiles
+using a Make `foreach` loop to preserve the order. This finishes the
+general view of the pipeline's implementation.
+
+In short, to keep things modular, readable and managable, follow these
+recommendations: 1) Set clear-to-understand names for the
+configuration-Makefiles, and workhorse-Makefiles, 2) Only import other
+Makefiles from top Makefile. These will let you know/remember generally
+which step you are taking before or after another. Projects will scale up
+very fast. Thus if you don't start and continue with a clean and robust
+convention like this, in the end it will become very dirty and hard to
+manage/understand (even for yourself). As a general rule of thumb, break
+your rules into as many logically-similar but independent steps as
+possible.
+
+The `reproduce/src/make/paper.mk` Makefile must be the final Makefile that
+is included. It ends with the rule to build `paper.pdf` (final target of
+the whole reproduction pipeline). If look in it, you will notice that it
+starts with a rule to create `tex/pipeline.tex`. `tex/pipeline.tex` is the
+connection between the processing/analysis steps of the pipeline, and the
+steps to build the final PDF. As you see, `tex/pipeline.tex` is only a
 merging/concatenation of LaTeX macros defined as the output of each
 high-level processing step (the separate work-horse Makefiles that you
 included).
@@ -372,11 +375,12 @@ through the `\bdir` macro.
 
 During the research, it often happens that you want to test a step that is
 not a prerequisite of any higher-level operation. In such cases, you can
-(temporarily) define the target of that rule as a prerequisite of
-`tex/pipeline.tex`. If your test gives a promising result and you want to
-include it in your research, set it as prerequisites to other rules and
-remove it from the list of prerequisites for `tex/pipeline.tex`. In fact,
-this is how a project is designed to grow in this framework.
+(temporarily) define that processing as a rule in the most relevant
+workhorse-Makefile and set its target as a prerequisite of its TeX
+macro. If your test gives a promising result and you want to include it in
+your research, set it as prerequisites to other rules and remove it from
+the list of prerequisites for TeX macro file. In fact, this is how a
+project is designed to grow in this framework.
 
 
 
@@ -391,8 +395,8 @@ mind are listed below.
  - Define new `reproduce/src/make/XXXXXX.mk` workhorse-Makefile(s) with
    good and human-friendly name(s) replacing `XXXXXX`.
 
- - Add `XXXXXX`, as a new line, to the loop which includes the
-   workhorse-Makefiles in the top-level `Makefile`.
+ - Add `XXXXXX`, as a new line, to the values in `makesrc` of the top-level
+   `Makefile`.
 
  - Do not use any constant numbers (or important names like filter names)
    in the workhorse-Makefiles or paper's LaTeX source. Define such
@@ -402,9 +406,9 @@ mind are listed below.
    the variable defined in it.
 
  - Through any number of intermediate prerequisites, all processing steps
-   should end in (be a prerequisite of)
-   `tex/pipeline.tex`. `tex/pipeline.tex` is the bridge between the
-   processing steps and PDF-building steps.
+   should end in (be a prerequisite of) `tex/pipeline.tex` (defined in
+   `reproduce/src/make/paper.mk`). `tex/pipeline.tex` is the bridge between
+   the processing steps and PDF-building steps.
 
 
 
@@ -498,12 +502,12 @@ advanced in later stages of your work.
 
  - **Title**, **short description** and **author** in source files: In this
      raw skeleton, the title or short description of your project should be
-     added in the following two files: `reproduce/src/make/Top-Makefile`
-     (the first line), and `tex/preamble-header.tex`. In both cases, the
-     texts you should replace are all in capital letters to make them
-     easier to identify. Of course, if you use a different LaTeX method of
-     managing the title and authors, please feel free to use your own
-     methods after finishing this checklist and doing your first commit.
+     added in the following two files: `reproduce/src/make/top.mk` (the
+     first line), and `tex/preamble-header.tex`. In both cases, the texts
+     you should replace are all in capital letters to make them easier to
+     identify. Of course, if you use a different LaTeX method of managing
+     the title and authors, please feel free to use your own methods after
+     finishing this checklist and doing your first commit.
 
  - **Gnuastro**: GNU Astronomy Utilities (Gnuastro) is currently a
      dependency of the pipeline which will be built and used. The main
diff --git a/configure b/configure
index 4701e3a..4753320 100755
--- a/configure
+++ b/configure
@@ -634,7 +634,7 @@ fi
 #
 # To see why this is the last step of the configuration, see above (when we
 # delete the top-level Makefile at the start of this script).
-ln -s $(pwd)/reproduce/src/make/Top-Makefile Makefile
+ln -s $(pwd)/reproduce/src/make/top.mk Makefile
 
 
 
diff --git a/reproduce/src/make/Top-Makefile b/reproduce/src/make/Top-Makefile
deleted file mode 100644
index 4c547be..0000000
--- a/reproduce/src/make/Top-Makefile
+++ /dev/null
@@ -1,122 +0,0 @@
-# A ONE-LINE DESCRIPTION OF THE WHOLE PIPELINE
-#
-# Original author:
-#     Mohammad Akhlaghi <mohammad@akhlaghi.org>
-# Contributing author(s):
-#     Your name <your@email.address>
-# Copyright (C) 2018, Your Name.
-#
-# This Makefile is free software: you can redistribute it and/or modify it
-# under the terms of the GNU General Public License as published by the
-# Free Software Foundation, either version 3 of the License, or (at your
-# option) any later version.
-#
-# This Makefile is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
-# Public License for more details.
-#
-# A copy of the GNU General Public License is available at
-# <http://www.gnu.org/licenses/>.
-
-
-
-
-
-# Ultimate target of this pipeline
-# --------------------------------
-#
-# The final paper (in PDF format) is the main target of this whole
-# reproduction pipeline. So as defined in the Make paradigm, we are
-# defining it here.
-#
-# Note that if you don't have LaTeX to build the PDF, or generally are just
-# interested in the processing, you can skip create the final PDF creation
-# with `pdf-build-final' of `reproduce/config/pipeline/pdf-build.mk'.
-all: paper.pdf
-
-
-
-
-
-# Include specific Makefiles
-# --------------------------
-#
-# To keep things clean, managable and readable, each set of operations is
-# (and must be) classified (modularized) by context into separate
-# Makefiles: the more the better. They are included in this top-level
-# Makefile through the command below.
-#
-# To further help in readability, it is best to avoid including Makefiles
-# within any other Makefile. So in short, it is best that the `foreach'
-# loop below contains all the `reproduce/src/make/*.mk' files.
-#
-# IMPORTANT NOTE: order matters in the inclusion of the processing
-# Makefiles. As the pipeline grows, some Makefiles will probably define
-# variables/dependencies that others need. Therefore unlike the
-# `reproduce/config/pipeline/*.mk' Makefiles which only define low-level
-# variables (not dependent on other variables and contain no rules), the
-# high-level processing Makefiles are included through the `foreach' loop
-# below by explicitly requesting them in a specific order here.
-include reproduce/config/pipeline/*.mk
-include $(foreach f, initialize                    \
-                     download                      \
-                     delete-me                     \
-                     paper                         \
-                   , reproduce/src/make/$(f).mk)
-
-
-
-
-
-# LaTeX macros for paper
-# ----------------------
-#
-# The final report's PDF (final target of this reproduction pipeline) takes
-# variable strings from the pipeline. Those variables are defined as LaTeX
-# macros in `tex/pipeline.tex'. This file is thus the interface between the
-# pipeline scripts and the final PDF.
-#
-# Each of the pipeline steps will save their macros into their own `.tex'
-# file in the `$(mtexdir)' directory. Those individual macros are the
-# pre-requisite to `tex/pipeline.txt'. `tex/pipeline.tex' is thus a
-# high-level output and is defined in this top-most Makefile (and not
-# `reproduce/src/make/paper.mk'). This enables a clear demonstration of the
-# top-level dependencies clearly.
-#
-# Note that if you don't want the final PDF and just want the processing
-# and file outputs, you can remove the value of the `pdf-build-final'
-# variable in `reproduce/config/pdf-build.mk'.
-tex/pipeline.tex: $(foreach f, initialize          \
-                               download            \
-                               delete-me           \
-                             , $(mtexdir)/$(f).tex)
-
-        # If no PDF is requested, or if LaTeX isn't available, don't
-        # continue to building the final PDF. Otherwise, merge all the TeX
-        # macros into one for building the PDF.
-	@if [ -f .local/bin/pdflatex ] && [ x"$(pdf-build-final)" != x ]; then
-	  cat $(mtexdir)/*.tex > $@
-	else
-	  echo
-	  echo "-----"
-	  echo "The processing has COMPLETED SUCCESSFULLY! But the final "
-	  echo "LaTeX-built PDF paper will not be built."
-	  echo
-	  if [ x$(more-on-building-pdf) = x1 ]; then
-	    echo "To do so, make sure you have LaTeX within the pipeline (you"
-	    echo "can check by running './.local/bin/latex --version'), _AND_"
-	    echo "make sure that the 'pdf-build-final' variable has a value."
-	    echo "'pdf-build-final' is defined in: "
-	    echo     "'reproduce/config/pipeline/pdf-build.mk'."
-	    echo
-	    echo "If you don't have LaTeX within the pipeline, please re-run"
-	    echo "'./configure' when you have internet access. To speed it up,"
-	    echo "you can keep the previous configuration files (answer 'n'"
-	    echo "when it asks about re-writing previous configuration files)."
-	  else
-	    echo "For more, run './.local/bin/make more-on-building-pdf=1'"
-	  fi
-	  echo
-	  echo "" > $@
-	fi
diff --git a/reproduce/src/make/paper.mk b/reproduce/src/make/paper.mk
index d50c59a..8e024ee 100644
--- a/reproduce/src/make/paper.mk
+++ b/reproduce/src/make/paper.mk
@@ -22,6 +22,62 @@
 
 
 
+# LaTeX macros for paper
+# ----------------------
+#
+# To report the input settings and results, the final report's PDF
+# (final target of this reproduction pipeline) uses macros generated
+# from various steps of the pipeline. All these macros are defined in
+# `tex/pipeline.tex'.
+#
+# `tex/pipeline.tex' is actually just a combination of separate files
+# that keep the LaTeX macros related to each workhorse Makefile (in
+# `reproduce/src/make/*.mk'). Those individual macros are
+# pre-requisites to `tex/pipeline.tex'. The only workhorse Makefile
+# that doesn't need to produce LaTeX macros is this Makefile
+# (`reproduce/src/make/paper.mk').
+#
+# This file is thus the interface between the pipeline scripts and the
+# final PDF: when we get to this point, all the processing has been
+# completed.
+#
+# Note that if you don't want the final PDF and just want the
+# processing and file outputs, you can remove the value of
+# `pdf-build-final' in `reproduce/config/pipeline/pdf-build.mk'.
+tex/pipeline.tex: $(foreach s, $(subst paper,,$(makesrc)), $(mtexdir)/$(s).tex)
+
+        # If no PDF is requested, or if LaTeX isn't available, don't
+        # continue to building the final PDF. Otherwise, merge all the TeX
+        # macros into one for building the PDF.
+	@if [ -f .local/bin/pdflatex ] && [ x"$(pdf-build-final)" != x ]; then
+	  cat $(mtexdir)/*.tex > $@
+	else
+	  echo
+	  echo "-----"
+	  echo "The processing has COMPLETED SUCCESSFULLY! But the final "
+	  echo "LaTeX-built PDF paper will not be built."
+	  echo
+	  if [ x$(more-on-building-pdf) = x1 ]; then
+	    echo "To do so, make sure you have LaTeX within the pipeline (you"
+	    echo "can check by running './.local/bin/latex --version'), _AND_"
+	    echo "make sure that the 'pdf-build-final' variable has a value."
+	    echo "'pdf-build-final' is defined in: "
+	    echo     "'reproduce/config/pipeline/pdf-build.mk'."
+	    echo
+	    echo "If you don't have LaTeX within the pipeline, please re-run"
+	    echo "'./configure' when you have internet access. To speed it up,"
+	    echo "you can keep the previous configuration files (answer 'n'"
+	    echo "when it asks about re-writing previous configuration files)."
+	  else
+	    echo "For more, run './.local/bin/make more-on-building-pdf=1'"
+	  fi
+	  echo
+	  echo "" > $@
+	fi
+
+
+
+
 
 # The bibliography
 # ----------------
diff --git a/reproduce/src/make/top.mk b/reproduce/src/make/top.mk
new file mode 100644
index 0000000..c002bf7
--- /dev/null
+++ b/reproduce/src/make/top.mk
@@ -0,0 +1,87 @@
+# A ONE-LINE DESCRIPTION OF THE WHOLE PIPELINE
+#
+# Original author:
+#     Mohammad Akhlaghi <mohammad@akhlaghi.org>
+# Contributing author(s):
+#     Your name <your@email.address>
+# Copyright (C) 2018, Your Name.
+#
+# This Makefile is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This Makefile is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.
+#
+# A copy of the GNU General Public License is available at
+# <http://www.gnu.org/licenses/>.
+
+
+
+
+
+# Ultimate target of this pipeline
+# --------------------------------
+#
+# The final paper (in PDF format) is the main target of this whole
+# reproduction pipeline. So as defined in the Make paradigm, we are
+# defining it here.
+#
+# Note that if you don't have LaTeX to build the PDF, or generally are just
+# interested in the processing, you can skip create the final PDF creation
+# with `pdf-build-final' of `reproduce/config/pipeline/pdf-build.mk'.
+all: paper.pdf
+
+
+
+
+
+# Define source Makefiles
+# -----------------------
+#
+# To keep things clean, managable and readable, each set of operations
+# is (and must be) classified (modularized) by context into separate
+# Makefiles: the more the better. These modular steps are then
+# included in this top-level Makefile through the `include' command of
+# the next step. Each Makefile should also produce a LaTeX macro file
+# with the same fixed name (used to keep all the parameters and
+# relevant outputs of the steps in it for the final paper).
+#
+# In the rare case that no special LaTeX macros are necessary in a
+# workhorse Makefile, you can simply make an empty file with `touch
+# $@'. This will not add any lines to the final combined LaTeX macros
+# file, but will create the file that is a prerequisite to the final
+# paper generation.
+#
+# To (significantly) help in readability, this top-level Makefile should be
+# the only one in charge of including Makefiles. So if you care about easy
+# maintainence and understandability (even for your self, in one year! It
+# is VERY IMPORTANT and as a scientist, you MUST care about it!), do not
+# include Makefiles from any other Makefile.
+#
+# IMPORTANT NOTE: order matters in the inclusion of the processing
+# Makefiles. As the pipeline grows, some Makefiles will define
+# variables/dependencies that later Makefiles need. Therefore we are using
+# a `foreach' loop in the next step to explicitly request loading them in
+# the same order that they are defined here (we aren't just using a
+# wild-card like the configuration Makefiles).
+makesrc = initialize                    \
+          download                      \
+          delete-me                     \
+          paper
+
+
+
+
+
+# Include necessary Makefiles
+# ---------------------------
+#
+# First, we'll include all the configuration-Makefiles (only defining
+# variables with no rules or order), then the workhorse Makefiles which
+# contain rules and order matters for them.
+include reproduce/config/pipeline/*.mk
+include $(foreach s,$(makesrc), reproduce/src/make/$(s).mk)