Better management for .tex directories to build from tarball

In order to collaborate effectively in the project, even project members
that don't necessarily want (or have the capacity) to do the whole analysis
must be able to contribute to the project. Until now, the users of the
distributed tarball could only modify the text and not the figures (built
with PGFPlots) of the paper.

With this commit, the management of TeX source files in the pipeline was
slightly modified to allow this as cleanly as I could think of now! In
short, the hand-written TeX files are now kept in `tex/src' and for the
pipeline's generated TeX files (in particular the old `tex/pipeline.tex'),
we now have a `tex/pipeline' symbolic-link/directory that points to the
`tex' directory under the build directory.

When packaging the project, `tex/pipeline' will be a full directory with a
copy of all the necessary files. Therefore as far as LaTeX is concerned,
having a build-directory is no longer relevant. Many other small changes
were made to do this job cleanly which will just make this commit message
too long!

Also, the old `tarball' and `zip' targets are now `dist' and `dist-zip' (as
in the standard GNU Build system).

14 files changed, 120 insertions, 85 deletions

diff --git a/.file-metadata b/.file-metadata
index 7ce229a..fe72296 100644
--- a/.file-metadata
+++ b/.file-metadata
diff --git a/.gitignore b/.gitignore
index 6338a5b..7074d37 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,10 +16,10 @@ Makefile
 tex/tikz
 .gnuastro
 .texlive*
+tex/pipeline
 LOCAL_tmp.mk
 LOCAL_old.mk
 reproduce/build
-tex/pipeline.tex
 gnuastro-local.conf
 reproduce/config/pipeline/LOCAL.mk
 
diff --git a/README-hacking.md b/README-hacking.md
index ddc9460..073138b 100644
--- a/README-hacking.md
+++ b/README-hacking.md
@@ -388,20 +388,13 @@ possible.
 The `reproduce/src/make/paper.mk` Makefile must be the final Makefile that
 is included. It ends with the rule to build `paper.pdf` (final target of
 the whole reproduction pipeline). If look in it, you will notice that it
-starts with a rule to create `tex/pipeline.tex`. `tex/pipeline.tex` is the
-connection between the processing/analysis steps of the pipeline, and the
-steps to build the final PDF. As you see, `tex/pipeline.tex` is only a
-merging/concatenation of LaTeX macros defined as the output of each
-high-level processing step (the separate work-horse Makefiles that you
-included).
-
-One of the LaTeX macros created by `reproduce/src/make/initialize.mk` is
-`\bdir`. It is the location of the build directory. In some cases you want
-tables and images to also be included in the final PDF. To keep these
-necessary LaTeX inputs, you can define other directories under
-`$(BDIR)/tex` in the relevant workhorse-Makefile. You can then easily guide
-LaTeX to look into the proper directory to import an image for example
-through the `\bdir` macro.
+starts with a rule to create `$(mtexdir)/pipeline.tex` (`mtexdir` is just a
+shorthand name for `$(BDIR)/tex/macros` mentioned before).
+`$(mtexdir)/pipeline.tex` is the connection between the processing/analysis
+steps of the pipeline, and the steps to build the final PDF. As you see,
+`$(mtexdir)/pipeline.tex` only instruct LaTeX to import the LaTeX macros of
+each high-level processing step during the analysis (the separate
+work-horse Makefiles that you defined and included).
 
 During the research, it often happens that you want to test a step that is
 not a prerequisite of any higher-level operation. In such cases, you can
@@ -446,9 +439,9 @@ mind are listed below.
    the variable defined in it.
 
  - Through any number of intermediate prerequisites, all processing steps
-   should end in (be a prerequisite of) `tex/pipeline.tex` (defined in
-   `reproduce/src/make/paper.mk`). `tex/pipeline.tex` is the bridge between
-   the processing steps and PDF-building steps.
+   should end in (be a prerequisite of) `$(mtexdir)/pipeline.tex` (defined
+   in `reproduce/src/make/paper.mk`). `$(mtexdir)/pipeline.tex` is the
+   bridge between the processing steps and PDF-building steps.
 
 
 
diff --git a/paper.tex b/paper.tex
index 19c202d..a36b3eb 100644
--- a/paper.tex
+++ b/paper.tex
@@ -17,12 +17,11 @@
 %% Necessary LaTeX preambles to include for relevant functionality. We want
 %% to start this file as fast as possible with the actual body of the
 %% paper, while keeping modularity in the preambles.
-\input{tex/pipeline.tex}
-\input{tex/preamble-style.tex}
-\input{tex/preamble-header.tex}
-\input{tex/preamble-biblatex.tex}
-\input{tex/preamble-pgfplots.tex}
-\input{tex/preamble-necessary.tex}
+\input{tex/src/preamble-style.tex}
+\input{tex/src/preamble-header.tex}
+\input{tex/src/preamble-biblatex.tex}
+\input{tex/src/preamble-pgfplots.tex}
+\input{tex/src/preamble-necessary.tex}
 
 
 
@@ -75,10 +74,11 @@ Just don't forget to \emph{never} use numbers or fixed strings (for example
 database urls like \url{\wfpctwourl}) directly within your \LaTeX{}
 source. Read them directly from your configuration files or outputs of the
 programs as part of the reproduction pipeline and import them into \LaTeX{}
-as macros through the \texttt{tex/pipeline.tex} file. See the several
-examples within the pipeline for a demonstration. For some recent
-real-world examples, the reproduction pipelines for Sections 4 and 7.3 of
-\citet{bacon17} are available at
+as macros through the \texttt{tex/pipeline/macros/pipeline.tex} file
+(created after running the pipeline). See the several examples within the
+pipeline for a demonstration. For some recent real-world examples, the
+reproduction pipelines for Sections 4 and 7.3 of \citet{bacon17} are
+available at
 \href{https://doi.org/10.5281/zenodo.1164774}{zenodo.1164774}\footnote{\url{https://gitlab.com/makhlaghi/muse-udf-origin-only-hst-magnitudes}},
 or
 \href{https://doi.org/10.5281/zenodo.1163746}{zenodo.1163746}\footnote{\url{https://gitlab.com/makhlaghi/muse-udf-photometry-astrometry}}. Working
diff --git a/reproduce/src/make/initialize.mk b/reproduce/src/make/initialize.mk
index b3180e3..ff06dcb 100644
--- a/reproduce/src/make/initialize.mk
+++ b/reproduce/src/make/initialize.mk
@@ -140,9 +140,9 @@ export CPPFLAGS        := -I$(installdir)/include
 # option: they add too many extra checks that make it hard to find what you
 # are looking for in this pipeline.
 .SUFFIXES:
-$(tikzdir): | $(texbdir); mkdir $@
 $(texdir) $(lockdir): | $(BDIR); mkdir $@
 $(mtexdir) $(texbdir): | $(texdir); mkdir $@
+$(tikzdir): | $(texbdir); mkdir $@ && ln -s $(tikzdir) tex/tikz
 
 
 
@@ -160,7 +160,7 @@ $(mtexdir) $(texbdir): | $(texdir); mkdir $@
 # when no file actually differs.
 packagebasename := $(shell echo paper-$$(git describe --dirty --always))
 packagecontents = $(texdir)/$(packagebasename)
-.PHONY: all clean distclean clean-mmap tarball zip $(packagecontents) \
+.PHONY: all clean dist dist-zip distclean clean-mmap $(packagecontents) \
         $(mtexdir)/initialize.tex
 
 # --------- Delete for no Gnuastro ---------
@@ -206,32 +206,67 @@ $(packagecontents): | $(texdir)
 	rm -rf $$dir
 	mkdir $$dir
 
+        # Build a small Makefile to help in automatizing the paper building
+        # (including the bibliography).
+	m=$$dir/Makefile
+	echo   "paper.pdf: paper.tex paper.bbl"                   > $$m
+	printf "\tpdflatex -shell-escape -halt-on-error paper\n" >> $$m
+	echo   "paper.bbl: tex/src/references.tex"               >> $$m
+	printf "\tpdflatex -shell-escape -halt-on-error paper\n" >> $$m
+	printf "\tbiber paper\n"                                 >> $$m
+	echo   ".PHONY: clean"                                   >> $$m
+	echo   "clean:"                                          >> $$m
+	printf "\trm -f *.aux *.auxlock *.bbl *.bcf\n"           >> $$m
+	printf "\trm -f *.blg *.log *.out *.run.xml\n"           >> $$m
+
         # Copy the top-level contents into it.
 	cp configure COPYING for-group README.md README-hacking.md $$dir/
 
-        # Since the tarball is mainly intended for high-level building of
-        # the PDF with LaTeX, we'll comment the `makepdf' LaTeX macro in
-        # the paper.
-	sed -e's|\\newcommand{\\makepdf}{}|%\\newcommand{\\makepdf}{}|' \
-	    paper.tex > $$dir/paper.tex
+        # Build the top-level directories.
+	mkdir $$dir/reproduce $$dir/tex $$dir/tex/tikz $$dir/tex/pipeline
 
         # Copy all the `reproduce' contents except for the `build' symbolic
         # link.
 	shopt -s extglob
-	mkdir $$dir/reproduce $$dir/tex $$dir/tex/tikz
-	cp tex/*.tex             $$dir/tex
-	cp -r reproduce/!(build) $$dir/reproduce
-	cp tex/tikz/*.pdf        $$dir/tex/tikz
+	cp -r tex/src                            $$dir/tex/src
+	cp tex/tikz/*.pdf                        $$dir/tex/tikz
+	cp -r reproduce/!(build)                 $$dir/reproduce
+	cp -r tex/pipeline/!($(packagebasename)) $$dir/tex/pipeline
+
+        # Clean up un-necessary/local files: 1) the $(texdir)/build*
+        # directories (when building in a group structure, there will be
+        # `build-user1', `build-user2' and etc), are just temporary LaTeX
+        # build files and don't have any relevant/hand-written files in
+        # them. 2) The `LOCAL.mk' and `gnuastro-local.conf' files just have
+        # this machine's local settings and are irrelevant for anyone else.
+	rm -rf $$dir/tex/pipeline/build*
+	rm $$dir/reproduce/config/pipeline/LOCAL.mk
+	rm $$dir/reproduce/config/gnuastro/gnuastro-local.conf
+
+        # PIPELINE SPECIFIC: under this comment, copy any other file for
+        # packaging, or remove any of the copied files above to suite your
+        # project.
+
+        # Since the packaging is mainly intended for high-level building of
+        # the PDF with LaTeX, we'll comment the `makepdf' LaTeX macro in
+        # the paper.
+	sed -e's|\\newcommand{\\makepdf}{}|%\\newcommand{\\makepdf}{}|' \
+	    paper.tex > $$dir/paper.tex
 
-        # PIPELINE SPECIFIC: add or remove any of the copied files above,
-        # specific to your pipeline here.
+        # Just in case the package users want to rebuild some of the
+        # figures (manually un-comments the `makepdf' command we commented
+        # above), correct the TikZ external directory, so the figures can
+        # be rebuilt.
+	pgfsettings="$$dir/tex/src/preamble-pgfplots.tex"
+	sed -e's|{tikz/}|{tex/tikz/}|' $$pgfsettings > $$pgfsettings.new
+	mv $$pgfsettings.new $$pgfsettings
 
-        # Clean all temporary files.
+        # Clean temporary (currently those ending in `~') files.
 	cd $(texdir)
 	find $(packagebasename) -name \*~ -delete
 
 # Package into `.tar.gz'.
-tarball: $(packagecontents)
+dist: $(packagecontents)
 	curdir=$$(pwd)
 	cd $(texdir)
 	tar -cf $(packagebasename).tar $(packagebasename)
@@ -240,7 +275,7 @@ tarball: $(packagecontents)
 	mv $(texdir)/$(packagebasename).tar.gz ./
 
 # Package into `.zip'.
-zip: $(packagecontents)
+dist-zip: $(packagecontents)
 	curdir=$$(pwd)
 	cd $(texdir)
 	zip -q -r $(packagebasename).zip $(packagebasename)
@@ -294,7 +329,6 @@ $(mtexdir)/initialize.tex: | $(mtexdir)
         # Version of the pipeline and build directory (for LaTeX inputs).
 	@v=$$(git describe --dirty --always);
 	echo "\newcommand{\pipelineversion}{$$v}"  > $@
-	@echo "\newcommand{\bdir}{$(BDIR)}"       >> $@
 
         # Versions of programs (same order as 'dependency-versions.mk'),
         # ordered alphabetically (by their executable name).
diff --git a/reproduce/src/make/paper.mk b/reproduce/src/make/paper.mk
index aea6c60..d148c08 100644
--- a/reproduce/src/make/paper.mk
+++ b/reproduce/src/make/paper.mk
@@ -25,17 +25,16 @@
 # LaTeX macros for paper
 # ----------------------
 #
-# To report the input settings and results, the final report's PDF
-# (final target of this reproduction pipeline) uses macros generated
-# from various steps of the pipeline. All these macros are defined in
-# `tex/pipeline.tex'.
+# To report the input settings and results, the final report's PDF (final
+# target of this reproduction pipeline) uses macros generated from various
+# steps of the pipeline. All these macros are defined in
+# `$(mtexdir)/pipeline.tex'.
 #
-# `tex/pipeline.tex' is actually just a combination of separate files
-# that keep the LaTeX macros related to each workhorse Makefile (in
-# `reproduce/src/make/*.mk'). Those individual macros are
-# pre-requisites to `tex/pipeline.tex'. The only workhorse Makefile
-# that doesn't need to produce LaTeX macros is this Makefile
-# (`reproduce/src/make/paper.mk').
+# `$(mtexdir)/pipeline.tex' is actually just a combination of separate
+# files that keep the LaTeX macros related to each workhorse Makefile (in
+# `reproduce/src/make/*.mk'). Those individual macros are pre-requisites to
+# `$(mtexdir)/pipeline.tex'. The only workhorse Makefile that doesn't need
+# to produce LaTeX macros is this Makefile (`reproduce/src/make/paper.mk').
 #
 # This file is thus the interface between the pipeline scripts and the
 # final PDF: when we get to this point, all the processing has been
@@ -44,13 +43,21 @@
 # Note that if you don't want the final PDF and just want the
 # processing and file outputs, you can remove the value of
 # `pdf-build-final' in `reproduce/config/pipeline/pdf-build.mk'.
-tex/pipeline.tex: $(foreach s, $(subst paper,,$(makesrc)), $(mtexdir)/$(s).tex)
+$(mtexdir)/pipeline.tex: $(foreach s, $(subst paper,,$(makesrc)), $(mtexdir)/$(s).tex)
 
         # If no PDF is requested, or if LaTeX isn't available, don't
         # continue to building the final PDF. Otherwise, merge all the TeX
         # macros into one for building the PDF.
 	@if [ -f .local/bin/pdflatex ] && [ x"$(pdf-build-final)" != x ]; then
-	  cat $(mtexdir)/*.tex > $@
+
+          # First make sure the `tex/pipeline' symbolic link exists.
+	  if [ ! -e tex/pipeline ]; then ln -s $(texdir) tex/pipeline; fi
+
+          # Put a LaTeX input command for all the necessary macro files.
+	  rm -f $(mtexdir)/pipeline.tex
+	  for t in $(subst paper,,$(makesrc)); do
+	    echo "\input{tex/pipeline/macros/$$t.tex}" >> $(mtexdir)/pipeline.tex
+	  done
 	else
 	  echo
 	  echo "-----"
@@ -86,15 +93,15 @@ tex/pipeline.tex: $(foreach s, $(subst paper,,$(makesrc)), $(mtexdir)/$(s).tex)
 # necessary bibliography before making the final paper. So we'll first have
 # one run of LaTeX (similar to the `paper.pdf' recipe), then `biber'.
 #
-# NOTE: `tex/pipeline.tex' is an order-only-prerequisite for
+# NOTE: `$(mtexdir)/pipeline.tex' is an order-only-prerequisite for
 # `paper.bbl'. This is because we need to run LaTeX in both the `paper.bbl'
-# recipe and the `paper.pdf' recipe. But if `tex/references.tex' hasn't
+# recipe and the `paper.pdf' recipe. But if `tex/src/references.tex' hasn't
 # been modified, we don't want to re-build the bibliography, only the final
 # PDF.
-$(texbdir)/paper.bbl: tex/references.tex                         \
-                      | $(tikzdir) $(texbdir) tex/pipeline.tex
-        # If `tex/pipeline.tex' is empty, then the PDF must not be built.
-	@macros=$$(cat tex/pipeline.tex)
+$(texbdir)/paper.bbl: tex/src/references.tex                         \
+                      | $(tikzdir) $(texbdir) $(mtexdir)/pipeline.tex
+        # If `$(mtexdir)/pipeline.tex' is empty, don't build PDF.
+	@macros=$$(cat $(mtexdir)/pipeline.tex)
 	if [ x"$$macros" != x ]; then
 
           # We'll run LaTeX first to generate the `.bcf' file (necessary
@@ -119,19 +126,13 @@ $(texbdir)/paper.bbl: tex/references.tex                         \
 # to run everything cleanly from there, it is necessary to add the current
 # directory (top reproduction pipeline directory) to the `TEXINPUTS'
 # environment variable.
-paper.pdf: tex/pipeline.tex paper.tex $(texbdir)/paper.bbl       \
+paper.pdf: $(mtexdir)/pipeline.tex paper.tex $(texbdir)/paper.bbl       \
 	   | $(tikzdir) $(texbdir)
 
-        # If `tex/pipeline.tex' is empty, then the PDF must not be built.
-	@macros=$$(cat tex/pipeline.tex)
+        # If `$(mtexdir)/pipeline.tex' is empty, don't build the PDF.
+	@macros=$$(cat $(mtexdir)/pipeline.tex)
 	if [ x"$$macros" != x ]; then
 
-          # If it doesn't exist, make the `tex/tikz' symbolic link
-          # directory. This is necessary for situations when we just want
-          # TeX to use built figures instead of actually building the
-          # figures.
-	  if [ ! -e tex/tikz ]; then ln -s $(tikzdir) tex/tikz; fi
-
           # Go into the top TeX build directory and make the paper.
 	  p=$$(pwd)
 	  export TEXINPUTS=$$p:$$TEXINPUTS
diff --git a/tex/delete-me-demo.tex b/tex/src/delete-me-demo.tex
index 65aa1c7..8174bc8 100644
--- a/tex/delete-me-demo.tex
+++ b/tex/src/delete-me-demo.tex
@@ -3,7 +3,7 @@
   %% The displayed WFPC2 image.
   \node[anchor=south west] (img) at (0,0)
        {\includegraphics[width=0.5\linewidth]
-         {\bdir/tex/delete-me-demo/wfpc2.pdf}};
+         {tex/pipeline/delete-me-demo/wfpc2.pdf}};
 
   %% Its label
   \node[anchor=south west] at (0.45\linewidth,0.45\linewidth)
@@ -25,7 +25,7 @@
       ]
     \addplot [const plot mark mid, fill=red]
     table [x index=0, y index=1]
-    {\bdir/tex/delete-me-demo/wfpc2-hist.txt}
+    {tex/pipeline/delete-me-demo/wfpc2-hist.txt}
     \closedcycle;
   \end{axis}
 
diff --git a/tex/delete-me.tex b/tex/src/delete-me.tex
index 32be11c..fc1a1f9 100644
--- a/tex/delete-me.tex
+++ b/tex/src/delete-me.tex
@@ -11,7 +11,7 @@
 
     %% A particular plot.
     \addplot+[scatter, only marks]
-    table {\bdir/tex/delete-me/data.txt};
+    table {tex/pipeline/delete-me/data.txt};
 
   \end{axis}
 \end{tikzpicture}
diff --git a/tex/preamble-biblatex.tex b/tex/src/preamble-biblatex.tex
index b0660a9..bd99502 100644
--- a/tex/preamble-biblatex.tex
+++ b/tex/src/preamble-biblatex.tex
@@ -7,7 +7,7 @@
 %% just be copied there and used.
 %%
 %% USAGE:
-%%  - `tex/references.tex': the file containing Bibtex source of each
+%%  - `tex/src/references.tex': the file containing Bibtex source of each
 %%     reference. The file suffix doesn't have to be `.bib'. This naming
 %%     helps in clearly identifying the files and avoiding places that
 %%     complain about `.bib' files.
@@ -41,7 +41,7 @@
     backend=biber,natbib]{biblatex}
 \DeclareFieldFormat[article]{pages}{#1}
 \DeclareFieldFormat{pages}{\mkfirstpage[{\mkpageprefix[bookpagination]}]{#1}}
-\addbibresource{tex/references.tex}
+\addbibresource{tex/src/references.tex}
 \renewbibmacro{in:}{}
 \renewcommand*{\bibfont}{\footnotesize}
 \DefineBibliographyStrings{english}{references = {References}}
diff --git a/tex/preamble-header.tex b/tex/src/preamble-header.tex
index eb83c27..eb83c27 100644
--- a/tex/preamble-header.tex
+++ b/tex/src/preamble-header.tex
diff --git a/tex/preamble-necessary.tex b/tex/src/preamble-necessary.tex
index bef5a2f..e6ac18e 100644
--- a/tex/preamble-necessary.tex
+++ b/tex/src/preamble-necessary.tex
@@ -1,4 +1,4 @@
-%% Some (commonly) necessary LaTeX packages.
+%% Necessary (independent of style) macros for this project.
 %%
 %% These are a set of packages that have been commonly necessary in most
 %% LaTeX usages. However, if any are not needed in your work, please feel
@@ -8,6 +8,13 @@
 
 
 
+%% Values from the analysis.
+\input{tex/pipeline/macros/pipeline.tex}
+
+
+
+
+
 % Macros for to help in typing, remove them if you don't need them, but
 % this can help as a demo on how you can simply writing of commonly used
 % words that need special formatting (like software names).
diff --git a/tex/preamble-pgfplots.tex b/tex/src/preamble-pgfplots.tex
index 26b98d6..4999656 100644
--- a/tex/preamble-pgfplots.tex
+++ b/tex/src/preamble-pgfplots.tex
@@ -33,12 +33,12 @@
 %%    present in the location LaTeX is run).
 %%
 %%  - Use `\includetikz{XXXX}' to make/use the figure. If a `makepdf' LaTeX
-%%    macro is not defined, then it will simply assume a `XXXX.pdf' file
-%%    exists in the `\bdir/tex/build/tikz' directory and simply import
-%%    it. If `makepdf' is defined, then TiKZ/PGFPlot will be called to
-%%    (possibly) build the plot based on `tex/XXXX.tex'. Note that if the
-%%    contents of `tex/XXXX.tex' hasn't changed since the las
-%%    build. TiKZ/PGFPlots won't rebuild the plot.
+%%    macro is not defined, then \includetikz will assume a `XXXX.pdf' file
+%%    exists in `tex/tikz' and simply import it. If `makepdf' is defined,
+%%    then TiKZ/PGFPlot will be called to (possibly) build the plot based
+%%    on `tex/XXXX.tex'. Note that if the contents of `tex/src/XXXX.tex'
+%%    hasn't changed since the last build. TiKZ/PGFPlots won't rebuild the
+%%    plot.
 
 
 
@@ -69,7 +69,7 @@
 \newcommand{\includetikz}[1]{%
   \ifdefined\makepdf%
     \tikzsetnextfilename{#1}%
-    \input{tex/#1.tex}%
+    \input{tex/src/#1.tex}%
   \else
     \includegraphics[width=\linewidth]{tex/tikz/#1.pdf}
   \fi
diff --git a/tex/preamble-style.tex b/tex/src/preamble-style.tex
index fb08df7..fb08df7 100644
--- a/tex/preamble-style.tex
+++ b/tex/src/preamble-style.tex
diff --git a/tex/references.tex b/tex/src/references.tex
index b3998a2..b3998a2 100644
--- a/tex/references.tex
+++ b/tex/src/references.tex