<!DOCTYPE html>
<!-- Copyright notes are just below the head and before body -->
<html lang="en-US">
<!-- HTML Header -->
<head>
<!-- Title of the page. -->
<title>Maneage -- Managing data lineage</title>
<!-- Enable UTF-8 encoding to easily use non-ASCII charactes -->
<meta charset="UTF-8">
<meta http-equiv="Content-type" content="text/html; charset=UTF-8">
<!-- Put logo beside the address bar -->
<link rel="shortcut icon" href="./img/favicon.svg" />
<!-- The viewport meta tag is placed mainly for mobile browsers
that are pre-configured in different ways (for example setting the
different widths for the page than the actual width of the device,
or zooming to different values. Without this the CSS media
solutions might not work properly on all mobile browsers.-->
<meta name="viewport"
content="width=device-width, initial-scale=1">
<!-- Basic styles -->
<link rel="stylesheet" href="css/base.css" />
</head>
<!--
Webpage of Maneage: a framework for managing data lineage
Copyright (C) 2020, Pedram Ashofteh Ardakani <pedramardakani@pm.me>
Copyright (C) 2020, Mohammad Akhlaghi <mohammad@akhlaghi.org>
This file is part of Maneage. Maneage is free software: you can
redistribute it and/or modify it under the terms of the GNU General
Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any later version.
Maneage is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details. See
<http://www.gnu.org/licenses/>. -->
<!-- Start the main body. -->
<body>
<div id="container">
<header role="banner">
<!-- global navigation -->
<nav role="navigation" id="nav-hamburger-wrapper">
<input type="checkbox" id="nav-hamburger-input"/>
<label for="nav-hamburger-input">|||</label>
<div id="nav-hamburger-items" class="button">
<a href="index.html">Home</a>
<a href="about.html">About</a>
<a href="http://git.maneage.org/project.git/">Git</a>
<a href="tutorial.html">Tutorial</a>
</div>
</nav>
</header>
<div class="banner">
<div>
<a href="index.html"><img src="img/maneage-logo.svg" /></a>
</div>
<div>
<h1>Maneage</h1><h2>Tutorial</h2>
<p>Copyright © 2020 Raul Infante-Sainz <a href="mailto:infantesainz@gmail.com">infantesainz@gmail.com</a><br />
Copyright © 2020 Mohammad Akhlaghi <a href="mailto:mohammad@akhlaghi.org">mohammad@akhlaghi.org</a><br />
<a href="#page-footer">License Conditions</a></p>
</div>
</div>
<p>This document is a tutorial in which it is described how <code>Maneage</code>
(management + lineage) works in practice. It is highly recommended to read
the <code>README-hacking.md</code> in order to have a clear idea of what is this
project about. Actually, in this tutorial it is assumed you have the project
already set up and working properly. In order to do it, please, read and
follow all the steps described in the sections <code>Customization checklist</code> up
to the section <code>Title, short description and author</code> (including the last
one).</p>
<p>With the current tutorial, the reader will be able to have a fully
reproducible paper describing a small research example carried out step by
step. The research example is very simple: it will consist in analyse a
dataset with two columns (time and population). The analysis will be just to
make a linear fitting of the data, and then, write the results in a small
paragraph into the final paper.</p>
<p>In the following, the tutorial assume you have three different directories.
You had to set up them in the configure step:</p>
<ul>
<li><p><code>input-directory</code>: Necessary input data for the project is in this
directory.</p></li>
<li><p><code>project-directory</code>: This directory contains the project itself (source
codes), it is under <code>Git</code> control.</p></li>
<li><p><code>build-directory</code>: Output directory of the project, it is where all the
necessary software and the results of the project are saved.</p></li>
</ul>
<p><strong><em>IMPORTANT NOTE</em></strong>: the tutorial assume you are always in
<code>project-directory</code> when considering command lines.</p>
<p><strong>In short:</strong> this hands on tutorial will guide you through a simple
research example in order to show the workflow in <code>Maneage</code>. The tutorial
describes by step how to download a small file containg data, analyse the
data (by making a linear fitting), and finally write a small paragraph with
the fitting parameters into the final paper. All of this will be done in the
same Makefile.</p>
<h2>Installing available software: Matplotlib</h2>
<p>If all steps above have been done successfully, you are ready to start
including your own analysis scripts. But, before that, let's install
<code>Matplotlib</code> Python package, which will be used later in the analysis of the
data when obtaining the linear fit figure. This Python package will be used
as an example on how to install programs that are already available in
<code>Maneage</code>. Just open the Makefile
<code>reproduce/software/config/installation/TARGETS.mk</code> and add to the
<code>top-level-python</code> line, the word <code>matplotlib</code>.</p>
<pre><code><span class="comment"># Python libraries/modules.</span>
top-level-python = astropy matplotlib</code></pre>
<p>After that, run the configure step again with the option <code>-e</code> to continue
using the same configuration options given before (input and build
directories). Also, run the prepare and make steps:</p>
<pre><code>./project configure -e
./project prepare
./project make</code></pre>
<p>Open 'paper.pdf' and see if everything is fine. Note that now, <code>Matplotlib</code>
is appearing in the software appendix at the end of the document.</p>
<p>Once you have verified that <code>Matplotlib</code> has been properly installed and it
appears into the final <code>paper.pdf</code>, you are ready to make the first commit
of the project. With the next commands, you will see which files have been
modified, what are the modifications, prepare them to be commited, and make
the commit. In the commit process, <code>Git</code> will open the text editor for
writting the commit message. Take into account that all changes commited
will be preserved in the history of your project. So, it is a good practice
to take some time to describe properly what have been done/changed/added.
Finally, as this is the very first commit of the project, tag this as the
zero-th version.</p>
<pre><code>git status <span class="comment"># See which files have been changed.</span>
git diff <span class="comment"># See the lines you have modified.</span>
git add -u <span class="comment"># Put all tracked changes in staging area.</span>
git status <span class="comment"># Make sure everything is fine.</span>
git commit <span class="comment"># Your first commit, add a nice description.</span>
git tag -a v0.0 <span class="comment"># Tag this as the zero-th version of your project.</span></code></pre>
<p>Now, have a look at the <code>Git</code> history of the project. Note that the local
master branch is one commit above than the remote origin/master branch.
After that, push your first commit and its tag to your remote repository
with the next commands. Since you had setup your <code>master</code> branch to follow
<code>origin/master</code>, you can just use <code>git push</code>.</p>
<pre><code>git log --oneline --decorate --all --graph <span class="comment"># Have a look at the Git history.</span>
git push <span class="comment"># Push the commit to the remote/origin.</span>
git push --tags <span class="comment"># Push all tags to the remote/origin.</span></code></pre>
<p>Now it is time to start including your own scripts to download and make the
analysis of the data. It is important to bear in mind that the goal of this
tutorial is to give a general view of the workflow in <code>Maneage</code>. In this
sense, only a few basic concepts about <code>Make</code> and how it is used into this
project will be given. <code>Maneage</code> is much more powerfull and much more things
than the ones showed in this tutorial can be done. So, read carefully all
the documentation and comments already available into each file, be creative
and experiment making your own research.</p>
<p>In the following, the tutorial will be focused in download the data, analyse
the data, and finally write the results into the final paper. As a
consequence, there are a lot of things already done that are not necessary.
For example, all the text of the final paper already written into the
<code>paper.tex</code> file, some Makefiles to download images from the Hubble Space
Telescope and analyse them, etc. In your own research, all of this work
would be removed. However, in this tutorial they are not removed because we
will only show how to do a simple analysis and include a small paragraph
with the result of the linear fitting.</p>
<p><strong>In short:</strong> in this section you have learnt how to install available
software in <code>Maneage</code>. In this particular case, you installed <code>Matplotlib</code></pre>
<h2>Including Python script to make the analysis</h2>
<p>You are going to use a small Python script to make the analysis of the data.
This Python script will be invoked from a Makefile that will be set up
later. For now, we are going to just create the Python script and put it in
an appropiate location. All analysis scripts are kept into a subfolder with
the name of the same file type in <code>reproduce/analysis</code>. For example, the
Makefiles are saved into the <code>make</code> directory, and bash scripts are saved
into the <code>bash</code> directory. Since there is any <code>python</code> directory, create it
with the following command.</p>
<pre><code>mkdir reproduce/analysis/python</code></pre>
<p>After that, you need the Python script itself. The code is very simple: it
will take an input file containing two columns (year and population), the
name of the output file in which the parameters of the linear fit will be
saved, and the name of the figure showing the original data and the fitted
curve. Paste the next Python script into a new file named <code>linear-fit.py</code>
into the directory generated in the above step
(<code>reproduce/analysis/python</code>).</p>
<pre><code><span class="comment"># Make a linear fit of an input data set
#
# This Python script makes a linear fitting of a data consisting in time and
# population. It generates a figure in which the original data and the
# fitted curve is plotted. Finally, it saves the fitting parameters.
#
<span class="comment"># Copyright (C) 2020 Raul Infante-Sainz <a href="mailto:infantesainz@gmail.com">infantesainz@gmail.com</a></span>
<span class="comment"># Copyright (C) YYYY Your Name <a href="mailto:your-email@example.xxx">your-email@example.xxx</a></span>
#
# This Python script is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or (at your
# option) any later version.
#
# This Python script is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details. See <a href="http://www.gnu.org/licenses/">http://www.gnu.org/licenses/</a>.
# Necessary packages</span>
<span class="comment"># Import necessary packages.</span>
import sys
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
<span class="comment"># Fitting function (linear fit)</span>
def func(x, a, b):
return a * x + b
<span class="comment"># Define input and output arguments</span>
ifile = sys.argv[1] <span class="comment"># Input file</span>
ofile = sys.argv[2] <span class="comment"># Output file</span>
ofig = sys.argv[3] <span class="comment"># Output figure</span>
<span class="comment"># Read the data from the input file.</span>
data = np.loadtxt(ifile)
<span class="comment"># Time and population:</span>
<span class="comment"># time ---------- x</span>
<span class="comment"># population ---- y</span>
x = data[:, 0]
y = data[:, 1]
<span class="comment"># Make the linear fit</span>
params, pcov = curve_fit(func, x, y)
<span class="comment"># Make and save the figure</span>
plt.clf()
plt.figure()
plt.plot(x, y, 'bo', label="Original data")
plt.plot(x, func(x, *params), 'r-', label="Fitted curve")
plt.title('Population along time')
plt.xlabel('Time (year)')
plt.ylabel('Population (million people)')
plt.legend()
plt.grid()
plt.savefig(ofig, format='PDF', bbox_inches='tight')
<span class="comment"># Save the fitting parameters</span>
np.savetxt(ofile, params, fmt='%.3f')
</code></pre>
<p>Have a look at this Python script. At the very beginning, it has a block of
commented lines with a descriptive title, a small paragraph describing the
the script, and the copyright with the contact information. For each file,
it is very important to have such kind of meta-data. Below these lines,
there is the source code itself.</p>
<p>As it can be seen, this Python script (<code>linear-fit.py</code>) is designed to be
invoked from the command line in the following way.</p>
<pre><code>python /path/to/linear-fit.py /path/to/input.dat /path/to/output.dat /path/to/figure.pdf</code></pre>
<p><code>/path/to/input.dat</code> is the input data file, <code>/path/to/output.dat</code> is the
output data file (with the fitted parameters), and <code>/path/to/figure.pdf</code> is
the plotted figure.</p>
<p>You will do this invokation inside of a Make rule (that will be set up
later). Now that you have included this Python script, make a commit in
order to save this work. With the first command you will see the files with
modifications. With the second command, you can check what are the changes.
Correct, add and modify whatever you want in order to include more
information, comments or clarify any step. After that, add the files and
commit the work. Finally, push the commit to the remote/origin.</p>
<pre><code>git status <span class="comment"># See which files you have changed.</span>
git diff <span class="comment"># See the lines you have added/changed.</span>
git add reproduce/analysis/python/linear-fit.py <span class="comment"># Put all tracked changes in staging area.</span>
git commit <span class="comment"># Commit, add a nice descriptions.</span>
git push <span class="comment"># Push the commit to the remote/origin.</span></code></pre>
<p>Check that everything is fine having a look at the <code>Git</code> history of the
project. Note that the <code>master</code> branch has been increased in one commit,
while the <code>template</code> branch is behind.</p>
<pre><code>git log --oneline --decorate --all --graph <span class="comment"># See the `Git` history.</span></code></pre>
<p><strong>In short</strong>: in this section you have included a <code>Python</code> script that will
be used for making the linear fitting.</p>
<h2>Downloading data</h2>
<p>As it was said before, there are multiple things that are already included
into the project. One of them is to use a dedicated Makefile to manage all
necessary download of the input data
(<code>reproduce/analysis/make/download.mk</code>). By appropiate modifications of this
file, you would be able to download the necessary data. However, in order to
keep this tutorial as simple as possible, we will describe how to download
the data you need more explicity.</p>
<p>The data needed by this tutorial consist in a simple plain text file
containing two rows: time (year) and population (in million of people). This
data correspond to Spain, and it can be downloaded from this URL:
<code>http://akhlaghi.org/data/template-tutorial/ESP.dat</code>. But don't do that
using your browser, you have to do it into <code>Maneage</code>!</p>
<p>Let's create a Makefile for downloading the data. Later, you will also
include (in the same Makefile) the necessary work in order to make the
analysis. Save this Makefile in the dedicated directory
(<code>reproduce/analysis/make</code>) with the name <code>getdata-analysis.mk</code>. In that
Makefile, paste the following code.</p>
<pre><code><span class="comment"># Download data for the tutorial</span>
<span class="comment">#</span>
<span class="comment"># In this Makefile, data for the tutorial is downloaded.</span>
<span class="comment">#</span>
<span class="comment"># Copyright (C) 2020 Raul Infante-Sainz <a href="mailto:infantesainz@gmail.com">infantesainz@gmail.com</a></span>
<span class="comment"># Copyright (C) YYYY Your Name <a href="mailto:your-email@example.xxx">your-email@example.xxx</a></span>
<span class="comment">#</span>
<span class="comment"># This Makefile is free software: you can redistribute it and/or modify it</span>
<span class="comment"># under the terms of the GNU General Public License as published by the</span>
<span class="comment"># Free Software Foundation, either version 3 of the License, or (at your</span>
<span class="comment"># option) any later version.</span>
<span class="comment">#</span>
<span class="comment"># This Makefile is distributed in the hope that it will be useful, but</span>
<span class="comment"># WITHOUT ANY WARRANTY; without even the implied warranty of</span>
<span class="comment"># MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General</span>
<span class="comment"># Public License for more details. See <a href="http://www.gnu.org/licenses/">http://www.gnu.org/licenses/</a>.</span>
<br />
<span class="comment"># Download data for the tutorial</span>
<span class="comment"># ------------------------------</span>
<span class="comment">#</span>
pop-data = $(indir)/ESP.dat
$(pop-data): | $(indir)
wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@
<br />
<span class="comment"># Final TeX macro</span>
<span class="comment"># ---------------</span>
<span class="comment">#</span>
<span class="comment"># It is very important to mention the address where the data were</span>
<span class="comment"># downloaded in the final report.</span>
$(mtexdir)/getdata-analysis.tex: $(pop-data) | $(mtexdir)
echo "\newcommand{\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@</code></pre>
<p>Have a look at this Makefile and see the different parts. The first line is
a descriptive title. Below, include your name, contact email, and finally,
the copyright. Please, take your time in order to add all relevant
information in each Makefile you modify. As you can see, these lines start
with <code>#</code> because they are comments.</p>
<p>After that information, there are five white lines in order to separate the
different parts. Then, you have the Make rule to download the data. Remember
the general structure of a Make rule:</p>
<pre><code>TARGETS: PREREQUISITES
RECIPE</code></pre>
<p>In a rule, it is said how to construct the <code>TARGETS</code> from the
<code>PREREQUISITES</code>, following the <code>RECIPE</code>. <strong>Note that the white space at the
beginning of the <code>RECIPE</code> are not spaces but a single <code>TAB</code>. Take into
account this if you copy/paste the code.</strong></p>
<p>Now you can see this structure in our particular case:</p>
<pre><code>(pop-data): | $(indir)
wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@</code></pre>
<p>Here we have:</p>
<ul>
<li><p><code>$(pop-data)</code> is the TARGET. It is previously defined just one line above:
<code>pop-data = $(indir)/ESP.dat</code>. As it can be seen, the target is just one
file named <code>ESP.dat</code> into the <code>indir</code> directory.</p></li>
<li><p><code>$(indir)</code> is the PREREQUISITE. In this case, nothing is needed for
obtaining the TARGET, just the output directory in which it is going to be
saved. This is the reason of having the pipe <code>|</code> at the beginning of the
prerequisite (it indicates an order-only-prerequisite).</p></li>
<li><p><code>wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@</code> is the
RECIPE. It states how to construct the <code>TARGET</code> from the <code>PREREQUISITE</code>.
In this case, it is just the use of <code>wget</code> to download the file specified
in the <code>URL</code> (<code>http://akhlaghi.org/data/template-tutorial/ESP.dat</code>) and
save it as the target: <code>-O $@</code>. Inisde of a Make rule, <code>$@</code> is the target.
So, in this case: <code>$@</code> is <code>$(pop-data)</code>.</p></li>
</ul>
<p>With this, you have included the rule that will download the data. Now, to
finish, you have to specify what is the final purpose of the Makefile:
download that data! This is done by setting <code>$(pop-data)</code> as a prerequisite
of the final rule. Remember that each Makefile will build a final target
with the same name as the Makefile, but with the extension <code>.tex</code>. As a
consequence, they will be <code>TeX</code> macros in which relevant information to be
included into the final paper are saved . Here, you are saving the <code>URL</code>.</p>
<pre><code>(mtexdir)/getdata-analysis.tex: $(pop-data) | $(mtexdir)
echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@</code></pre>
<p>In this final rule we have:</p>
<ul>
<li><p><code>$(mtexdir)/getdata-analysis.tex</code> is the TARGET. It is the <code>TeX</code> macro.
Note that it has the same name as the Makefile itself, but it will be
saved into the <code>$(mtexdir)</code> directory. What do I need for constructing
this target? The prerequisites.</p></li>
<li><p><code>$(pop-data) | $(mtexdir)</code> are the PREREQUISITES. In this case you have
two prerequisites. First, <code>$(pop-data)</code>, which indicates that the final
<code>TeX</code> macro has to be generated after this file has been obtained. The
second prerequisite is order-only-prerequisite, and it is the directory in
which the target is saved: <code>$(mtexdir)</code>.</p></li>
<li><p><code>echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@</code>
is the RECIPE. Basically, it writes the text
<code>\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}</code> into
the TARGET (<code>$@</code>). As you can see, this is the definition of a new
command in <code>TeX</code>. The definition of this new command <code>\popurl</code> will be used
for writting the final paper.</p></li>
</ul>
<p>Only one step is remaining to finally make the download of the data. You
have to add the name (without the extension .mk) of this Makefile into the
<code>reproduce/analysis/make/top-make.mk</code> Makefile. There it is defined which
Makefiles have to be executed. You have to end up having:</p>
<pre><code>makesrc = initialize \
download \
getdata-analyse \
delete-me \
paper</code></pre>
<p>As always, read carefully all comments and information in order to know
what is going ong. Also, add your own comments and information in order to
be clear and explain each step with enough level of detail. If everything is
fine, now the project is ready to download the data in the make step. Try
it!</p>
<pre><code>./project make</code></pre>
<p>Hopefully, it will download and save the file into the folder called
<code>inputs</code> under the <code>build-directory</code>. Check that it is there, and also have
a look at the <code>TeX</code> macro in order to see that the new command has been
included, it is into the top-build directory:
<code>build-directory/tex/macros/getdata-analysis.tex</code>.</p>
<p>Now that all of this changes have been included and it works fine, it is
time to check little by little everything and make a commit order to save
this work. Remember to put a good commit title and a nice commit message
describing what you have done and why. Then, push the commit to the
remote/origin.</p>
<p>Congratulations! You have included you first Makefile and the data is now
ready to be analysed!</p>
<p><strong>In short</strong>, to download the data you did the following:</p>
<ul>
<li>Create a Makefile: <code>reproduce/analysis/make/getdata-analysis.mk</code></li>
<li>Write meta-data at the beginning: title, your name, email, copyright, etc.</li>
<li>Define the file you want to download, and the rule to do it.</li>
<li>Write the rule to generate the <code>TeX</code> macro, putting as prerequisite, the
file you are downloading.</li>
<li>Add the name of the Makefile (without the <code>.tex</code>) into
<code>reproduce/analysis/make/top-make.mk</code></li>
<li><code>$ ./project make</code> in order to execute the project and download
the data.</li>
<li>Check that everything worked fine by loking at the downloaded file and the
<code>TeX</code> macro.</li>
<li>Commit and push all the work included.</li>
</ul>
<h2>Adding the analysis rule</h2>
<p>Until this point, you have included the Python script that will do the
linear fitting, and the rule for downloading the data. Now, it is necessary
to construct the Make rule in which this Python script is invoked to do the
analysis. This rule will be put in the same Makefile you have already
generated for downloading the data. But, before this, define the directory
in which the target is going to be saved.</p>
<pre><code>odir = $(BDIR)/fit-parameters</code></pre>
<p>This is a folder under the <code>build-directory</code> called <code>fit-parameters</code>. After
that, define the target: a plain text file in which the linear fit
parameters are saved (by the Python script). Put it into the previously
defined directory. As the data is from Spain, name it <code>ESP.txt</code>.</p>
<pre><code>param-file = $(odir)/ESP.txt</code></pre>
<p>Now, include a rule to construct the output directory <code>odir</code>. This is
necessary because this directory is needed for saving the file <code>ESP.txt</code>.</p>
<pre><code>(odir):
mkdir $@</code></pre>
<p>With all the previous definitions, now it is possible to set the rule for
making the analysis:</p>
<pre><code>(param-file): $(indir)/ESP.dat | $(odir)
python reproduce/analysis/python/linear-fit.py $< $@ $(odir)/ESP.pdf</code></pre>
<p>In this rule you have:</p>
<ul>
<li><p><code>$(param-file)</code> is the TARGET. It is the file previously defined in which
the fitting parameters will be saved.</p></li>
<li><p><code>$(indir)/ESP.dat | $(odir)</code> are the PREREQUISITES. In this case you have
two prerequisites. First, <code>$(indir)/ESP.dat</code>, which is the input file
previously downloaded by the rule above. In this file there is the input
data that the Python script will use for making the linear fit. <code>$(odir)</code>
is the second prerequisite. It is order-only-prerequisite (indicated by
the pipe <code>|</code>), and it is the directory where the target is saved.</p></li>
<li><p><code>python reproduce/analysis/python/linear-fit.py $< $@ $(odir)/ESP.pdf</code> is
the RECIPE. Basically, it call <code>python</code> to run the script
<code>reproduce/analysis/python/linear-fit.py</code> with the necessary arguments:
the input file <code>$<</code>, the target <code>$@</code>, and the name of the figure
<code>$(odir)/ESP.pdf</code> (a PDF figure saved into the same directory than the
target.</p></li>
</ul>
<p>Finally, in order to indicate you want to obtain the target you have just
included (<code>$(param-file)</code>), it is necessary to add it as a prerequisite of
the final TARGET <code>$(mtexdir)/linear-fit.tex</code>. So, in the last rule (which
creates the <code>TeX</code> macro), remove <code>$(pop-data)</code> and put <code>$(param-file)</code>
instead. By doing this, you are telling to the Makefile that you want to
obtain the file in which it is saved the fitted parameters. Inside of the
rule, define a couple of bash variables (<code>a</code> and <code>b</code>) that are the fitted
parameters extracted from the prerequisite. For <code>a</code>:</p>
<pre><code>a=$$(cat $< | awk 'NR==1{print $1}')</code></pre>
<p>Similarly, for obtaining the parameter <code>b</code> (which is in the second row):</p>
<pre><code>b=$$(cat $< | awk 'NR==2{print $1}')</code></pre>
<p>Then you have to specify the new <code>TeX</code> commands for these two parameters,
just write them as it was done before for the <code>URL</code>:</p>
<pre><code>echo "\newcommand{\afitparam}{$$a}" >> $@
echo "\newcommand{\bfitparam}{$$b}" >> $@</code></pre>
<p>So, at the end you will have the final rule like this:</p>
<pre><code>(mtexdir)/getdata-analysis.tex: $(param-file) | $(mtexdir)
echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
a=$$(cat $< | awk 'NR==1{print $1}')
b=$$(cat $< | awk 'NR==2{print $1}')
echo "\newcommand{\afitparam}{$$a}" >> $@
echo "\newcommand{\bfitparam}{$$b}" >> $@</code></pre>
<p><strong>Important notes: you have to use two <code>$</code> in order to use the bash <code>$</code>
character inside of a Make rule. Also, note that you have to put <code>>></code> in
order to not create a new target each time you write someting into the
target. With the double <code>></code> it will only add the line at the end of the file
without generating a new file.</strong></p>
<p>With all the above modifications, you are ready to obtain the fitting
parameters. If you add the necessary comments and information, the final
Makefile would look similar to:</p>
<pre><code><span class="comment"># Download data and linear fitting for the tutorial</span>
<span class="comment">#</span>
<span class="comment"># In this Makefile, data for the tutorial is downloaded. Then, a Python</span>
<span class="comment"># script is used to make a linear fitting. Finally, fitted parameters as</span>
<span class="comment"># well as the URL is saved into a TeX macro.</span>
<span class="comment">#</span>
<span class="comment"># Copyright (C) 2020 Raul Infante-Sainz <a href="mailto:infantesainz@gmail.com">infantesainz@gmail.com</a></span>
<span class="comment"># Copyright (C) YYYY Your Name <a href="mailto:your-email@example.xxx">your-email@example.xxx</a></span>
<span class="comment">#</span>
<span class="comment"># This Makefile is free software: you can redistribute it and/or modify it</span>
<span class="comment"># under the terms of the GNU General Public License as published by the</span>
<span class="comment"># Free Software Foundation, either version 3 of the License, or (at your</span>
<span class="comment"># option) any later version.</span>
<span class="comment">#</span>
<span class="comment"># This Makefile is distributed in the hope that it will be useful, but</span>
<span class="comment"># WITHOUT ANY WARRANTY; without even the implied warranty of</span>
<span class="comment"># MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General</span>
<span class="comment"># Public License for more details. See <a href="http://www.gnu.org/licenses/">http://www.gnu.org/licenses/</a>.</span>
<br />
<span class="comment"># Download data for the tutorial</span>
<span class="comment"># ------------------------------</span>
<span class="comment"># The input file is defined and downloaded using the following rule</span>
pop-data = $(indir)/ESP.dat
$(pop-data): | $(indir)
<span class="comment"> # Use wget to download the data</span>
wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@
<br />
<span class="comment"># Output directory</span>
<span class="comment"># ----------------</span>
<span class="comment"># Small rule for constructing the output directory, previously defined</span>
odir = $(BDIR)/fit-parameters
$(odir):
mkdir $@
<br />
<span class="comment"># Linear fitting of the data</span>
<span class="comment"># --------------------------</span>
<span class="comment"># The output file is defined into the output directory. The fitted</span>
<span class="comment"># parameters will be saved into this directory by the Python script.</span>
param-file = $(odir)/ESP.txt
$(param-file): $(indir)/ESP.dat | $(odir)
<span class="comment"> # Invoke Python to run the script with the input data</span>
python reproduce/analysis/python/linear-fit.py $< $@ $(odir)/ESP.pdf
<br />
<span class="comment"># TeX macros final target</span>
<span class="comment"># -----------------------</span>
<span class="comment"># This is how we write the necessary parameters in the final PDF. In this</span>
<span class="comment"># rule, new TeX parameters are defined from the URL, and the fitted</span>
<span class="comment"># parameters.</span>
$(mtexdir)/getdata-analysis.tex: $(param-file) | $(mtexdir)
<span class="comment"> # Write the URL into the target</span>
echo "\newcommand{\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
<span class="comment"> # Read the fitted parameters into shell variables.</span>
a=$$(cat $< | awk 'NR==1{print $1}')
b=$$(cat $< | awk 'NR==2{print $1}')
<span class="comment"> # Write the parameters into the target as LaTeX macros.</span>
echo "\newcommand{\afitparam}{$$a}" >> $@
echo "\newcommand{\bfitparam}{$$b}" >> $@</code></pre>
<p>Have look at this Makefile and note that it is what it has been described
above. Take your time for making useful comments and modifying whatever you
think it is necessary. If everything is fine, now the project is ready to
download the data <strong>and</strong> make the linear fitting. Try it!</p>
<pre><code>./project make</code></pre>
<p>Hopefully, now you will have the fitted parameters into the
<code>build-directory/fit-parameters/ESP.txt</code> file, and the figure in the same
directory. Do not pay to much attention at the quality of the fitting. It is
just an example. Also, check that the <code>TeX</code> macro has been created
successfully by having a look at
<code>build-directory/tex/macros/getdata-analyse.tex</code>. Finally, now that you have
ensured that everything is fine, make a commit in order to keep the work
safe. In the next step, you will see how to include this data into the final
paper.</p>
<p><strong>In short:</strong> with the work included in this section, the project is able to
download and make the linear fitting of the data. The result is the fitted
parameters that are also saved in a <code>TeX</code> macro, and the figure showing the
data with the fitted curve.</p>
<h2>Editing the final paper</h2>
<p>With all the previous work, the project is able to download the file
containing the data (two columns, year and population of Spain), and analyse
them by making a linear fitting (y=ax+b). The result is a <code>TeX</code> macro in
which there are the information about the <code>URL</code> of the data and the linear
fitting parameters (<code>a</code> and <code>b</code>). Now, it is time to add a small paragraph
into the paper, just to ilustrate how to write the relevant parameters from
the analysis.</p>
<p>Before all, make a copy of the current <code>paper.pdf</code> document you have into
the <code>project-directory</code>. This paper is an example that <code>Maneage</code> constructs
by default. Now, you will modify it by adding a small paragraph including
the fitting parameters and the <code>URL</code>. So, open <code>project-directory/paper.tex</code>
and add the following paragraph just at the beginning of the abstract
section.</p>
<pre><code>By following the steps described in the tutorial, I have been able to obtain this reproducible paper!
The project is very simple and it consists in download a file (from \popurl), and make an easy linear fit using a Python script.
The linear fitting is $y=a*x+b$, with the following parameters: $a=\afitparam$ and $b=\bfitparam$</code></pre>
<p>As you can see, the <code>TeX</code> definitions done before in the Makefiles, are now
included into the paper: <code>\popurl</code>, <code>\afitparam</code>, and <code>\bfitparam</code>. If you
do again the make step <code>$ ./project make</code>, you will re-compile the paper
including this paragraph. Check that it is true and compare with the
previous version, of the paper. Contratulations! You have complete this
tutorial and now you are able to use <code>Maneage</code> for making your exciting
research in a reproducible way!</p>
<footer role="contentinfo" id="page-footer">
<ul>
<li><p>Maneage is currently based in the Instituto de Astrofísica de Canarias (IAC).</p></li>
<li><p>Address: IAC, Calle Vía Láctea, s/n, E38205 - La Laguna (Tenerife), Spain.</p></li>
<li><p>Contact: with <a href="https://savannah.nongnu.org/support/?func=additem&group=reproduce">this form</a>, or project PI (<a href="http://akhlaghi.org">Mohammad Akhlaghi</a>).</p></li>
<li><p>Copyright © 2020 Maneage volunteers</p></li>
<li>This page is distributed under GNU General Public License (<a href="https://www.gnu.org/licenses/gpl-3.0.en.html">GPL</a>).</li>
</ul>
</footer>
</body>
</html>