path: root/paper.tex

\documentclass[10.5pt]{article}

%% This is a convenience variable if you are using PGFPlots to build plots
%% within LaTeX. If you want to import PDF files for figures directly, you
%% can use the standard `\includegraphics' command. See the definition of
%% `\includetikz' in `tex/preamble-pgfplots.tex' for where the files are
%% assumed to be if you use `\includetikz' when `\makepdf' is not defined.
\newcommand{\makepdf}{}

%% When defined (value is irrelevant), `\highlightchanges' will cause text
%% in `\tonote' and `\new' to become colored. This is useful in cases that
%% you need to distribute drafts that is undergoing revision and you want
%% to highlight to your colleagues which parts are new and which parts are
%% only for discussion.
%\newcommand{\highlightchanges}{}

%% Import the necessary preambles.
\input{tex/src/preamble-style.tex}
\input{tex/build/macros/project.tex}
\input{tex/src/preamble-pgfplots.tex}
\input{tex/src/preamble-biblatex.tex}





\title{Maneage, a Customizable Framework for Managing Data Lineage}
\author{\large\mpregular \authoraffil{Mohammad Akhlaghi}{1,2,3},
        \large\mpregular \authoraffil{Ra\'ul Infante-Sainz}{1,2},
        \large\mpregular \authoraffil{Boudewijn F. Roukema}{4,3}
        \large\mpregular \authoraffil{David Valls-Gabaud}{5},
        \large\mpregular \authoraffil{Roberto Baena-Gall\'e}{1,2}\\
  {
    \footnotesize\mplight
    \textsuperscript{1} Instituto de Astrof\'isica de Canarias, Calle V\'ia L\'actea s/n, 38205 La Laguna, Tenerife, Spain.\\
    \textsuperscript{2} Departamento de Astrof\'isica, Universidad de La Laguna, Avenida Astrof\'isico Francisco S\'anchez s/n, 38200 La Laguna, Tenerife, Spain.\\
    \textsuperscript{3} Univ Lyon, Ens de Lyon, Univ Lyon1, CNRS, Centre de Recherche Astrophysique de Lyon UMR5574, F-69007, Lyon, France.\\
    \textsuperscript{4} Institute of Astronomy, Faculty of Physics, Astronomy and Informatics, Nicolaus Copernicus University, Grudziadzka 5, 87-100 Toru\'n, Poland.\\
    \textsuperscript{5} LERMA, CNRS, Observaoire de Paris, 61 Avenue de l'Observatoire, 75014 Paris, France.\\
    Corresponding author: Mohammad Akhlaghi
    (\href{mailto:mohammad@akhlaghi.org}{\textcolor{black}{mohammad@akhlaghi.org}})
  }}
\date{}





\begin{document}%\layout
\thispagestyle{firstpage}
\maketitle

%% Abstract
{\noindent\mpregular
  The era of big data has ushered an era of big responsibility.
  In the absence of reproducibility, as a test on understanding data lineage, the result can be the subject of perpetual debate.
  To address this problem, we introduce Maneage (management + lineage) which is founded on the principles of completeness (e.g., no dependency beyond a POSIX-compatible operating system, no administrator privileges, and no network connection), modular and straightforward design, temporal provenance, scalability, and free software.
  A project using Maneage is fully stored in machine\--action\-able, and human\--read\-able plain-text format, facilitating version-control, publication, archival, and automatic parsing to extract data provenance.
  The provided lineage is not limited to high-level processing, but also includes building the necessary software from source with fixed versions and build configurations.
  Additionally, a project's final visualizations and narrative report are also included, establishing direct links between the analysis and the narrative or visualizations, to the precision of a word within a sentence or a point in a plot.
  Maneage also enables incremental projects, where a new project can branch off an existing one, with moderate changes to enable experimentation on published methods.
  Once Maneage is implemented in a sufficiently wide scale, automatic and optimized workflow creation through machine learning, or automating data management plans, can easily be set up.
  Maneage was a recipient of a research data alliance (RDA) Europe Adoption Grant in 2019,  and has already been tested and used in several scientific papers, including the present one, with snapshot \projectversion.
  \horizontalline

  \noindent
  {\mpbold Keywords:} Data Lineage, Data Provenance, Reproducibility, Scientific Pipelines, Workflows

%  \noindent
%  {\mpbold Note to DSJ editors or referees:} The distributed source of this project (described in Section \ref{sec:publishing}) is available in this URL: \url{https://akhlaghi.org/dsj-paper-\projectversion.tar.gz}
}

\horizontalline










\section{Introduction}
\label{sec:introduction}

The increasing volume and complexity of data analysis has been extraordinarily productive, giving rise to a new branch of ``Big Data'' in many fields of the sciences and industry.
However, given its inherent complexity, the results are barely useful alone, questions naturally arise on their lineage or provenance:
What inputs were used?
How were the configurations or training data chosen?
What operations were done on those inputs, how were the plots made?
See Figure \ref{fig:questions} for some similar questions, classified by their place in project.
\tonote{Johan: add some general references.}

Due to the complexity of modern data analysis, a small deviation in the final result can be due to many different steps, which may be significant for its interpretation.
Integrity checks are a critical component of the scientific method, but are only possible with access to the data \emph{and} its lineage (workflows).
For example, \citet{smart18} describes how a 7-year old conflict in theoretical condensed matter physics was only identified after the relative codes were shared.
\citet{miller06} found a mistaken column flipping in a project's workflow, leading to the retraction of 5 papers in major journals, including \emph{Science}.
\citet{baggerly09} highlighted the inadequate narrative description of the analysis and showed the prevalence of simple errors in published results, ultimately calling their work ``\emph{forensic bioinformatics}''.
\citet{herndon14} and \citet[a self-correction]{horvath15} also reported similar situations and \citet{ziemann16} concluded that one-fifth of papers contain erroneous gene name conversions.
These are mostly from genomics and bioinformatics because publishing workflows is commonly practiced already (for example \href{https://www.myexperiment.org}{myexperiment.org}, \href{https://www.genepattern.org}{genepattern.org}, and \href{https://galaxyproject.org}{galaxy\-project.org}).
The status in other fields, without a culture of publishing workflows, is highly likely to be worse.
Nature is already a black box which we are trying hard to unlock.
Not being able to experiment on the methods of other researchers is a self-imposed back box over it.

\begin{figure}[t]
  \begin{center}
    \includetikz{figure-project-outline}
  \end{center}
  \vspace{-17mm}
  \caption{\label{fig:questions}Graph of a generic project's workflow (connected through arrows), highlighting the various issues/questions on each step.
    The green boxes with sharp edges are inputs and the blue boxes with rounded corners are intermediate or final outputs.
    The red boxes with dashed edges highlight the main questions at each respective stage.
    The box covering software download and build phases shows some common tools software developers use for this phase, but a scientific project is clearly much more involved.
  }
\end{figure}

The completeness of a project's published workflow (usually within the ``Methods'' section) can be measured by the ability to reproduce the result without needing to contact the authors.
Several studies have attempted to answer this with different levels of detail. For example, \citet{allen18} found that roughly half of the papers in astrophysics do not even mention the names of any analysis software, while \citet{menke20} found this fraction has greatly improved in medical/biological field and is currently above $80\%$.
\citet{ioannidis2009} attempted to reproduce 18 published results by two independent groups, but fully succeeded in only 2 of them and partially in 6.
\citet{chang15} attempted to reproduce 67 papers in well-regarded  journals in Economics with data and code: only 22 could be reproduced without contacting authors, and more than half could not be replicated at all. \tonote{DVG: even after contacting the authors?}
\citet{stodden18} attempted to replicate the results of 204 scientific papers published in the journal \emph{Science} \emph{after} that journal adopted a policy of publishing the data and code associated with the papers.
Even though the authors were contacted, the success rate was an abysmal $26\%$.
Overall, this problem is unambiguously assessed as being very serious in the community: \citet{baker16} surveyed 1574 researchers and found that only $3\%$ did not see a ``\emph{reproducibility crisis}''.

Yet, this is not a new problem in the sciences: back in 2011, Elsevier conducted an ``\emph{Executable Paper Grand Challenge}'' \citep{gabriel11} and the proposed solutions were published in a special edition.\tonote{DVG: which were the results?}
Even before that, in an attempt to simulate research projects, \citet{ioannidis05} proved that ``\emph{most claimed research findings are false}''.
In the 1990s, \citet{schwab2000, buckheit1995, claerbout1992} described the same problem very eloquently and also provided some solutions they used.\tonote{DVG: more details here, one is left wondering ...}
Even earlier, through his famous quartet, \citet{anscombe73} qualitatively showed how distancing of researchers from the intricacies of algorithms/methods can lead to misinterpretation of the results.
One of the earliest such efforts we are aware of is the work of \citet{roberts69}, who discussed conventions in Fortran programming and documentation to help in publishing research codes.
While the situation has somewhat improved, all these papers still resonate strongly with the frustrations of today's scientists.

In this paper, we introduce Maneage as a solution to the collective problem of preserving a project's data lineage and its software dependencies.
A project using Maneage starts by branching from its main Git branch, allowing the authors to customize it: specifying the necessary software tools for that particular project, adding analysis steps and adding visualizations and a narrative based on the results.
In Sections \ref{sec:definitions} \& \ref{sec:principles} the basic concepts are defined and the founding principles of Maneage are discussed.
Section \ref{sec:maneage} describes the internal structure of Maneage and Section \ref{sec:discussion} is a discussion on its benefits, caveats and future prospects and we conclude with a summary in Section \ref{sec:conclusion}










\section{Definitions}
\label{sec:definitions}

The concepts and terminologies of reproducibility and project/workflow management and design are used differently by different research communities or different solution providers.
It is therefore important to clarify some specific terms used in this paper.

\begin{enumerate}[label={\bf D\arabic*}]
\item \label{definition:input}\textbf{Input:}
  A project's input is any file that may be usable in other projects.
  The inputs of a project include data or software source code \citep[see][on the fundamental similarity of data and source code]{hinsen16}.
  Inputs may have initially been created/written (e.g., software source code) or collected (e.g., data) for one specific project. However, they can, and most often will, be used in later projects as well.

\item \label{definition:output}\textbf{Output:}
  A project's output is any file that is published at the end.
  The output(s) of a project can be a narrative paper or report with visualizations, datasets (e.g., table(s), image(s), a number, or Boolean: confirming a hypothesis as being true or false), automatically-generated software source code, or any other computer file.

\item \label{definition:project}\textbf{Project:}
  A project is the series of operations that are done on input(s) to produce outputs.
  This definition is therefore very similar to ``\emph{workflow}'' \citep[e.g.,][]{oinn04, goecks10}, but because the published narrative paper/report is also an output, a project also includes both the source of the narrative (e.g., \LaTeX{} or MarkDown) \emph{and} how
  its visualizations were created.

  In a well-designed project, all analysis steps (e.g., written in Python, packages in R, libraries/programs in C/C++, etc.) are written to be modular, or executable independent of the rest with well-defined inputs, outputs and no side-effects.
  This is crucial help for debugging and experimenting during the project, and also for their re-usability in later projects.
  As a consequence, such analysis scripts/programs are defined above as ``inputs'' for the project.
  A project hence does not include any analysis source code (to the extent this is possible), it only manages calls to them.

\item \label{definition:provenance}\textbf{Data Provenance:}
  A dataset's provenance is defined as the set of metadata (in any ontology, standard or structure) that connects it to the components (other datasets or scripts) that produced it.
  Data provenance thus provides a high-level \emph{and structured} view of a project's lineage.
  A good example of this is Research Objects \citep{belhajjame15}.

% This definition of data lineage is inspired from https://stackoverflow.com/questions/43383197/what-are-the-differences-between-data-lineage-and-data-provenance:

% "data provenance includes only high level view of the system for business users, so they can roughly navigate where their data come from.
% It's provided by variety of modeling tools or just simple custom tables and charts.
% Data lineage is a more specific term and includes two sides - business (data) lineage and technical (data) lineage.
% Business lineage pictures data flows on a business-term level and it's provided by solutions like Collibra, Alation and many others.
% Technical data lineage is created from actual technical metadata and tracks data flows on the lowest level - actual tables, scripts and statements.
% Technical data lineage is being provided by solutions such as MANTA or Informatica Metadata Manager. "
\item \label{definition:lineage}\textbf{Data Lineage:}
  Data lineage is commonly used interchangeably with Data provenance \citep[for example][]{cheney09}.
  For clarity, we define the term ``\emph{Data lineage}'' as a low-level and fine-grained recording of the data's trajectory in an analysis (not meta-data, but actual commands).
  Therefore, data lineage is synonymous with ``\emph{project}'' as defined above.
\item \label{definition:reproduction}\textbf{Reproducibility \& Replicability:}
  These two terms have been used in the literature with various meanings, sometimes in a contradictory way.
  It is important to highlight that in this paper we are only considering computational analysis: \emph{after} data has been collected and stored as a file.
  Therefore, many of the definitions reviewed in \citet{plesser18}, which are about data collection, do not apply here, and
  we adopt the same definition of \citet{leek17,fineberg19}, among others.
  \citet{fineberg19} define reproducibility as \emph{obtaining consistent [not necessarily identical] results using the same input data; computational steps, methods, and code; and conditions of analysis}, or same inputs $\rightarrow$ consistent result.
  They define Replicability as \emph{obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data}, or different inputs $\rightarrow$ consistent result.
\end{enumerate}




\section{Principles}
\label{sec:principles}

The core principle of Maneage is simple: science is defined primarily by its method, not its result.
As \citet{buckheit1995} describe it, modern scientific papers are merely advertisements of scholarship, while the actual scholarship is the coding behind the plots/results.
Many solutions have been proposed in the last decades, including (but not limited to)
1992: \href{https://sep.stanford.edu/doku.php?id=sep:research:reproducible}{RED},
2003: \href{https://taverna.incubator.apache.org}{Apache Taverna},
2004: \href{https://www.genepattern.org}{GenePattern},
2010: \href{https://wings-workflows.org}{WINGS},
2011: \href{https://www.ipol.im}{Image Processing On Line journal} (IPOL),
      \href{https://www.activepapers.org}{Active papers},
      \href{https://is.ieis.tue.nl/staff/pvgorp/share}{SHARE},
2015: \href{https://sciunit.run}{Sciunit};
2017: \href{https://falsifiable.us}{Popper};
2019: \href{https://wholetale.org}{WholeTale}.
To help in the comparison, the founding principles of Maneage are listed below.

\begin{enumerate}[label={\bf P\arabic*}]
\item \label{principle:complete}\textbf{Completeness:}
  A project that is complete, or self-contained,
  (P1.1) has no dependency beyond the Port\-able Operating System (OS) Interface, or POSIX.
  (P1.2) does not affect the host,
  (P1.3) does not require root, or administrator, privileges,
  (P1.4) builds its software for an independent environment,
  (P1.5) can be run locally (without internet connection),
  (P1.6) contains the full project's analysis, visualization \emph{and} narrative, from access to raw inputs to producing final published format (e.g., PDF or HTML),
  (P1.7) requires no manual/human interaction and can run automatically \citep[according to][``\emph{a clerk can do it}'']{claerbout1992}.
  A consequence of P1.1 is that the project itself must be stored in plain-text: not needing any specialized software to open, parse or execute.

  \emph{Comparison with existing:} with many dependencies beyond POSIX, except for IPOL, none of the tools above are complete.
  For example, most recent solutions need Python (for the workflow, not the analysis), or rely on Jupyter notebooks.
  High-level tools have short lifespans and evolve fast (e.g., Python 2 code cannot run with Python 3).
  They also have complex dependency trees, making them hard to maintain.
  For example, see the dependency tree of Matlplotlib in \citet[][Figure 1]{alliez19}, its one of the simpler Jupyter dependencies.
  The longevity of a workflow is determined by its shortest-lived dependency.

  As a result the primary storage format of most recent solutions is pre-built binary blobs like containers or virtual machines.
  They are large (Giga-bytes) and expensive to archive, furthermore generic package managers (e.g., Conda), or OS's (e.g., \inlinecode{apt} or \inlinecode{yum}) are used to setup its environment.
  Because exact versions of \emph{every software} are rarely included, and the servers remove old binaries, recreating them is very hard.
  Blobs also have a short lifespan (e.g., Docker containers only run on long-term support OSs, in GNU/Linux systems, this corresponds to Linux 3.2.x, released in 2012).
  A plain-text project consumes below one megabyte, is human-readable and parsable by any machine, even if it can't be executed.

\item \label{principle:modularity}\textbf{Modularity:}
A project should be compartmentalized into independent modules with well-defined inputs/outputs having no side effects.
Communication between the independent modules should be explicit, providing several optimizations:
(1) independent modules can run in parallel.
Modules that do not need to be run (because their dependencies have not changed) will not be re-run.
(2) Data provenance extraction (recording any dataset's origins).
(3) Citation: others can credit specific parts of a project.
(4) Usage in other projects.
(5) Most importantly: they are easy to debug and improve.

\emph{Comparison with existing:} Visual workflow tools like Apache Taverna, GenePattern, Kepler or VisTrails encourage this, but the more recent tools leave this design choice to project authors.
However, designing a modular project needs to be encouraged and facilitated.
Otherwise, scientists, who are not usually trained in data management, will rarely design a modular project, leading to great inefficiencies in terms of project cost and/or scientific accuracy (testing/validating will be expensive).

\item \label{principle:complexity}\textbf{Minimal complexity:}
  This is Ockham's razor extrapolated to project management \citep[``\emph{Never posit pluralities without necessity}''][]{schaffer15}:
  1) avoid complex relations between analysis steps (related to \ref{principle:modularity}).
  2) avoid the programming language that is currently in vogue, because it is going to fall out of fashion soon and require significant resources to translate or rewrite it every few years (to stay fashionable).
  The same job can be done with more stable/basic tools, requiring less long-term effort.

  \emph{Comparison with existing:} IPOL stands out here too (requiring only ISO C), however most existing solutions use tools that were most popular at their creation date.
  For example, as we approach the present, successively larger fractions of tools are written in Python, and use Conda or Jupyter (see \ref{principle:complete}).

\item \label{principle:verify}\textbf{Verifiable inputs and outputs:}
The project should automatically verify its inputs (software source code and data) \emph{and} outputs.
Thus not needing any expert knowledge to confirm a reproduction.

\emph{Comparison with existing:} Such verification is usually possible in most systems, but is usually the responsibility of the project authors.
As with \ref{principle:modularity}, due to lack of training, this must be actively encouraged and facilitated, otherwise most will not be able to implement it.

\item \label{principle:history}\textbf{History and temporal provenance:}
No project is done in a single/first attempt.
Projects evolve as they are being completed.
It is natural that earlier phases of a project are redesigned/optimized only after later phases have been completed.
This is often seen in exploratory research papers, with statements like ``\emph{we [first] tried method [or parameter] X, but Y is used here because it gave lower random error}''.
A project's ``history'' is thus as scientifically relevant as the final, or published version.

\emph{Comparison with existing:} The solutions above that implement version control usually support this principle.
However, because the systems as a whole are rarely complete (see \ref{principle:complete}), their histories are also incomplete.
IPOL fails here, because only the final snapshot is published.

\item \label{principle:scalable}\textbf{Scalability:}
A project should be scalable to arbitrarily large and/or complex projects.

\emph{Comparison with existing:}
Most of the more recent solutions above are scalable.
However, IPOL, which uniquely stands out in satisfying most principles, fails here: IPOL is devoted to low-level image processing algorithms that \emph{can be} done with no dependencies beyond an ISO C compiler.
IPOL is thus not scalable to large projects, which commonly involve dozens of high-level dependencies, with complex data formats and analysis.

\item \label{principle:freesoftware}\textbf{Free and open source software:}
  Technically, reproducibility (see \ref{definition:reproduction}) is possible with non-free or non-open-source software (a black box).
  This principle is thus necessary to complement that definition to ensure that the project is reproducible \emph{and} free/open:
  (1) When the project itself is free software, others can learn from and build upon it.
  (2) The lineage can be traced to free software's implemented algorithms, enabling optimizations on that level.
  (3) A free-software package that does not execute on particular hardware can be modified to work on it.
  (4) A non-free software project typically cannot be distributed by others, making the whole community reliant on the owner's server (even if the owner does not ask for payments).

  \emph{Comparison with existing:} The existing solutions listed above are all free software.
  Based on this principle, we do not consider non-free solutions.
\end{enumerate}








\section{Maneage}
\label{sec:maneage}
Maneage is an implementation of the principles of Section \ref{sec:principles}.
In practice, Maneage is a collection of plain-text files that are distributed in pre-defined sub-directories by context (a modular source), and are all under version control, currently with Git.
The main Maneage Branch is a fully working skeleton of a project without much flesh: it contains all the low-level infrastructure, but without any actual high-level analysis operations.
Maneage contains a file called \inlinecode{README-hacking.md} (the \inlinecode{README.md} file is reserved for the project using Maneage, not Maneage itself) that has a complete checklist of steps to start a new project and remove demonstration parts.
There are also hands-on tutorials to help new users.

To start a new project, the authors \emph{clone} Maneage, create a branch, and start their project by customizing it.
Thus, projects start with a good data management strategy rather than imposing it in the end, as recommended by \citet{fineberg19}.
Customization is done by adding the names of the necessary software, references to input data, analysis and visualization commands and writting a narrative description.
This will be done in multiple commits during the project (perhaps years), preserving the project's history: the descriptions of, and motivations for, changes or test failures and successes, as well as the authors and timestamps of each change.

\begin{lstlisting}[language=bash]
  git clone https://gitlab.com/maneage/project.git  # Clone Maneage, default branch `maneage'.
  mv project my-project && cd my-project            # Set custom name and enter directory.
  git remote rename origin origin-maneage           # Rename remote server to use `origin' later.
  git checkout -b master                            # Make new `master' branch, start customizing.
\end{lstlisting}

\begin{figure}[t]
  \begin{center}
    \includetikz{figure-file-architecture}
  \end{center}
  \vspace{-5mm}
  \caption{\label{fig:files}
    Directory and file structure in a hypothetical project using Maneage.
    Files are shown with small green boxes that have a suffix in their names (for example \inlinecode{format.mk} or \inlinecode{download.tex}).
    Directories (containing multiple files) are shown as large brown boxes, where the name ends in a slash (\inlinecode{/}).
    Directories with dashed lines and no files (just a description) are symbolic links that are created after building the project, pointing to commonly-needed built directories.
    Symbolic links and their contents are not considered part of the source and are not under version control.
    Files and directories are shown within their parent directory.
    For example, the full address of \inlinecode{format.mk} from the top project directory is \inlinecode{reproduce/analysis/make/format.mk}.
  }
\end{figure}

Figure \ref{fig:files} shows the directory structure of the cloned project and typical files.
The top-level source has only very high-level components: the \inlinecode{project} shell script (POSIX-compliant) that is the main interface to the project, as well as the paper's \LaTeX{} source, documentation and a copyright statement.
Two sub-directories are also present: \inlinecode{tex/} (containing \LaTeX{} files) and \inlinecode{reproduce/} (containing all other parts of the project).
Maneage has two main phases: (1) configuration, where the necessary software is built and the environment is set up, and (2) analysis, where data are accessed and the software is run to create the final visualizations and report:

\begin{lstlisting}[language=bash]
  ./project configure      # Build all necessary software from source.
  ./project make           # Do the analysis (download data, run software on data, build PDF).
\end{lstlisting}

Section \ref{sec:usingmake} elaborates why Make was chosen as the main job manager.
Sections \ref{sec:projectconfigure} \& \ref{sec:projectanalysis} are on the operations done during the configuration and analysis phase.
The benefit from version control is described in Section \ref{sec:projectgit}.
Section \ref{sec:collaborating} discusses the sharing of a built environment, and finally, Section \ref{sec:publishing} is about the publication, or archival, of Maneage projects.

\subsection{Job orchestration with Make}
\label{sec:usingmake}

Scripts (e.g. shell, Perl, or Python) are an obvious solution for non-interactive (batch) processing.
However, the inherent complexity and non-linearity of progress as a project evolves makes it hard to manage scripts.
For example, if $90\%$ of a research project is done and only the final $10\%$ must be executed, a script will re-do the whole project.
Completed parts can be manually ignored (with conditionals), but this adds to the complexity and discourages experimentation on already completed parts.
These problems motivated the creation of Make in the early Unix OS \citep{feldman79}.
Make contiues to be a core component of modern OSs, is actively maintained, and has withstood the test of time.

The Make paradigm starts from the end: the final \emph{target}.
In Make, the project is broken into atomic \emph{rules}, where each rule has a single \emph{target} file which can depend on any number of \emph{prerequisite} files.
To build the target from the prerequisites, each rule also has a \emph{recipe} (an atomic script).
The plain-text files containing Make source code are called Makefiles.
Make does not replace scripting languages like the shell, Python or R.
It is a higher-level structure enabling modular/atomic scripts (in any language) to be put into a workflow.

Besides formalizing a project's data lineage, Make also greatly encourages experimentation in a project, because a recipe is executed only when at least one prerequisite file is more recent than its target.
For example, when only $5\%$ of a project's targets are affected by a change, the other $95\%$ remain dormant.
Furthermore, Make first examines the full lineage before starting the execution of recipes, and it can thus execute independent rules in parallel, improving speed and encouraging experimentation.

Make is well known by many outside of software development communities.
For example, geophysics students have easily adopted it for the RED project management tool \citep{schwab2000}.
We also received very good feedback on the simplicity of using Make from early adopters of Maneage, especially graduate students and postdocs.






\subsection{Project configuration}
\label{sec:projectconfigure}

Maneage organizes both the building of its software and the analysis pipeline using Make (see Section \ref{sec:usingmake}).
Thus, a researcher using Maneage for high-level analysis easily understands and can customize the software environment without needing to learn third-party tools.
The existing tools listed in Section \ref{sec:principles} mostly use package managers like Conda to maintain the software environment, but Conda itself is written in Python, contrary to our completeness principle \ref{principle:complete}.
Highly-robust solutions like Nix and GNU Guix exist, but these require root permissions, contrary to principle P1.3.

Project configuration (building the software environment) is managed by the files under \inlinecode{reproduce\-/soft\-ware} of Maneage's source (see Fig.~\ref{fig:files}).
At the start of project configuration, Maneage needs a top-level directory to build itself on the host (software and analysis).
We call this the ``build directory'' and it must not be located inside the source directory (see \ref{principle:modularity}).
No other location on the running OS will be affected by the project, including the source directory.
Two other local directories can optionally be specified by the project when inputs (\ref{definition:input}) are present locally: 1) software tarball directory and 2) input data directory.
Sections \ref{sec:buildsoftware} and \ref{sec:softwarecitation} detail the building of the required software and the important issue of software citation.

\subsubsection{Verifying and building necessary software from source}
\label{sec:buildsoftware}

To compile the necessary software from source, Maneage currently needs the host to have a C and C++ compiler (available on any POSIX-compliant OS).
Maneage will build and install (in the build directory) all necessary software and their dependencies, all with fixed versions and configurations.
The dependency tree continues down to core OS components including GNU Bash, GNU AWK, GNU Coreutils on all supported OSs (including macOS, not just GNU/Linux).
On GNU/Linux OSs, a fixed version of the GNU Binutils and GNU C Compiler (GCC) is also built from source, and a fixed version of the GNU C Library will soon be added to be fully independent of the host on such systems (task 15390).

Except for the Kernel, Maneage thus builds all other OS components necessary for the project.
A Maneage project can be configured in a container or virtual machine to facilitate moving the project without rebuilding everything from source.
However, such binary blobs are not Maneage's primary storage/archival format.

Before building the software, their source codes are validated by their SHA-512 checksum (stored in the project).
Maneage includes a large collection of scientific software (and its dependencies), much of which is superfluous for one project.
Therefore, each project has to identify its high-level software in the \inlinecode{TARGETS.conf} file under \inlinecode{re\-produce\-/soft\-ware\-/config} directory (Fig.~\ref{fig:files}).

\subsubsection{Software citation}
\label{sec:softwarecitation}

Maneage contains the full list of software that were built for the project but this information is buried deep into the source.
Maneage prints a simplified description of this information in the project's final report, blended into the narrative, as in the Acknowledgments of this paper.
Furthermore, when the software is associated with a published paper, that paper's Bib\TeX{} entry is added to the final report and is duly cited with the software's name and version.
This paper uses basic software without associated scientific papers. For software citation examples, see \citet{akhlaghi19} and \citet{infante20}.

This is particularly important for research software, where citation is critical to justify continued development.
A notable example is GNU Parallel \citep{tange18} which prints citation information each time it is run, proposing to either cite the paper or support it with 10000 euros.
It provides a \inlinecode{--citation} option to disable the notice.
This is justified by an uncomfortably true statement: ``\emph{history has shown that researchers forget to [cite software] if they are not reminded explicitly. ... If you feel the benefit from using GNU Parallel is too small to warrant a citation, then prove that by simply using another tool}''.
Most software does not resort to such drastic measures. However, proper citation is not only useful practically, it is also an ethical imperative.

Given the increasing role of software in research \citep{clement19}, automatic citation (as presented here) is a step forward.
For a review of the necessity and basic elements of software citation, see \citet{katz14} and \citet{smith16}.
The CodeMeta and Citation file format (CFF) aim to expand software citation beyond Bib\TeX, while Software Heritage \citep{dicosmo18} also includes archival and citation abilities.
These will be tested and enabled in Maneage.





\subsection{Project analysis}
\label{sec:projectanalysis}

Once the project is configured (Section \ref{sec:projectconfigure}), a unique and fully-controlled environment is available to execute the analysis.
All analysis operations run with no influence from the host OS, enabling an isolated environment without the extra layer of containers or a virtual machine.
In Maneage, a project's analysis is broken into two phases: 1) preparation, and 2) analysis.
The former is mostly necessary to optimize extremely large datasets and is only useful for advanced users, while following an identical internal structure to the later.
We will therefore not go any further into it and refer the interested reader to the documentation.

A project consists of many steps, including data access (possibly by downloading), running various steps of the analysis on the raw inputs, and creating the necessary plots, figures or tables for a published report, or output datasets for a database.
If all of these steps were organized in a single Makefile, it would become very large, or long, and would be hard to maintain, extend/grow, read, reuse, and cite.
Large files are in general a bad practice and do not fulfil the modularity principle (\ref{principle:modularity}).

Maneage is thus designed to encourage and facilitate modularity by distributing the analysis into many Makefiles that contain contextually-similar analysis steps.
Hereafter, these modular or lower-level Makefiles will be called \emph{subMakefiles}.
When run with the \inlinecode{make} argument, the \inlinecode{project} script (Section \ref{sec:maneage}), calls \inlinecode{top-make.mk} which loads the subMakefiles with a certain order.
They are loaded using Make's \inlinecode{include} feature (so Make sees everything as one file in one instance of Make).
By default Maneage does not use recursion (where one instance of Make, calls another instance of Make within itself) to comply with minimal complexity principle (\ref{principle:complexity}) and keep the code's logic clear and simple.
All the analysis Makefiles are in \inlinecode{re\-produce\-/anal\-ysis\-/make} (see Figure \ref{fig:files}) and Figure \ref{fig:datalineage} shows their inter-relation with the target/built files that they manage.

\begin{figure}[t]
  \begin{center}
    \includetikz{figure-data-lineage}
  \end{center}
  \vspace{-7mm}
  \caption{\label{fig:datalineage}Schematic representation of a project's data lineage, or workflow, for the demonstration analysis of this paper.
    Each colored box is a file in the project and the arrows show the dependencies between them.
    Green files/boxes are plain text files that are under version control and in the source directory.
    Blue files/boxes are output files of various steps in the build-directory, shown within the Makefile (\inlinecode{*.mk}) where they are defined as a \emph{target}.
    For example, \inlinecode{paper.pdf} depends on \inlinecode{project.tex} (in the build directory and generated automatically) and \inlinecode{paper.tex} (in the source directory and written by hand).
    The solid arrows and built boxes with full opacity are described in Section \ref{sec:projectanalysis}.
    The dashed arrows and lower opacity built boxes, show the scalability by adding hypothetical steps to the project.
  }
\end{figure}

To avoid getting too abstract in the subsections below, where necessary we will do a basic analysis on the data of \citet{menke20} (hereafter M20) and replicate one of the results.
Note that because we are not using the same software, this is not a reproduction (see \ref{definition:reproduction}).
We cannot use the same software because M20 use Microsoft Excel for the analysis which violates several of our principles: \ref{principle:complete}, \ref{principle:complexity} and \ref{principle:freesoftware}.
In the subsections below, this paper's analysis on that dataset is described using the data lineage graph of Figure \ref{fig:datalineage}.
We will follow Make's paradigm (see Section \ref{sec:usingmake}) of starting the lineage backwards form the ultimate target in Section  \ref{sec:paperpdf} (bottom of Figure \ref{fig:datalineage}) to the configuration files \ref{sec:configfiles} (top of Figure \ref{fig:datalineage}).
To better understand this project, we encourage looking into this paper's own Maneage source, published as a supplement.

\subsubsection{Ultimate target: the project's paper or report (\inlinecode{paper.pdf})}
\label{sec:paperpdf}

The ultimate purpose of a project is to report the data analysis result, as raw visualizations, or numbers blended in with a narrative.
In Figure \ref{fig:datalineage}, this is \inlinecode{paper.pdf}. Note that it is the only built file (blue box) with no outwards arrows leaving it.
The instructions to build \inlinecode{paper.pdf} are in the \inlinecode{paper.mk} subMakefile.
Its prerequisites include \inlinecode{paper.tex} and \inlinecode{references.tex} (Bib\TeX{} entries for possible citations) in the project source and \inlinecode{project.tex} which is a built product.
\inlinecode{references.tex} formalizes the connections of this project with previous projects on a higher level.

\subsubsection{Values within text (\inlinecode{project.tex})}
\label{sec:valuesintext}

Figures, plots, tables and narrative are not the only analysis products that are included in the paper/report.
In many cases, quantitative values from the analysis are also blended into the sentences of the report's narration.
For example, this sentence in the abstract of \citet[which is written in Maneage]{akhlaghi19}: ``\emph{... detect the outer wings of M51 down to S/N of 0.25 ...}''.
The value `0.25', for the signal-to-noise ratio (S/N), depends on the analysis, and is an output of the analysis just like paper's figures and plots.
Manually typing such numbers in the narrative is prone to very important errors and discourages testing in scientific papers.
Therefore, they must \emph{also} be automatically generated.

To automatically generate and blend them in the text, Maneage uses \LaTeX{} macros.
In the quote above, the \LaTeX{} source\footnote{\citet{akhlaghi19} is written in Maneage and its \LaTeX{} source is available in multiple ways: 1) direct download from arXiv:\href{https://arxiv.org/abs/1909.11230}{1909.11230}, by clicking on ``other formats'', or 2) the Git or \href{https://doi.org/10.5281/zenodo.3408481}{zenodo.3408481}, links are also available on arXiv's top page.} looks like this: ``\inlinecode{\small detect the outer wings of M51 down to S/N of \$\textbackslash{}demo\-sf\-optimized\-sn\$}''.
T\-he ma\-cro ``\inlinecode{\small\textbackslash{}demosfoptimizedsn}'' is automatically created during in the project and expands to the value ``\inlinecode{0.25}'' when the PDF output is built.
The built \inlinecode{project.tex} file stores all such reported values.

However, managing all the necessary \LaTeX{} macros in one file is against the modularity principle and can be frustrating and buggy.
To address this problem, Maneage adopts the convention that all subMakefiles \emph{must} contain a fixed target with the same base-name, but with a \inlinecode{.tex} suffix to store reported values generated in that subMakefile.
If it does not need to report any values in the text, the file can indeed be empty.
In Figure \ref{fig:datalineage}, these macro files can be seen in every subMakefile, except for \inlinecode{paper.mk} (which does not need it).
These \LaTeX{} macro files thus form the core skeleton of a Maneage project: as shown in Figure \ref{fig:datalineage}, the outward arrows of all built files of any subMakefile ultimately leads to one of these \LaTeX{} macro files, possibly in another subMakefile.

\subsubsection{Verification of outputs (\inlinecode{verify.mk})}
\label{sec:outputverification}

Before the modular \LaTeX{} macro files of Section \ref{sec:valuesintext} are merged into the single \inlinecode{project.tex} file, they need to pass through the verification filter, which is another core principle of Maneage (\ref{principle:verify}).
Note that simply confirming the checksum of the final PDF, or figures and datasets is not generally possible: many tools write the creation date into the produced files.
To avoid such cases the raw data must be verified, independent of metadata like date.
Some standards include such features, for example, the \inlinecode{DATASUM} keyword in the FITS format \citep{pence10}.
To facilitate output verification, Maneage has the \inlinecode{verify.mk} subMakefile that is the boundary between the analytical phase of the paper, and the production of the report (see Figure \ref{fig:datalineage}).
It has some tests on pre-defined formats, and other formats can easily be added.

\subsubsection{The analysis}
\label{sec:analysis}

The basic concepts behind organizing the analysis into modular subMakefiles have already been discussed above.
We will thus describe it here with the practical example of replicating Figure 1C of M20, with some enhancements in Figure \ref{fig:toolsperyear}.
As shown in Figure \ref{fig:datalineage}, in this project we have broken this goal into two subMakefiles: \inlinecode{format.mk} and \inlinecode{demo-plot.mk}.
The former is in charge of converting the Excel-formatted input into the simple comma-separated value (CSV) format, and the latter is in charge of generating the table to build Figure \ref{fig:toolsperyear}.
In a real project, subMakefiles could, and will, be much more complex.
Figure \ref{fig:topmake} shows how the two subMakefiles are placed as values to the \inlinecode{makesrc} variable of \inlinecode{top-make.mk}, without their suffix (see Section \ref{sec:valuesintext}).
Note that their location after the standard starting subMakefiles (initialization and download) and before the standard ending subMakefiles (verification and final paper) is important, along with their order.

\begin{figure}[t]
  \begin{center}
    \includetikz{figure-tools-per-year}
  \end{center}
  \vspace{-5mm}
  \caption{\label{fig:toolsperyear}Ratio of papers mentioning software tools (green line, left vertical axis) to total number of papers studied in that year (light red bars, right vertical axis in log-scale).
    This is an enhanced replica of figure 1C \citet{menke20}, shown here for demonstrating Maneage, see Figure \ref{fig:datalineage} for its lineage and Section \ref{sec:analysis} for how it was organized.
  }
\end{figure}

\begin{figure}[t]
  \input{tex/src/figure-src-topmake.tex}
  \vspace{-3mm}
  \caption{\label{fig:topmake} General view of the high-level \inlinecode{top-make.mk} Makefile which manages the project's analysis that is in various subMakefiles.
    See Figures \ref{fig:files} \& \ref{fig:datalineage} for its location in the project's file structure and its data lineage, as well as the subMakefiles it includes.
  }
\end{figure}

To enhance the original plot, Figure \ref{fig:toolsperyear} also shows the number of papers that were studied each year.
Furthermore, its horizontal axis shows the full range of the data (starting from \menkefirstyear), while the original starts from 1997.
This was probably because they did not have sufficient data for older papers, for example, in \menkenumpapersdemoyear, they only had \menkenumpapersdemocount{} papers.
Note that both the numbers of the previous sentence (\menkenumpapersdemoyear{} and \menkenumpapersdemocount), and the dataset's oldest year (mentioned above: \menkefirstyear) are automatically generated \LaTeX{} macros, see Section \ref{sec:valuesintext}.
They are \emph{not} typeset manually in this narrative explanation.
This step (generating the macros) is shown schematically in Figure \ref{fig:datalineage} with the arrow from \inlinecode{tools-per-year.txt} to \inlinecode{demo-plot.tex}.

To create Figure \ref{fig:toolsperyear}, we used the PGFPlots package within \LaTeX{}.
Therefore, the necessary analysis output to feed into PGFPlots was a simple plain-text table with 3 columns (year, paper per year, tool fraction per year).
This table is shown in the lineage graph of Figure \ref{fig:datalineage} as \inlinecode{tools-per-year.txt} and The PGFPlots source to generate this figure is located in \inlinecode{tex\-/src\-/figure\--tools\--per\--year\-.tex}.
If another plotting tool was desired (for example \emph{Python}'s Matplotlib, or Gnuplot), the built graphic file (for example \inlinecode{tools-per-year.pdf}) could be the target instead of the raw table.

Note that \inlinecode{tools-per-year.txt} is a value-added table with only \menkenumyears{} rows (one row for every year).
The original dataset had \menkenumorigrows{} rows (one row for each year of each journal).
We see in Figure \ref{fig:datalineage} that it is defined as a Make \emph{target} in \inlinecode{demo-plot.mk} and that its prerequisite is \inlinecode{menke20-table-3.txt} (schematically shown by the arrow connecting them).
Note that both the row numbers mentioned at the start of this paragraph are also macros.
Again from Figure \ref{fig:datalineage}, we see that \inlinecode{menke20-table-3.txt} is a target in \inlinecode{format.mk} and its prerequisite is the input file \inlinecode{menke20.xlsx}.
The input files (which come from outside the project) are all \emph{targets} in \inlinecode{download.mk} and futher discussed in Section  \ref{sec:download}.



\subsubsection{Importing and validating inputs (\inlinecode{download.mk})}
\label{sec:download}

The \inlinecode{download.mk} subMakefile is present in all Maneage projects and contains the common steps for importing the input dataset(s) into the project.
All necessary input datasets for the project are imported through this subMakefile.
Irrespective of where the dataset is \emph{used} in the project's lineage, it helps to maintain relation with the outside world (to the project) in one subMakefile (see the modularity and minimal complexity principles \ref{principle:modularity} \& \ref{principle:complexity}).

Each external dataset has some basic information, including its expected name on the local system (for offline access), the necessary checksum to validate it (either the whole file or just its main ``data'', as discussed in Section \ref{sec:outputverification}), and its URL/PID.
In Maneage, such information regarding a project's input dataset(s) is in the \inlinecode{INPUTS.conf} file.
See Figures \ref{fig:files} \& \ref{fig:datalineage} for the position of \inlinecode{INPUTS.conf} in the project's file structure and data lineage, respectively.
For demonstration, we are using the datasets of M20 which are stored in one \inlinecode{.xlsx} file on bioXriv\footnote{\label{footnote:dataurl}Full data URL: \url{\menketwentyurl}}.
Figure \ref{fig:inputconf} shows the corresponding \inlinecode{INPUTS.conf} where the necessary information are stored as Make variables and are automatically loaded into the full project when Make starts (and is most often used in \inlinecode{download.mk}).

\begin{figure}[t]
  \input{tex/src/figure-src-inputconf.tex}
  \vspace{-3mm}
  \caption{\label{fig:inputconf} The \inlinecode{INPUTS.conf} configuration file keeps references to external (input) datasets of a project, as well as their checksums for validation, see Sections \ref{sec:download} \& \ref{sec:configfiles}.
    Shown here are the entries for the demonstration dataset of \citet{menke20}.
    Note that the original URL (footnote \ref{footnote:dataurl}) was too long to display properly here.
  }
\end{figure}


\subsubsection{Configuration files}
\label{sec:configfiles}

The subMakefiles discussed above should only contain the organization of an analysis, they should not contain any fixed numbers, settings or parameters, as
such elements should only be used as variables which are defined in configuration files.
Configuration files enable the logical separation between the low-level implementation and high-level running of a project.
In the data lineage plot of Figure \ref{fig:datalineage}, configuration files are shown as the sharp-edged, green \inlinecode{*.conf} files in the top row (for example, the \inlinecode{INPUTS.conf} file that was shown in Figure \ref{fig:inputconf} and mentioned in Section \ref{sec:download}).
All the configuration files of a project are placed under the \inlinecode{reproduce/analysis/config} (see Figure \ref{fig:files}) subdirectory, and are loaded into \inlinecode{top-make.mk} before any of the subMakefiles, see Figure \ref{fig:topmake}.

The demo analysis of Section \ref{sec:analysis} is a good demonstration of their usage: during that discussion we reported the number of papers studied by M20 in \menkenumpapersdemoyear.
However, the year's number is not written by hand in \inlinecode{demo-plot.mk}.
It is referenced through the \inlinecode{menke-year-demo} variable, which is defined in \inlinecode{menke-demo-year.conf}, that is a prerequisite of the \inlinecode{demo-plot.tex} rule.
This is also visible in the data lineage of Figure \ref{fig:datalineage}.
If we later would decide to report the number in another year, we would simply have to change the value in \inlinecode{menke-demo-year.conf}.
A configuration file is a prerequisite of the target that uses it, hence its date will be newer than \inlinecode{demo-plot.tex}.
Therefore Make will re-execute the recipe to generate the macro file before this paper is re-built and the corresponding year and value will be updated in this paper, always in synchronization with each other and no matter how many times they are used.
Combined with the fact that all source files in Maneage are under version control, this encourages testing of various settings of the
analysis as the project  evolves, leading to more robust scientific results.

\subsubsection{Project initialization (\inlinecode{initialize.mk})}
\label{sec:initialize}

The \inlinecode{initial\-ize\-.mk} subMakefile is present in all projects and is the first subMakefile that is loaded into \inlinecode{top-make.mk} (see Figure \ref{fig:datalineage}).
It does not contain any analysis or major processing steps, it just initializes the system by setting the necessary Make environment as well as other general jobs like defining the Git commit hash of the run as a \LaTeX{} (\inlinecode{\textbackslash{}projectversion}) macro that can be loaded into the narrative.
Papers using Maneage usually put this hash as the last word in their abstract, for example, see \citet{akhlaghi19} and \citet{infante20}.
For the current version of this paper, it expands to \projectversion.

\subsection{Projects as Git branches of Maneage}
\label{sec:projectgit}

Maneage contains only plain-text files, and therefore it can be maintained under version control systems (currently using Git).
Every commit in the version-controlled history contains \emph{a complete} snapshot of the data lineage (for more, see the completeness principle \ref{principle:complete}).
Maneage is maintained by its developers in a central branch, which we will call \inlinecode{man\-eage} hereafter.
The \inlinecode{man\-eage} branch contains all the low-level infrastructure, or skeleton, that is necessary for any project as described in the sections above.
As mentioned in Section \ref{sec:maneage}, to start a new project, users simply clone it from its reference repository and build their own Git branch over it
This is demonstrated in Figure \ref{fig:branching}(a) where a project has started by branching-off of commit \inlinecode{0c120cb}.

%% Exact URLs of imported images.
%% Collaboration icon: https://www.flaticon.com/free-icon/collaboration_809522
%% Paper done:         https://www.flaticon.com/free-icon/file_2521838
%% Paper processing:   https://www.flaticon.com/free-icon/file_2521989
\begin{figure}[t]
  \includetikz{figure-branching}
  \vspace{-3mm}
  \caption{\label{fig:branching} Harvesting the power of version-control in project management with Maneage.
    Maneage is maintained as a core branch, with projects created by branching-off of it.
    (a) shows how projects evolve on their own branch, but can always update their low-level structure by merging with the core branch
    (b) shows how a finished/published project can be revitalized for new technologies simply by merging with the core branch.
    Each Git ``commit'' is shown on their branches as colored ellipses, with their hash printed in them.
    The commits are colored based on the team that is working on that branch.
    The collaboration and paper icons are respectively made by `mynamepong' and `iconixar' and downloaded from \url{www.flaticon.com}.
  }
\end{figure}

After a project starts, Maneage will evolve, for example, new features will be added or low-level bugs will be fixed.
Because all projects branch-off from the same branch that these infrastructure improvements are made, updating the project's low-level skeleton is as easy as merging the \inlinecode{maneage} branch into the project's branch.
For example, in Figure \ref{fig:branching}(a), see how Maneage's \inlinecode{3c05235} commit has been merged into project's branch trough commit \inlinecode{2ed0c82}.
This allows infrastructure improvements and fixes to be easily propagated to all projects.

Another useful scenario is reviving a finished/published project at later date, possibly by other researchers as shown in Figure \ref{fig:branching}(b), e.g., assuming the original project was completed years ago, and is no longer directly executable.
Other scenarios include projects that are created by merging various other projects.
Modern version control systems provide many more capabilities that can be leveraged through Maneage in project management, thanks to the shared branch it has with \emph{all} derived projects, and that it is complete (\ref{principle:complete}).

\subsection{Multi-user collaboration on single build directory}
\label{sec:collaborating}

Because the project's source and build directories are separate, it is possible for different users to share a build directory, while working on their own separate project branches during a collaboration.
Similar to the parallel branch that is later merged in Figure \ref{fig:branching}(a).
To enable this mode, \inlinecode{./project} script has a special \inlinecode{--group} option which takes the name of a (POSIX) user group in the host OS.
All files built in the build directory are then automatically assigned to this user group, with read and write permissions.
Of course, avoiding conflicts in the build directory, while members are working on different branches is up to the team.

\subsection{Publishing the project}
\label{sec:publishing}

Once the project is complete, it needs to be published.
In a scientific scenario, it is submitted to a journal, while in an industrial context it is submitted to the customers or employers.
To facilitate the publication of the project's source, Maneage has a special \inlinecode{dist} target during the build process which is activated with the command \inlinecode{./project make dist}.
In this mode, Maneage will not do any analysis, it will simply copy the full project's source (on the given commit) into a temporary directory and compress it into a \inlinecode{.tar.gz} file.
If a Zip compression is necessary, the \inlinecode{dist-zip} target can be called instead \inlinecode{dist}.
Since the complete project is in plain text, this compressed file has usually a size of around 100 kilobytes.

However, the necessary inputs (\ref{definition:input}) and outputs may be arbitrarily large, from megabytes to petabytes or more.
Therefore, there are two scenarios for the publication of the project: 1) only publishing the source, 2) publishing the source with the data.
In the former case, the output of \inlinecode{dist} (described above) can be submitted to the journal as a supplement, or uploaded to pre-print servers like arXiv that will actually compile the \LaTeX{} source and build their own PDFs.
The Git history can also be archived as a single ``bundle'' file and also submitted as a supplement.
When publishing with datasets, the project's outputs, and inputs (if necessary), can be published on servers like Zenodo.
For example, \citet[\href{https://doi.org/10.5281/zenodo.3408481}{zenodo.3408481}]{akhlaghi19} uploaded all the project's necessary software and its final PDF along with the project's source tarball and Git ``bundle'' to Zenodo.










\section{Discussion \& Caveats}
\label{sec:discussion}

Maneage was created, and has evolved during various research projects.
The primordial implementation was written for \citet{akhlaghi15}.
It later evolved in \citet{bacon17}, and in particular the two sections of that paper that were done by M. Akhlaghi: \href{http://doi.org/10.5281/zenodo.1163746}{zenodo.1163746} and \href{http://doi.org/10.5281/zenodo.1164774}{zenodo.1164774}.
With these projects, the skeleton of the system was written as a more abstract ``template'' that could be customized for separate projects.
That template later matured into Maneage by including the installation of all necessary software from source and it was used in \citet[\href{https://doi.org/10.5281/zenodo.3408481}{zenodo.3408481}]{akhlaghi19} and \citet[\href{https://doi.org/10.5281/zenodo.3524937}{zenodo.3524937}]{infante20}.
Bugs will still be found and Maneage will continue to evolve after this paper is published.
A list of the notable changes after the publication of this paper will be kept in the \inlinecode{README-hacking.md} file.

Once Maneage is adopted on a wide scale in a special topic, they can be fed them into machine learning algorithms for automatic workflow generation, optimized for certain aspects of the result.
Because Maneage is complete, even inputs (software and input data), or failed tests during the projects can enter this optimization process.
Furthermore, writing parsers of Maneage projects to generate Research Objects is trivial, and very useful for meta-research and data provenance studies.
Combined with Software Heritage \citep{dicosmo18}, precise parts Maneage projects (high-level science) can be cited, at various points in its history (e.g., failed/abandoned tests).
Many components of Machine actionable data management plans \citep{miksa19b} can also be automatically filled with Maneage, greatly helping the project PI and and grant organizations.

Maneage was awarded a Research Data Alliance (RDA) adoption grant for implementing the recommendations of the Publishing Data Workflows working group \citep{austin17}.
Its user base, and thus its development, grew phenomenally afterwards and highlighted some caveats.
The first is that Maneage uses very low-level tools that are not widely used by scientists, e.g., Git, \LaTeX, Make and the command-line.
We have discovered that this is primarily because of a lack of exposure.
Many (in particular early career researchers) have started mastering them as they adopt Maneage once they witness their advantages, but it does take time.

A second caveat is the fact that Maneage is effectively an almost complete GNU OS, tailored to each project.
Maintaining the various packages is time consuming for us (Maneage maintainers).
However, because software installation is also in Make, some users are already adding their necessary software to the core Maneage branch, thus propagating the improvements to all projects using Maneage.
Another caveat that has been raised is that publishing the project's reproducible data lineage immediately after publication may hamper their ability to continue with followup papers because others may do it before them.
We propose these solutions:
1) Through the Git history, the added work by another team, at any phase of the project, can be quantified, contributing to a new concept of authorship in scientific projects and helping to quantify Newton's famous ``\emph{standing on the shoulders of giants}'' quote.
This is however a long-term goal and requires major changes to academic value systems.
2) Authors can be given a grace period where the journal, or some third authority, keeps the source and publishes it a certain interval after publication.










\section{Conclusion \& Summary}
\label{sec:conclusion}

To effectively leverage the power of big data, we need to have a complete view of its lineage.
Scientists are however rarely trained sufficiently in data management or software development, and the plethora of high-level tools that change every few years does not help.
Such high-level tools are primarily targetted at software developers, who are paid to learn them and use them effectively for short-term projects.
Scientists on the other hand need to focus on their own research fields, and need to think about longevity.

Maneage is designed as a complete template, providing scientists with a built low-level skeleton, using simple and robust tools that have withstood the test of time while being actively developed.
Scientists can customize its existing data management for their own projects, enabling them to learn and master the lower-level tools in the meantime.
This improves their efficiency and the robustness of their scientific result, while also enabling future scientists to reproduce and build-upon their work.

We discussed the founding principles of Maneage that are completeness, modularity, minimal complexity, verifiable inputs and outputs, temporal provenance, and free software.
We showed how these principles are implemented in an already built structure, ready for customization and discussed the caveats and advantages of this implementation.
With a larger user-base and wider application in scientific (and hopefully industrial) applications, Maneage will certainly grow and become even more robust, stable and user friendly.










%% Acknowledgements
\section*{Acknowledgments}
The authors wish to thank (sorted alphabetically)
Julia Aguilar-Cabello,
Alice Allen,
Pedram Ashofteh Ardakani,
Roland Bacon,
Surena Fatemi,
Konrad Hinsen,
Mohammad-reza Khellat,
Johan Knapen,
Ryan O'Connor,
Simon Portegies Zwart,
Elham Saremi,
Yahya Sefidbakht,
Zahra Sharbaf,
and Ignacio Trujillo
for their useful help, suggestions and feedback on Maneage and this paper.

During its development, Maneage has been partially funded/supported by the following institutions:
The Japanese Ministry of Education, Culture, Sports, Science, and Technology ({\small MEXT}) PhD scholarship to M. Akhl\-aghi and its Grant-in-Aid for Scientific Research (21244012, 24253003).
The European Research Council (ERC) advanced grant 339659-MUSICOS.
The European Union (EU) Horizon 2020 (H2020) research and innovation programmes No 777388 under RDA EU 4.0 project, and Marie Sk\l{}odowska-Curie grant agreement No 721463 to the SUNDIAL ITN.
The State Research Agency (AEI) of the Spanish Ministry of Science, Innovation and Universities (MCIU) and the European Regional Development Fund (ERDF) under the grant AYA2016-76219-P.
The IAC project P/300724, financed by the MCIU, through the Canary Islands Department of Economy, Knowledge and Employment.
The Fundaci\'on BBVA under its 2017 programme of assistance to scientific research groups, for the project ``Using machine-learning techniques to drag galaxies from the noise in deep imaging''.
The ``A next-generation worldwide quantum sensor network with optical atomic clocks'' project of the TEAM IV programme of the Foundation for Polish Science co-financed by the EU under ERDF.
The Polish MNiSW grant DIR/WK/2018/12.
The Pozna\'n Supercomputing and Networking Center (PSNC) computational grant 314.


\input{tex/build/macros/dependencies.tex}

\section*{Competing Interests}
The authors have no competing interests to declare.

\section*{Author Contributions}
\begin{enumerate}
\item Mohammad Akhlaghi: principal author of the Maneage source code and this paper, also principal investigator (PI) of the RDA Adoption grant awarded to Maneage.
\item Ra\'ul Infante-Sainz: contributed many commits to the source of Maneage, also involved in early testing and writing this paper.
\item Boudewijn F. Roukema: involved in Maneage and its testing, contributed to writing this paper.
\item David Valls-Gabaud: involved in the Maneage project and its testing and contributed to writing this paper.
\item Roberto Baena-Gall\'e: contributed to early testing of Maneage and in writing this paper.
\end{enumerate}

%% Tell BibLaTeX to put the bibliography list here.
\printbibliography

%% Finish LaTeX
\end{document}

%% This file is part of a paper describing the Maneage workflow system
%%   https://gitlab.com/makhlaghi/maneage-paper
%
%% This file is free software: you can redistribute it and/or modify it
%% under the terms of the GNU General Public License as published by the
%% Free Software Foundation, either version 3 of the License, or (at your
%% option) any later version.
%
%% This file is distributed in the hope that it will be useful, but WITHOUT
%% ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
%% for more details.
%
%% You should have received a copy of the GNU General Public License along
%% with this file.  If not, see <https://www.gnu.org/licenses/>.