diff options
Diffstat (limited to 'tex/src/appendix-existing-solutions.tex')
| -rw-r--r-- | tex/src/appendix-existing-solutions.tex | 108 |
1 files changed, 54 insertions, 54 deletions
diff --git a/tex/src/appendix-existing-solutions.tex b/tex/src/appendix-existing-solutions.tex index 8bbbe1c..62c98e6 100644 --- a/tex/src/appendix-existing-solutions.tex +++ b/tex/src/appendix-existing-solutions.tex @@ -23,7 +23,7 @@ \section{Survey of common existing reproducible workflows} \label{appendix:existingsolutions} The problem of reproducibility has received considerable attention over the last three decades and various solutions have already been proposed. -The core principles that many of the existing solutions (including Maneage) aim to achieve are nicely summarized by the FAIR principles \citeappendix{wilkinson16}. +The core principles that many of the existing solutions (including Maneage) aim to achieve are nicely summarized by the FAIR principles\citeappendix{wilkinson16}. In this appendix, \emph{some} of the solutions are reviewed. We are not just reviewing solutions that can be used today. The main focus of this paper is longevity, therefore we also spent considerable time on finding and inspecting solutions that have been aborted, discontinued or abandoned. @@ -33,7 +33,7 @@ Otherwise their paper's publication year is used. For each solution, we summarize its methodology and discuss how it relates to the criteria proposed in this paper. Freedom of the software/method is a core concept behind scientific reproducibility, as opposed to industrial reproducibility where a black box is acceptable/desirable. Therefore proprietary solutions like Code Ocean\footnote{\inlinecode{\url{https://codeocean.com}}} or Nextjournal\footnote{\inlinecode{\url{https://nextjournal.com}}} will not be reviewed here. -Other studies have also attempted to review existing reproducible solutions, for example, see \citeappendix{konkol20}. +Other studies have also attempted to review existing reproducible solutions, for example, see Konkol et al.\citeappendix{konkol20}. We have tried our best to test and read through the documentation of almost all reviewed solutions to a sufficient level. However, due to time constraints, it is inevitable that we may have missed some aspects the solutions, or incorrectly interpreted their behavior and outputs. @@ -43,12 +43,12 @@ In this case, please let us know and we will correct it in the text on the paper \subsection{Suggested rules, checklists, or criteria} Before going into the various implementations, it is useful to review some existing suggested rules, checklists, or criteria for computationally reproducible research. -Sandve et al. \citeappendix{sandve13} propose ``ten simple rules for reproducible computational research'' that can be applied in any project. +Sandve et al.\citeappendix{sandve13} propose ``ten simple rules for reproducible computational research'' that can be applied in any project. Generally, these are very similar to the criteria proposed here and follow a similar spirit, but they do not provide any actual research papers following up all those points, nor do they provide a proof of concept. -The Popper convention \citeappendix{jimenez17} also provides a set of principles that are indeed generally useful, among which some are common to the criteria here (for example, automatic validation, and, as in Maneage, the authors suggest providing a template for new users), but the authors do not include completeness as a criterion nor pay attention to longevity: Popper has already changed its core workflow language once and is written in Python with many dependencies that evolve fast, see \ref{appendix:highlevelinworkflow}. +The Popper convention\citeappendix{jimenez17} also provides a set of principles that are indeed generally useful, among which some are common to the criteria here (for example, automatic validation, and, as in Maneage, the authors suggest providing a template for new users), but the authors do not include completeness as a criterion nor pay attention to longevity: Popper has already changed its core workflow language once and is written in Python with many dependencies that evolve fast, see \ref{appendix:highlevelinworkflow}. For more on Popper, please see Section \ref{appendix:popper}. -For improved reproducibility Jupyter notebooks, \citeappendix{rule19} propose ten rules and also provide links to example implementations. +For improved reproducibility Jupyter notebooks, Rule et al.\citeappendix{rule19} propose ten rules and also provide links to example implementations. These can be very useful for users of Jupyter but are not generic for non-Jupyter-based computational projects. Some criteria (which are indeed very good in a more general context) do not directly relate to reproducibility, for example their Rule 1: ``Tell a Story for an Audience''. Generally, as reviewed in @@ -58,7 +58,7 @@ the main body of this paper (section on the longevity of existing tools)% Section \ref{sec:longevityofexisting}% \fi and Section \ref{appendix:jupyter} (below), Jupyter itself has many issues regarding reproducibility. -To create Docker images, N\"ust et al. propose ``ten simple rules'' in \citeappendix{nust20}. +To create Docker images, N\"ust et al. propose\citeappendix{nust20} ``ten simple rules''. They recommend some issues that can indeed help increase the quality of Docker images and their production/usage, such as their rule 7 to ``mount datasets [only] at run time'' to separate the computational environment from the data. However, the long-term reproducibility of the images is not included as a criterion by these authors. For example, they recommend using base operating systems, with version identification limited to a single brief identifier such as \inlinecode{ubuntu:18.04}, which has a serious problem with longevity issues @@ -70,21 +70,21 @@ For example, they recommend using base operating systems, with version identific Furthermore, in their proof-of-concept Dockerfile (listing 1), \inlinecode{rocker} is used with a tag (not a digest), which can be problematic due to the high risk of ambiguity (as discussed in Section \ref{appendix:containers}). Previous criteria are thus primarily targeted to immediate reproducibility and do not consider longevity. -Therefore, they lack a strong/clear completeness criterion (they mainly only suggest, rather than require, the recording of versions, and their ultimate suggestion of storing the full binary OS in a binary VM or container is problematic (as mentioned in \ref{appendix:independentenvironment} and \citeappendix{oliveira18}). +Therefore, they lack a strong/clear completeness criterion (they mainly only suggest, rather than require, the recording of versions, and their ultimate suggestion of storing the full binary OS in a binary VM or container is problematic (as mentioned in \ref{appendix:independentenvironment} and Oliveira et al.\citeappendix{oliveira18}). \subsection{Reproducible Electronic Documents, RED (1992)} \label{appendix:red} -RED\footnote{\inlinecode{\url{http://sep.stanford.edu/doku.php?id=sep:research:reproducible}}} is the first attempt that we could find on doing reproducible research, see \cite{claerbout1992,schwab2000}. +RED\footnote{\inlinecode{\url{http://sep.stanford.edu/doku.php?id=sep:research:reproducible}}} is the first attempt\cite{claerbout1992,schwab2000} that we could find on doing reproducible research. It was developed within the Stanford Exploration Project (SEP) for Geophysics publications. Their introductions on the importance of reproducibility, resonate a lot with today's environment in computational sciences. In particular, the heavy investment one has to make in order to re-do another scientist's work, even in the same team. -RED also influenced other early reproducible works, for example \citeappendix{buckheit1995}. +RED also influenced other early reproducible works, for example Buckheit \& Donoho\citeappendix{buckheit1995}. -To orchestrate the various figures/results of a project, from 1990, they used ``Cake'' \citeappendix{somogyi87}, a dialect of Make, for more on Make, see Appendix \ref{appendix:jobmanagement}. -As described in \cite{schwab2000}, in the latter half of that decade, they moved to GNU Make, which was much more commonly used, better maintained, and came with a complete and up-to-date manual. +To orchestrate the various figures/results of a project, from 1990, they used ``Cake''\citeappendix{somogyi87}, a dialect of Make, for more on Make, see Appendix \ref{appendix:jobmanagement}. +As described in Schwab et al.\cite{schwab2000}, in the latter half of that decade, they moved to GNU Make, which was much more commonly used, better maintained, and came with a complete and up-to-date manual. The basic idea behind RED's solution was to organize the analysis as independent steps, including the generation of plots, and organizing the steps through a Makefile. This enabled all the results to be re-executed with a single command. Several basic low-level Makefiles were included in the high-level/central Makefile. @@ -94,7 +94,7 @@ The reader could later select which figures/parts of the project to reproduce by At the time, Make was already practiced by individual researchers and projects as a job orchestration tool, but SEP's innovation was to standardize it as an internal policy, and define conventions for the Makefiles to be consistent across projects. This enabled new members to benefit from the already existing work of previous team members (who had graduated or moved to other jobs). However, RED only used the existing software of the host system, it had no means to control them. -Therefore, with wider adoption, they confronted a ``versioning problem'' where the host's analysis software had different versions on different hosts, creating different results, or crashing \citeappendix{fomel09}. +Therefore, with wider adoption, they confronted a ``versioning problem'' where the host's analysis software had different versions on different hosts, creating different results, or crashing\citeappendix{fomel09}. Hence in 2006 SEP moved to a new Python-based framework called Madagascar, see Appendix \ref{appendix:madagascar}. @@ -103,7 +103,7 @@ Hence in 2006 SEP moved to a new Python-based framework called Madagascar, see A \subsection{Taverna (2003)} \label{appendix:taverna} -Taverna\footnote{\inlinecode{\url{https://github.com/taverna}}} \citeappendix{oinn04} was a workflow management system written in Java with a graphical user interface. +Taverna\footnote{\inlinecode{\url{https://github.com/taverna}}}\citeappendix{oinn04} was a workflow management system written in Java with a graphical user interface. In 2014 it was sponsored by the Apache Incubator project and called ``Apache Taverna'', but its developers \href{https://lists.apache.org/thread.html/r559e0dd047103414fbf48a6ce1bac2e17e67504c546300f2751c067c\%40\%3Cdev.taverna.apache.org\%3E}{voted} to \emph{retire} it in 2020 because development has come to a standstill (as of April 2021, latest public Github commit was in 2016). In Taverna, a workflow is defined as a directed graph, where nodes are called ``processors''. @@ -116,7 +116,7 @@ lineage figure of the main paper). Figure \ref{fig:datalineage}). \fi Taverna is only a workflow manager and is not integrated with a package manager, hence the versions of the used software can be different in different runs. -Ref.\/~\citeappendix{zhao12} studied the problem of workflow decays in Taverna. +Zhao et al. \citeappendix{zhao12} studied the problem of workflow decays in Taverna. @@ -124,9 +124,9 @@ Ref.\/~\citeappendix{zhao12} studied the problem of workflow decays in Taverna. \subsection{Madagascar (2003)} \label{appendix:madagascar} -Madagascar\footnote{\inlinecode{\url{http://ahay.org}}} \citeappendix{fomel13} is a set of extensions to the SCons job management tool (reviewed in \ref{appendix:scons}). +Madagascar\footnote{\inlinecode{\url{http://ahay.org}}}\citeappendix{fomel13} is a set of extensions to the SCons job management tool (reviewed in \ref{appendix:scons}). Madagascar is a continuation of the Reproducible Electronic Documents (RED) project that was discussed in Appendix \ref{appendix:red}. -Madagascar has been used in the production of hundreds of research papers or book chapters\footnote{\inlinecode{\url{http://www.ahay.org/wiki/Reproducible_Documents}}}, 120 prior to \citeappendix{fomel13}. +Madagascar has been used in the production of hundreds of research papers or book chapters\footnote{\inlinecode{\url{http://www.ahay.org/wiki/Reproducible_Documents}}}, 120 prior to Fomel et al.\citeappendix{fomel13}. Madagascar does include project management tools in the form of SCons extensions. However, it is not just a reproducible project management tool. @@ -149,17 +149,17 @@ Furthermore, the blending of the workflow component with the low-level analysis \subsection{GenePattern (2004)} \label{appendix:genepattern} -GenePattern\footnote{\inlinecode{\url{https://www.genepattern.org}}} \citeappendix{reich06} (first released in 2004) is a client-server software containing many common analysis functions/modules, primarily focused for Gene studies. +GenePattern\footnote{\inlinecode{\url{https://www.genepattern.org}}}\citeappendix{reich06} (first released in 2004) is a client-server software containing many common analysis functions/modules, primarily focused for Gene studies. Although it is highly focused to a special research field, it is reviewed here because its concepts/methods are generic. Its server-side software is installed with fixed software packages that are wrapped into GenePattern modules. -The modules are used through a web interface, the modern implementation is GenePattern Notebook \citeappendix{reich17}. +The modules are used through a web interface, the modern implementation is GenePattern Notebook\citeappendix{reich17}. It is an extension of the Jupyter notebook (see Appendix \ref{appendix:editors}), which also has a special ``GenePattern'' cell that will connect to GenePattern servers for doing the analysis. However, the wrapper modules just call an existing tool on the running system. Given that each server may have its own set of installed software, the analysis may differ (or crash) when run on different GenePattern servers, hampering reproducibility. %% GenePattern shutdown announcement (although as of November 2020, it does not open any more): https://www.genepattern.org/blog/2019/10/01/the-genomespace-project-is-ending-on-november-15-2019 -The primary GenePattern server was active since 2008 and had 40,000 registered users with 2000 to 5000 jobs running every week \citeappendix{reich17}. +The primary GenePattern server was active since 2008 and had 40,000 registered users with 2000 to 5000 jobs running every week\citeappendix{reich17}. However, it was shut down on November 15th 2019 due to the end of funding. All processing with this sever has stopped, and any archived data on it has been deleted. Since GenePattern is free software, there are alternative public servers to use, so hopefully, work on it will continue. @@ -173,14 +173,14 @@ The data and software may have backups in other places, but the high-level proje \subsection{Kepler (2005)} -Kepler\footnote{\inlinecode{\url{https://kepler-project.org}}} \citeappendix{ludascher05} is a Java-based Graphic User Interface workflow management tool. +Kepler\footnote{\inlinecode{\url{https://kepler-project.org}}}\citeappendix{ludascher05} is a Java-based Graphic User Interface workflow management tool. Users drag-and-drop analysis components, called ``actors'', into a visual, directional graph, which is the workflow (similar to \ifdefined\separatesupplement the lineage figure shown in the main paper). \else Figure \ref{fig:datalineage}). \fi -Each actor is connected to others through Ptolemy II\footnote{\inlinecode{\url{https://ptolemy.berkeley.edu}}} \citeappendix{eker03}. +Each actor is connected to others through Ptolemy II\footnote{\inlinecode{\url{https://ptolemy.berkeley.edu}}}\citeappendix{eker03}. In many aspects, the usage of Kepler and its issues for long-term reproducibility is like Taverna (see Section \ref{appendix:taverna}). @@ -189,7 +189,7 @@ In many aspects, the usage of Kepler and its issues for long-term reproducibilit \subsection{VisTrails (2005)} \label{appendix:vistrails} -VisTrails\footnote{\inlinecode{\url{https://www.vistrails.org}}} \citeappendix{bavoil05} was a graphical workflow managing system. +VisTrails\footnote{\inlinecode{\url{https://www.vistrails.org}}}\citeappendix{bavoil05} was a graphical workflow managing system. According to its web page, VisTrails maintenance has stopped since May 2016, its last Git commit, as of this writing, was in November 2017. However, given that it was well maintained for over 10 years is an achievement. @@ -197,7 +197,7 @@ VisTrails (or ``visualization trails'') was initially designed for managing visu Each analysis step, or module, is recorded in an XML schema, which defines the operations and their dependencies. The XML attributes of each module can be used in any XML query language to find certain steps (for example those that used a certain command). Since the main goal was visualization (as images), apparently its primary output is in the form of image spreadsheets. -Its design is based on a change-based provenance model using a custom VisTrails provenance query language (vtPQL), for more see \citeappendix{scheidegger08}. +Its design is based on a change-based provenance model using a custom VisTrails provenance query language (vtPQL), for more see Scheidegger et al.\citeappendix{scheidegger08}. Since XML is a plain text format, as the user inspects the data and makes changes to the analysis, the changes are recorded as ``trails'' in the project's VisTrails repository that operates very much like common version control systems (see Appendix \ref{appendix:versioncontrol}). . However, even though XML is in plain text, it is very hard to read/edit without the VisTrails software (which is no longer maintained). @@ -215,7 +215,7 @@ Besides the fact that it is no longer maintained, VisTrails did not control the \subsection{Galaxy (2010)} \label{appendix:galaxy} -Galaxy\footnote{\inlinecode{\url{https://galaxyproject.org}}} is a web-based Genomics workbench \citeappendix{goecks10}. +Galaxy\footnote{\inlinecode{\url{https://galaxyproject.org}}} is a web-based Genomics workbench\citeappendix{goecks10}. The main user interface is the ``Galaxy Pages'', which does not require any programming: users graphically manipulate abstract ``tools'' which are wrappers over command-line programs. Therefore the actual running version of the program can be hard to control across different Galaxy servers. Besides the automatically generated metadata of a project (which include version control, or its history), users can also tag/annotate each analysis step, describing its intent/purpose. @@ -228,7 +228,7 @@ For example the very large cost of maintaining such a system, being based on a g \subsection{Image Processing On Line journal, IPOL (2010)} \label{appendix:ipol} -The IPOL journal\footnote{\inlinecode{\url{https://www.ipol.im}}} \citeappendix{limare11} (first published article in July 2010) publishes papers on image processing algorithms as well as the the full code of the proposed algorithm. +The IPOL journal\footnote{\inlinecode{\url{https://www.ipol.im}}}\citeappendix{limare11} (first published article in July 2010) publishes papers on image processing algorithms as well as the the full code of the proposed algorithm. An IPOL paper is a traditional research paper, but with a focus on implementation. The published narrative description of the algorithm must be detailed to a level that any specialist can implement it in their own programming language (extremely detailed). The author's own implementation of the algorithm is also published with the paper (in C, C++ or MATLAB/Octave and recently Python), the code can only have a very limited set of external dependencies (with pre-defined versions), must be commented well enough, and link each part of it with the relevant part of the paper. @@ -251,7 +251,7 @@ A paper written in Maneage (the proof-of-concept solution presented in this pape \subsection{WINGS (2010)} \label{appendix:wings} -WINGS\footnote{\inlinecode{\url{https://wings-workflows.org}}} \citeappendix{gil10} is an automatic workflow generation algorithm. +WINGS\footnote{\inlinecode{\url{https://wings-workflows.org}}}\citeappendix{gil10} is an automatic workflow generation algorithm. It runs on a centralized web server, requiring many dependencies (such that it is recommended to download Docker images). It allows users to define various workflow components (for example datasets, analysis components, etc), with high-level goals. It then uses selection and rejection algorithms to find the best components using a pool of analysis components that can satisfy the requested high-level constraints. @@ -264,15 +264,15 @@ It then uses selection and rejection algorithms to find the best components usin \subsection{Active Papers (2011)} \label{appendix:activepapers} Active Papers\footnote{\inlinecode{\url{http://www.activepapers.org}}} attempts to package the code and data of a project into one file (in HDF5 format). -It was initially written in Java because its compiled byte-code outputs in JVM are portable on any machine \citeappendix{hinsen11}. -However, Java is not a commonly used platform today, hence it was later implemented in Python \citeappendix{hinsen15}. +It was initially written in Java because its compiled byte-code outputs in JVM are portable on any machine\citeappendix{hinsen11}. +However, Java is not a commonly used platform today, hence it was later implemented in Python\citeappendix{hinsen15}. Dependence on high-level platforms (Java or Python) is therefore a fundamental issue. In the Python version, all processing steps and input data (or references to them) are stored in an HDF5 file. %However, it can only account for pure-Python packages using the host operating system's Python modules \tonote{confirm this!}. When the Python module contains a component written in other languages (mostly C or C++), it needs to be an external dependency to the Active Paper. -As mentioned in \citeappendix{hinsen15}, the fact that it relies on HDF5 is a caveat of Active Papers, because many tools are necessary to merely open it. +As mentioned in Hinsen\citeappendix{hinsen15}, the fact that it relies on HDF5 is a caveat of Active Papers, because many tools are necessary to merely open it. Downloading the pre-built ``HDF View'' binaries (a GUI browser of HDF5 files that is provided by the HDF group) is not possible anonymously/automatically: as of January 2021 login is required\footnote{\inlinecode{\url{https://www.hdfgroup.org/downloads/hdfview}}} (this was not the case when Active Papers moved to HDF5). % From K. Hinsen in a private email to M. Akhlaghi: This is true today, but wasn't when I started ActivePapers. Otherwise I'd never have built on HDF5. Installing HDF View using the Debian or Arch Linux package managers also failed due to dependencies in our trials. @@ -280,7 +280,7 @@ Furthermore, like most high-level tools, the HDF5 library evolves very fast: on While data and code are indeed fundamentally similar concepts technically\citeappendix{hinsen16}, they are used by humans differently. The hand-written code of a large project involving Terabytes of data can be 100 kilo bytes. -When the two are bundled together in one remote file, merely seeing one line of the code, requires downloading Terabytes volume that is not needed, this was also acknowledged in \citeappendix{hinsen15}. +When the two are bundled together in one remote file, merely seeing one line of the code, requires downloading Terabytes volume that is not needed, this was also acknowledged in Hinsen\citeappendix{hinsen15}. It may also happen that the data are proprietary (for example medical patient data). In such cases, the data must not be publicly released, but the methods that were applied to them can. @@ -294,13 +294,13 @@ This is not a fundamental feature of the approach, but rather an effect of the i \subsection{Collage Authoring Environment (2011)} \label{appendix:collage} -The Collage Authoring Environment \citeappendix{nowakowski11} was the winner of Elsevier Executable Paper Grand Challenge \citeappendix{gabriel11}. +The Collage Authoring Environment\citeappendix{nowakowski11} was the winner of Elsevier Executable Paper Grand Challenge\citeappendix{gabriel11}. It is based on the GridSpace2\footnote{\inlinecode{\url{http://dice.cyfronet.pl}}} distributed computing environment, which has a web-based graphic user interface. Through its web-based interface, viewers of a paper can actively experiment with the parameters of a published paper's displayed outputs (for example figures) through a web interface. In their Figure 3, they nicely vizualize how the ``Executable Paper'' of Collage operates through two servers and a computing backend. Unfortunately in the paper no webpage has been provided follow up on the work and find its current status. -A web search also only pointed us to its main paper (\citeappendix{nowakowski11}). +A web search also only pointed us to its main paper\citeappendix{nowakowski11}. In the paper they do not discuss the major issue of software versioning and its verification to ensure that future updates to the backend do not affect the result; apparently it just assumes the software exist on the ``Computing backend''. Since we could not access or test it, from the descriptions in the paper, it seems to be very similar to the modern day Jupyter notebook concept (see \ref{appendix:jupyter}), which had not yet been created in its current form in 2011. So we expect similar longevity issues with Collage. @@ -308,8 +308,8 @@ So we expect similar longevity issues with Collage. \subsection{SHARE (2011)} \label{appendix:SHARE} -SHARE\footnote{\inlinecode{\url{https://is.ieis.tue.nl/staff/pvgorp/share}}} \citeappendix{vangorp11} is a web portal that hosts virtual machines (VMs) for storing the environment of a research project. -SHARE was recognized as the second position in the Elsevier Executable Paper Grand Challenge \citeappendix{gabriel11}. +SHARE\footnote{\inlinecode{\url{https://is.ieis.tue.nl/staff/pvgorp/share}}}\citeappendix{vangorp11} is a web portal that hosts virtual machines (VMs) for storing the environment of a research project. +SHARE was recognized as the second position in the Elsevier Executable Paper Grand Challenge\citeappendix{gabriel11}. Simply put, SHARE was just a VM library that users could download or connect to, and run. The limitations of VMs for reproducibility were discussed in Appendix \ref{appendix:virtualmachines}, and the SHARE system does not specify any requirements or standards on making the VM itself reproducible, or enforcing common internals for its supported projects. As of January 2021, the top SHARE web page still works. @@ -321,13 +321,13 @@ However, upon selecting any operation, a notice is printed that ``SHARE is offli \subsection{Verifiable Computational Result, VCR (2011)} \label{appendix:verifiableidentifier} -A ``verifiable computational result''\footnote{\inlinecode{\url{http://vcr.stanford.edu}}} is an output (table, figure, etc) that is associated with a ``verifiable result identifier'' (VRI), see \citeappendix{gavish11}. -It was awarded the third prize in the Elsevier Executable Paper Grand Challenge \citeappendix{gabriel11}. +A ``verifiable computational result''\footnote{\inlinecode{\url{http://vcr.stanford.edu}}} is an output (table, figure, etc) that is associated with a ``verifiable result identifier'' (VRI), see\citeappendix{gavish11}. +It was awarded the third prize in the Elsevier Executable Paper Grand Challenge\citeappendix{gabriel11}. A VRI is a hash that is created using tags within the programming source that produced that output, also recording its version control or history. This enables the exact identification and citation of results. The VRIs are automatically generated web-URLs that link to public VCR repositories containing the data, inputs, and scripts, that may be re-executed. -According to \citeappendix{gavish11}, the VRI generation routine has been implemented in MATLAB, R, and Python, although only the MATLAB version was available on the webpage in January 2021. +According to Gavish \& Donoho\citeappendix{gavish11}, the VRI generation routine has been implemented in MATLAB, R, and Python, although only the MATLAB version was available on the webpage in January 2021. VCR also has special \LaTeX{} macros for loading the respective VRI into the generated PDF. In effect this is very similar to what have done at the end of the caption of \ifdefined\separatesupplement @@ -340,7 +340,7 @@ However, instead of a long and hard to read hash, we simply point to the plotted Unfortunately, most parts of the web page are not complete as of January 2021. The VCR web page contains an example PDF\footnote{\inlinecode{\url{http://vcr.stanford.edu/paper.pdf}}} that is generated with this system, but the linked VCR repository\footnote{\inlinecode{\url{http://vcr-stat.stanford.edu}}} did not exist (again, as of January 2021). -Finally, the date of the files in the MATLAB extension tarball is set to May 2011, hinting that probably VCR has been abandoned soon after the publication of \citeappendix{gavish11}. +Finally, the date of the files in the MATLAB extension tarball is set to May 2011, hinting that probably VCR has been abandoned soon after the publication of Gavish \& Donoho\citeappendix{gavish11}. @@ -348,13 +348,13 @@ Finally, the date of the files in the MATLAB extension tarball is set to May 201 \subsection{SOLE (2012)} \label{appendix:sole} -SOLE (Science Object Linking and Embedding) defines ``science objects'' (SOs) that can be manually linked with phrases of the published paper \citeappendix{pham12,malik13}. +SOLE (Science Object Linking and Embedding) defines ``science objects'' (SOs) that can be manually linked with phrases of the published paper\citeappendix{pham12,malik13}. An SO is any code/content that is wrapped in begin/end tags with an associated type and name. For example, special commented lines in a Python, R, or C program. The SOLE command-line program parses the tagged file, generating metadata elements unique to the SO (including its URI). -SOLE also supports workflows as Galaxy tools \citeappendix{goecks10}. +SOLE also supports workflows as Galaxy tools\citeappendix{goecks10}. -For reproducibility, \citeappendix{pham12} suggest building a SOLE-based project in a virtual machine, using any custom package manager that is hosted on a private server to obtain a usable URI. +For reproducibility, Pham et al. \citeappendix{pham12} suggest building a SOLE-based project in a virtual machine, using any custom package manager that is hosted on a private server to obtain a usable URI. However, as described in Appendices \ref{appendix:independentenvironment} and \ref{appendix:packagemanagement}, unless virtual machines are built with robust package managers, this is not a sustainable solution (the virtual machine itself is not reproducible). Also, hosting a large virtual machine server with fixed IP on a hosting service like Amazon (as suggested there) for every project in perpetuity will be very expensive. @@ -366,7 +366,7 @@ In Maneage, instead of using artificial/commented tags, the analysis inputs and \subsection{Sumatra (2012)} -Sumatra\footnote{\inlinecode{\url{http://neuralensemble.org/sumatra}}} \citeappendix{davison12} attempts to capture the environment information of a running project. +Sumatra\footnote{\inlinecode{\url{http://neuralensemble.org/sumatra}}}\citeappendix{davison12} attempts to capture the environment information of a running project. It is written in Python and is a command-line wrapper over the analysis script. By controlling a project at running-time, Sumatra is able to capture the environment it was run in. The captured environment can be viewed in plain text or a web interface. @@ -385,10 +385,10 @@ It just captures the environment, it does not store \emph{how} that environment \subsection{Research Object (2013)} \label{appendix:researchobject} -The Research object\footnote{\inlinecode{\url{http://www.researchobject.org}}} is collection of meta-data ontologies, to describe aggregation of resources, or workflows, see \citeappendix{bechhofer13} and \citeappendix{belhajjame15}. +The Research object\footnote{\inlinecode{\url{http://www.researchobject.org}}} is collection of meta-data ontologies, to describe aggregation of resources, or workflows\citeappendix{bechhofer13,belhajjame15}. It thus provides resources to link various workflow/analysis components (see Appendix \ref{appendix:existingtools}) into a final workflow. -Ref.\/~\citeappendix{bechhofer13} describes how a workflow in Taverna (Appendix \ref{appendix:taverna}) can be translated into research objects. +Bechhofer et al. \citeappendix{bechhofer13} describes how a workflow in Taverna (Appendix \ref{appendix:taverna}) can be translated into research objects. The important thing is that the research object concept is not specific to any special workflow, it is just a metadata bundle/standard which is only as robust in reproducing the result as the running workflow. Therefore if implemented over a complete workflow like Maneage, it can be very useful in analysing/optimizing the workflow, finding common components between many Maneage'd workflows, or translating to other complete workflows. @@ -398,7 +398,7 @@ Therefore if implemented over a complete workflow like Maneage, it can be very u \subsection{Sciunit (2015)} \label{appendix:sciunit} -Sciunit\footnote{\inlinecode{\url{https://sciunit.run}}} \citeappendix{meng15} defines ``sciunit''s that keep the executed commands for an analysis and all the necessary programs and libraries that are used in those commands. +Sciunit\footnote{\inlinecode{\url{https://sciunit.run}}}\citeappendix{meng15} defines ``sciunit''s that keep the executed commands for an analysis and all the necessary programs and libraries that are used in those commands. It automatically parses all the executable files in the script and copies them, and their dependency libraries (down to the C library), into the sciunit. Because the sciunit contains all the programs and necessary libraries, it is possible to run it readily on other systems that have a similar CPU architecture. Sciunit was originally written in Python 2 (which reached its end-of-life on January 1st, 2020). @@ -412,17 +412,17 @@ This is a major problem for scientific projects: in principle (not knowing how t \subsection{Umbrella (2015)} -Umbrella \citeappendix{meng15b} is a high-level wrapper script for isolating the environment of the analysis. +Umbrella\citeappendix{meng15b} is a high-level wrapper script for isolating the environment of the analysis. The user specifies the necessary operating system, and necessary packages for the analysis steps in various JSON files. Umbrella will then study the host operating system and the various necessary inputs (including data and software) through a process similar to Sciunits mentioned above to find the best environment isolator (maybe using Linux containerization, containers, or VMs). -We could not find a URL to the source software of Umbrella (no source code repository is mentioned in the papers we reviewed above), but from the descriptions in \citeappendix{meng17}, it is written in Python 2.6 (which is now deprecated). +We could not find a URL to the source software of Umbrella (no source code repository is mentioned in the papers we reviewed above), but from the descriptions\citeappendix{meng17}, it is written in Python 2.6 (which is now deprecated). \subsection{ReproZip (2016)} -ReproZip\footnote{\inlinecode{\url{https://www.reprozip.org}}} \citeappendix{chirigati16} is a Python package that is designed to automatically track all the necessary data files, libraries, and environment variables into a single bundle. +ReproZip\footnote{\inlinecode{\url{https://www.reprozip.org}}}\citeappendix{chirigati16} is a Python package that is designed to automatically track all the necessary data files, libraries, and environment variables into a single bundle. The tracking is done at the kernel system-call level, so any file that is accessed during the running of the project is identified. The tracked files can be packaged into a \inlinecode{.rpz} bundle that can then be unpacked into another system. @@ -430,7 +430,7 @@ ReproZip is therefore very good to take a ``snapshot'' of the running environmen However, the bundle can become very large when many/large datasets are involved, or if the software environment is complex (many dependencies). Since it copies the binary software libraries, it can only be run on systems with a similar CPU architecture to the original. Furthermore, ReproZip just copies the binary/compiled files used in a project, it has no way to know how the software was built. -As mentioned in this paper, and also \citeappendix{oliveira18} the question of ``how'' the environment was built is critical for understanding the results, and simply having the binaries cannot necessarily be useful. +As mentioned in this paper, and also Oliveira et al. \citeappendix{oliveira18} the question of ``how'' the environment was built is critical for understanding the results, and simply having the binaries cannot necessarily be useful. For the data, it is similarly not possible to extract which data server they came from. Hence two projects that each use a 1-terabyte dataset will need a full copy of that same 1-terabyte file in their bundle, making long-term preservation extremely expensive. @@ -445,7 +445,7 @@ Users simply add a set of Binder-recognized configuration files to their reposit One good feature of Binder is that the imported Docker image must be tagged, although as mentioned in Appendix \ref{appendix:containers}, tags do not ensure reproducibility. However, it does not make sure that the Dockerfile used by the imported Docker image follows a similar convention also. So users can simply use generic operating system names. -Binder is used by \citeappendix{jones19}. +Binder is used by Jones et al.\citeappendix{jones19}. @@ -470,7 +470,7 @@ However, there is one directory that can be used to store files that must not be \subsection{Popper (2017)} \label{appendix:popper} -Popper\footnote{\inlinecode{\url{https://falsifiable.us}}} is a software implementation of the Popper Convention \citeappendix{jimenez17}. +Popper\footnote{\inlinecode{\url{https://falsifiable.us}}} is a software implementation of the Popper Convention\citeappendix{jimenez17}. The Popper team's own solution is through a command-line program called \inlinecode{popper}. The \inlinecode{popper} program itself is written in Python. However, job management was initially based on the HashiCorp configuration language (HCL) because HCL was used by ``GitHub Actions'' to manage workflows at that time. @@ -493,7 +493,7 @@ Hence any future change in the low level features will directly propagated to al \subsection{Whole Tale (2017)} \label{appendix:wholetale} -Whole Tale\footnote{\inlinecode{\url{https://wholetale.org}}} is a web-based platform for managing a project and organizing data provenance, see \citeappendix{brinckman17}. +Whole Tale\footnote{\inlinecode{\url{https://wholetale.org}}} is a web-based platform for managing a project and organizing data provenance\citeappendix{brinckman17}. It uses online editors like Jupyter or RStudio (see Appendix \ref{appendix:editors}) that are encapsulated in a Docker container (see Appendix \ref{appendix:independentenvironment}). The web-based nature of Whole Tale's approach and its dependency on many tools (which have many dependencies themselves) is a major limitation for future reproducibility. @@ -503,7 +503,7 @@ But as all the second-order dependencies evolve, it is not hard to envisage such Furthermore, the fact that a Tale is stored as a binary Docker container causes two important problems: 1) it requires a very large storage capacity for every project that is hosted there, making it very expensive to scale if demand expands. 2) It is not possible to see how the environment was built accurately (when the Dockerfile uses operating system package managers like \inlinecode{apt}). -This issue with Whole Tale (and generally all other solutions that only rely on preserving a container/VM) was also mentioned in \citeappendix{oliveira18}, for more on this, please see Appendix \ref{appendix:packagemanagement}. +This issue with Whole Tale (and generally all other solutions that only rely on preserving a container/VM) was also mentioned in Oliveira et al.\citeappendix{oliveira18}, for more on this, please see Appendix \ref{appendix:packagemanagement}. @@ -511,7 +511,7 @@ This issue with Whole Tale (and generally all other solutions that only rely on \subsection{Occam (2018)} \label{appendix:occam} -Occam\footnote{\inlinecode{\url{https://occam.cs.pitt.edu}}} \citeappendix{oliveira18} is a web-based application to preserve software and its execution. +Occam\footnote{\inlinecode{\url{https://occam.cs.pitt.edu}}}\citeappendix{oliveira18} is a web-based application to preserve software and its execution. To achieve long-term reproducibility, Occam includes its own package manager (instructions to build software and their dependencies) to be in full control of the software build instructions, similar to Maneage. Besides Nix or Guix (which are primarily a package manager that can also do job management), Occam has been the only solution in our survey here that attempts to be complete in this aspect. |