path: root/peer-review/1-answer.txt

1.  [EiC] Some reviewers request additions, and overview of other
    tools.

ANSWER: Indeed, there is already a large body work in various issues that
have been touched upon in this paper. Before submitting the paper, we had
already done a very comprehensive review of the tools (as you may notice
from the Git repository[1]). However, the CiSE Author Information
explicitly states: "The introduction should provide a modicum of background
in one or two paragraphs, but should not attempt to give a literature
review". This is also practiced in previously published papers at CiSE and
is in line with the very limited word-count and maximum of 12 references to
be used in bibliography.

We were also eager to get that extensive review out (which took a lot of
time, and most of the tools were actually run and tested). Hence we
discussed this privately with the editors and this solution was agreed
upon: we include that extended review as appendices on the arXiv[2] and
Zenodo[3] pre-prints of this paper and mention those publicly available
appendices in the submitted paper for an interested reader to followup.

[1] https://gitlab.com/makhlaghi/maneage-paper/-/blob/master/tex/src/paper-long.tex#L1579
[2] https://arxiv.org/abs/2006.03018
[3] https://doi.org/10.5281/zenodo.3872247

------------------------------





2.  [Associate Editor] There are general concerns about the paper
    lacking focus

ANSWER:

------------------------------





3.  [Associate Editor] Some terminology is not well-defined
    (e.g. longevity).

ANSWER: It has now been clearly defined in the first paragraph of Section
II. With this definition, the main argument of the paper is much more clear,
thank you (and the referees for highlighting this).

------------------------------





4.  [Associate Editor] The discussion of tools could benefit from some
    categorization to characterize their longevity.

ANSWER: The longevity of the general tools reviewed in Section II are now
mentioned immediately after each (highlighted in green).

------------------------------





5.  [Associate Editor] Background and related efforts need significant
    improvement. (See below.)

ANSWER: This has been done, as mentioned in (1).

------------------------------





6.  [Associate Editor] There is consistency among the reviews that
    related work is particularly lacking.

ANSWER: This has been done, as mentioned in (1).

------------------------------





7.  [Associate Editor] The current work needs to do a better job of
    explaining how it deals with the nagging problem of running on CPU
    vs. different architectures.

ANSWER: The CPU architecture of the running system is now reported in the
"Acknowledgments" section and a description of the problem and its solution
in Maneage is also added in the "Proof of concept: Maneage" Section.

------------------------------





8.  [Associate Editor] At least one review commented on the need to
    include a discussion of continuous integration (CI) and its
    potential to help identify problems running on different
    architectures. Is CI employed in any way in the work presented in
    this article?

ANSWER: CI has been added in the discussion as one solution to find
breaking points in operating system updates and new/different
architectures. For the core Maneage branch, we have defined task #15741 [1]
to add CI on many architectures in the near future.

[1] http://savannah.nongnu.org/task/?15741

------------------------------





9.  [Associate Editor] The presentation of the Maneage tool is both
    lacking in clarity and consistency with the public
    information/documentation about the tool. While our review focus
    is on the article, it is important that readers not be confused
    when they visit your site to use your tools.

###########################
ANSWER [NOT COMPLETE]: We should separate the various sections of the
README-hacking.md webpage into smaller pages that can be entered.
###########################

------------------------------





10. [Associate Editor] A significant question raised by one review is
    how this work compares to "executable" papers and Jupyter
    notebooks.  Does this work embody similar/same design principles
    or expand upon the established alternatives? In any event, a
    discussion of this should be included in background/motivation and
    related work to help readers understand the clear need for a new
    approach, if this is being presented as new/novel.

ANSWER: Thank you for highlighting this important point. We saw that its
necessary to contrast our proof of concept demonstration more directly with
Maneage. Two paragraphs have been added in Sections II and IV for this.

------------------------------





11. [Reviewer 1] Adding an explicit list of contributions would make
    it easier to the reader to appreciate these. These are not
    mentioned/cited and are highly relevant to this paper (in no
    particular order):
     1.  Git flows, both in general and in particular for research.
     2.  Provenance work, in general and with git in particular
     3.  Reprozip: https://www.reprozip.org/
     4.  OCCAM: https://occam.cs.pitt.edu/
     5.  Popper: http://getpopper.io/
     6.  Whole Tale: https://wholetale.org/
     7.  Snakemake: https://github.com/snakemake/snakemake
     8.  CWL https://www.commonwl.org/ and WDL https://openwdl.org/
     9.  Nextflow: https://www.nextflow.io/
     10. Sumatra: https://pythonhosted.org/Sumatra/
     11. Podman: https://podman.io
     12. AppImage (https://appimage.org/)
     13. Flatpack (https://flatpak.org/)
     14. Snap (https://snapcraft.io/)
     15. nbdev https://github.com/fastai/nbdev and jupytext
     16. Bazel: https://bazel.build/
     17. Debian reproducible builds: https://wiki.debian.org/ReproducibleBuilds

ANSWER:

1.  In Section IV, we have added that "Generally, any git flow (branching
    strategies) can be used by the high-level project authors or future
    readers."
2.  We have mentioned research objects as one mode of provenance tracking
    and the related provenance work that has already been done and can be
    exploited using these criteria and our proof of concept is indeed very
    large. However, the 6250 word-count limit is very tight and if we add
    more on it in this length, we would have to remove more directly
    relevant points. Hopefully this can be the subject of a follow up
    paper.
3.  A review of ReproZip is in Appendix B.
4.  A review of Occam is in Appendix B.
5.  A review of Popper is in Appendix B.
6.  A review of Whole tale is in Appendix B.
7.  A review of Snakemake is in Appendix A.
8.  CWL and WDL are described in Appendix A (job management).
9.  Nextflow is described in Appendix A (job management).
10. Sumatra is described in Appendix B.
11. Podman is mentioned in Appendix A (containers).
12. AppImage is mentioned in Appendix A (package management).
13. Flatpak is mentioned in Appendix A (package management).
14. nbdev and jupytext are high-level tools to generate documentation and
    packaging custom code in Conda or pypi. High-level package managers
    like Conda and Pypi have already been thoroughly reviewed in Appendix A
    for their longevity issues, so we feel there is no need to include
    these.
15. Bazel has been mentioned in Appendix A (job management).
16. Debian's reproducible builds is only for ensuring that software
    packaged for Debian are bitwise reproducible. As mentioned in the
    discussion of this paper, the bitwise reproducibility of software is
    not an issue in the context discussed here, the reproducibility of the
    relevant output data of the software is the main issue.


------------------------------





12. [Reviewer 1] Existing guidelines similar to the proposed "Criteria
    for longevity". Many articles of these in the form "10 simple
    rules for X", for example (not exhaustive list):
     * https://doi.org/10.1371/journal.pcbi.1003285
     * https://arxiv.org/abs/1810.08055
     * https://osf.io/fsd7t/
     * A model project for reproducible papers: https://arxiv.org/abs/1401.2000
     * Executable/reproducible paper articles and original concepts

ANSWER: Thank you for highlighting these points. Appendix B starts with a
subsection titled "suggested rules, checklists or criteria" that review of
existing criteria. That include the proposed sources here (and others).

arXiv:1401.2000 has been added in Appendix A as an example paper using
virtual machines. We thank the referee for bringing up this paper, because
the link to the VM provided in the paper no longer works (the file has been
removed on the server). Therefore added with SHARE, it very nicely
highlighting our main issue with binary containers or VMs and their lack of
longevity.

------------------------------





13. [Reviewer 1] Several claims in the manuscript are not properly
    justified, neither in the text nor via citation. Examples (not
    exhaustive list):
     1. "it is possible to precisely identify the Docker “images” that
        are imported with their checksums, but that is rarely practiced
        in most solutions that we have surveyed [which ones?]"
     2. "Other OSes [which ones?] have similar issues because pre-built
        binary files are large and expensive to maintain and archive."
     3. "Researchers using free software tools have also already had
        some exposure to it"
     4. "A popular framework typically falls out of fashion and
        requires significant resources to translate or rewrite every
        few years."

ANSWER: They have been clarified in the highlighted parts of the text:

1. Many examples have been given throughout the newly added appendices. To
   avoid confusion in the main body of the paper, we have removed the "we
   have surveyed" part. It is already mentioned above it that a large
   survey of existing methods/solutions is given in the appendices.

2. Due to the thorough discussion of this issue in the appendices with
   precise examples, this line has been removed to allow space for the
   other points raised by the referees. The main point (high cost of
   keeping binaries) is aldreay abundantly clear.

   On a similar topic, Dockerhub's recent announcement that inactive images
   (for over 6 months) will be deleted has also been added. The announcemnt
   URL is here (it was too long to include in the paper, if IEEE has a
   special short-url format, we can add it):
   https://www.docker.com/blog/docker-hub-image-retention-policy-delayed-and-subscription-updates

3. A small statement has been added, reminding the readers that almost all
   free software projects are built with Make (note that CMake is just a
   high-level wrapper over Make: it finally produces a 'Makefile').

4. The example of Python 2 has been added.


------------------------------





14. [Reviewer 1] As mentioned in the discussion by the authors, not
    even Bash, Git or Make is reproducible, thus not even Maneage can
    address the longevity requirements. One possible alternative is
    the use of CI to ensure that papers are re-executable (several
    papers have been written on this topic). Note that CI is
    well-established technology (e.g. Jenkins is almost 10 years old).

ANSWER: Thank you for raising this issue. We had initially planned to add
this issue also, but like many discussion points, we were forced to remove
it before the first submission due to the very tight word-count limit. We
have now added a sentence on CI in the discussion.

On the initial note, indeed, the "executable" files of Bash, Git or Make
are not bitwise reproducible/identical on different systems. However, as
mentioned in the discussion, we are concerned with the _output_ of the
software's executable file, _after_ the execution of its job. We (or any
user of Bash) is not interested in the executable file itself. The
reproducibility of the binary file only becomes important if a bug is found
(very rare for common usage in such core software of the OS).  Hence even
though the compiled binary files of specific versions of Git, Bash or Make
will not be bitwise reproducible/identical on different systems, their
outputs are exactly reproducible: 'git describe' or Bash's 'for' loop will
have the same output on GNU/Linux, macOS or FreeBSD (that produce bit-wise
different executables).

------------------------------





15. [Reviewer 1] Criterion has been proposed previously. Maneage itself
    provides little novelty (see comments below).

ANSWER: The previously suggested criteria that were mentioned are reviewed
in the newly added Appendix B, and the novelty/necessity of the proposed
criteria is shown by comparison there.

------------------------------





16. [Reviewer 2] Authors should add indication that using good practices it
    is possible to use Docker or VM to obtain identical OS usable for
    reproducible research.

ANSWER: In the submitted version we had stated that "Ideally, it is
possible to precisely identify the Docker “images” that are imported with
their checksums ...". But to be more clear and directly to the point, it
has been edited to explicity say "... to recreate an identical OS image
later".

------------------------------





17. [Reviewer 2] The CPU architecture of the platform used to run the
    workflow is not discussed in the manuscript. Authors should probably
    take into account the architecture used in their workflow or at least
    report it.

ANSWER: Thank you very much for raising this important point. We hadn't
seen other reproducibility papers mention this important point and missed
it. In the acknowledgments (where we also mention the commit hashes) we now
explicity mention the exact CPU architecture used to build this paper:
"This project was built on an x86_64 machine with Little Endian byte-order
and address sizes 39 bits physical, 48 bits virtual.". This is because we
have already seen cases where the architecture is the same, but programs
fail because of the byte-order.

Generally, Maneage will now extract this information from the running
system during its configuration phase and provide the users with three
different LaTeX macros that they can use anywhere in their paper.

------------------------------





18. [Reviewer 2] I don’t understand the "no dependency beyond
    POSIX". Authors should more explained what they mean by this sentence.

ANSWER: This has been clarified with the short extra statement "a minimal
Unix-like standard that is shared between many operating systems". We would
have liked to explain this more, but the word-limit is very constraining.

------------------------------





19. [Reviewer 2] Unfortunately, sometime we need proprietary or specialized
    software to read raw data... For example in genetics, micro-array raw
    data are stored in binary proprietary formats. To convert this data
    into a plain text format, we need the proprietary software provided
    with the measurement tool.

ANSWER: Thank you very much for this good point. A description of a
possible solution to this has been added after criteria 8.

------------------------------





20. [Reviewer 2] I was not able to properly set up a project with
    Maneage. The configuration step failed during the download of tools
    used in the workflow. This is probably due to a firewall/antivirus
    restriction out of my control. How frequent this failure happen to
    users?

ANSWER: Thank you for mentioning this. This has been fixed by archiving all
Maneage'd software on Zenodo (https://doi.org/10.5281/zenodo.3883409) and
also downloading from there.

Until recently we would directly access each software's own webpage to
download the files, and this caused many problems like this. In other
cases, we were very frustrated when a software's webpage would temporarily
be unavailable (for maintainance reasons), this wouldn't allow us to build
new projects.

Since all the software are free, we are allowed to re-distribute them and
Zenodo is defined for long-term archival of academic artifacts, so we
figured that a software source code repository on Zenodo would be the most
reliable solution. At configure time, Maneage now accesses Zenodo's DOI and
resolves the most recent URL to automatically download any necessary
software source code that the project needs from there.

Generally, we also keep all software in a Git repository on our own
webpage: http://git.maneage.org/tarballs-software.git/tree. Also, Maneage
users can also identify their own custom URLs for downloading software,
which will be given higher priority than Zenodo (useful for situations when
a custom software is downloaded and built in a project branch (not the core
'maneage' branch).

------------------------------





21. [Reviewer 2] The time to configure a new project is quite long because
    everything needs to be compiled. Authors should compare the time
    required to set up a project Maneage versus time used by other
    workflows to give an indication to the readers.

ANSWER: Thank you for raising this point. it takes about 1.5 hours to
configure the default Maneage branch on an 8-core CPU (more than half of
this time is devoted to GCC on GNU/Linux operating systems, and the
building of GCC can optionally be disabled with the '--host-cc' option to
significantly speed up the build when the host's GCC is
similar). Furthermore, Maneage can be built within a Docker container.

Generally, a paragraph has been added in Section IV on this issue (the
build time and building within a Docker container). We have also defined
task #15818 [1] to have our own core Docker image that is ready to build a
Maneaged project and will be adding it shortly.

[1] https://savannah.nongnu.org/task/index.php?15818

------------------------------





22. [Reviewer 3] Authors should define their use of the term [Replicability
    or Reproducibility] briefly for their readers.

ANSWER: "Reproducibility" has been defined along with "Longevity" and
"usage" at the start of Section II.

------------------------------





23. [Reviewer 3] The introduction is consistent with the proposal of the
    article, but deals with the tools separately, many of which can be used
    together to minimize some of the problems presented. The use of
    Ansible, Helm, among others, also helps in minimizing problems.

ANSWER: Ansible and Helm are primarily designed for distributed
computing. For example Helm is just a high-level package manager for a
Kubernetes cluster that is based on containers. A review of them can be
added in the Appendix, but we feel they may not be too relevant for this
paper.

------------------------------





24. [Reviewer 3] When the authors use the Python example, I believe it is
    interesting to point out that today version 2 has been discontinued by
    the maintaining community, which creates another problem within the
    perspective of the article.

ANSWER: Thank you very much for highlighting that this point was not included
for the sake of length, it has been fitted into the introduction now.

------------------------------





25. [Reviewer 3] Regarding the use of VM's and containers, I believe that
    the discussion presented by THAIN et al., 2015 is interesting to
    increase essential points of the current work.

ANSWER: Thank you very much for pointing this the works by Thain. We
couldn't find any first-author papers in 2015, but found Meng & Thain
(https://doi.org/10.1016/j.procs.2017.05.116) which had a related
discussion of why they didn't use Docker containers in their work. That
paper is now cited in the discussion of Containers in Appendix A.

------------------------------





26. [Reviewer 3] About the Singularity, the description article was missing
    (Kurtzer GM, Sochat V, Bauer MW, 2017).

ANSWER: Thank you for the reference, we could not put it in the main body
of the paper (like many others) due to the strict bibliography limit of 12,
but it has been cited in Appendix A (where we discuss Singularity).

------------------------------





27. [Reviewer 3] I also believe that a reference to FAIR is interesting
    (WILKINSON et al., 2016).

ANSWER: The FAIR principles have been mentioned in the main body of the
paper, but unfortunately we had to remove its citation the main paper (like
many others) within the maximum limit 12 references. We have cited it in
Appendix B.

------------------------------





28. [Reviewer 3] In my opinion, the paragraph on IPOL seems to be out of
    context with the previous ones. This issue of end-to-end
    reproducibility of a publication could be better explored, which would
    further enrich the tool presented.

#####################################
ANSWER:
#####################################

------------------------------





29. [Reviewer 3] On the project website, I suggest that the information
    contained in README-hacking be presented on the same page as the
    Tutorial. A topic breakdown is interesting, as the markdown reading may
    be too long to find information.

#####################################
ANSWER:
#####################################

------------------------------





31. [Reviewer 3] The tool is suitable for Unix users, keeping users away
    from Microsoft environments.

ANSWER: The issue of building on Windows has been discussed in Section IV,
either using Docker (or VMs) or using the Windows Subsystem for Linux.

------------------------------




32. [Reviewer 3] Important references are missing; more references are
    needed

ANSWER: Two comprehensive Appendices have beed added to address this issue.

------------------------------





33. [Reviewer 4] Revisit the criteria, show how you have come to decide on
    them, give some examples of why they are important, and address
    potential missing criteria.

for example the referee already points to "how code is written" as a
criteria (for example for threading or floating point errors), or
"performance".

#################################
ANSWER:
#################################

------------------------------





34. [Reviewer 4] Clarify the discussion of challenges to adoption and make
    it clearer which tradeoffs are important to practitioners.

##########################
ANSWER:
##########################

------------------------------





35. [Reviewer 4] Be clearer about which sorts of research workflow are best
    suited to this approach.

################################
ANSWER:
################################

------------------------------





36. [Reviewer 4] There is also the challenge of mathematical
    reproducibility, particularly of the handling of floating point number,
    which might occur because of the way the code is written, and the
    hardware architecture (including if code is optimised / parallelised).

################################
ANSWER:
################################

------------------------------





37. [Reviewer 4] Performance ... is never mentioned

################################
ANSWER:
################################

------------------------------

38. [Reviewer 4] Tradeoff, which might affect Criterion 3 is time to result,
    people use popular frameworks because it is easier to use them.

################################
ANSWER:
################################

------------------------------





39. [Reviewer 4] I would liked to have seen explanation of how these
    challenges to adoption were identified: was this anecdotal, through
    surveys? participant observation?

ANSWER: The results mentioned here are based on private discussions after
holding multiple seminars and Webinars with RDA's support, and also a
workshop that was planned for non-astronomers. We even invited (funded)
early career researchers to come to the workshop with the RDA funding,
however, that workshop was cancelled due to the pandemic and we had private
communications after.

We would very much like to elaborate on this experience of training new
researchers with these tools. However, as with many of the cases above, the
very strict word-limit doesn't allow us to elaborate beyond what is already
there.

------------------------------





40. [Reviewer 4] Potentially an interesting sidebar to investigate how
    LaTeX/TeX has ensured its longevity!

##############################
ANSWER:
##############################

------------------------------





41. [Reviewer 4] The title is not specific enough - it should refer to the
    reproducibility of workflows/projects.

##############################
ANSWER:
##############################

------------------------------





42. [Reviewer 4] Whilst the thesis stated is valid, it may not be useful to
    practitioners of computation science and engineering as it stands.

ANSWER: We would appreciate if you could clarify this point a little
more. We have shown how it has already been used in many research projects
(also outside of observational astronomy which is the first author's main
background). It is precisely defined for computational science and
engineering problems where _publication_ of the human-readable workflow
source is also important.

------------------------------





43. [Reviewer 4] Longevity is not defined.

ANSWER: It has been defined now at the start of Section II.

------------------------------





44. [Reviewer 4] Whilst various tools are discussed and discarded, no
    attempt is made to categorise the magnitude of longevity for which they
    are relevant. For instance, environment isolators are regarded by the
    software preservation community as adequate for timescale of the order
    of years, but may not be suitable for the timescale of decades where
    porting and emulation are used.

ANSWER: Statements on quantifying their longevity have been added in
Section II. For example in the case of Docker images: "their longevity is
determined by the host kernel, usually a decade", for Python packages:
"Python installation with a usual longevity of a few years", for Nix/Guix:
"with considerably better longevity; same as supported CPU architectures."

------------------------------





45. [Reviewer 4] The title of this section "Commonly used tools and their
    longevity" is confusing - do you mean the longevity of the tools or the
    longevity of the workflows that can be produced using these tools?
    What happens if you use a combination of all four categories of tools?

##########################
ANSWER:
##########################

------------------------------





46. [Reviewer 4] It wasn't clear to me if code was being run to generate
    the results and figures in a LaTeX paper that is part of a project in
    Maneage. It appears to be suggested this is the case, but Figure 1
    doesn't show how this works - it just has the LaTeX files, the data
    files and the Makefiles. Is it being suggested that LaTeX itself is the
    programming language, using its macro functionality?

ANSWER: Thank you for highlighting this point of confusion. The caption of
Figure 1 has been edited to hopefully clarify the point. In short, the
arrows represent the operation of software on their inputs (the file they
originate from) to generate their outputs (the file they point to). In the
case of generating 'paper.pdf' from its three dependencies
('references.tex', 'paper.tex' and 'project.tex'), yes, LaTeX is used. But
in other steps, other tools are used. For example as you see in [1] the
main step of the arrow connecting 'table-3.txt' to 'tools-per-year.txt' is
an AWK command (there are also a few 'echo' commands for meta data and
copyright in the output plain-text file [2]).

[1] https://gitlab.com/makhlaghi/maneage-paper/-/blob/master/reproduce/analysis/make/demo-plot.mk#L51
[2] https://zenodo.org/record/3911395/files/tools-per-year.txt

------------------------------





47. [Reviewer 4] I was a bit confused on how collaboration is handled as
    well - this appears to be using the Git branching model, and the
    suggestion that Maneage is keeping track of all components from all
    projects - but what happens if you are working with collaborators that
    are using their own Maneage instance?

ANSWER: Indeed, Maneage operates based on the Git branching model. As
mentioned in the text, Maneage is itself a Git branch. People create their
own branch from the 'maneage' branch and start customizing it for their
particular project in their own particular repository. They can also use
all types of Git-based collaborating models to work together on a project
that is not yet finished.

Figure 2 in fact explicitly shows such a case: the main project leader is
committing on the "project" branch. But a collaborator creates a separate
branch over commit '01dd812' and makes a couple of commits ('f69e1f4' and
'716b56b'), and finally asks the project leader to merge them into the
project. This can be generalized to any Git based collaboration model.

------------------------------





48. [Reviewer 4] I would also liked to have seen a comparison between this
    approach and other "executable" paper approaches e.g. Jupyter
    notebooks, compared on completeness, time taken to write a "paper",
    ease of depositing in a repository, and ease of use by another
    researcher.

#######################
ANSWER:
#######################

------------------------------





49. [Reviewer 4] The weakest aspect is the assumption that research can be
    easily compartmentalized into simple and complete packages. Given that
    so much of research involves collaboration and interaction, this is not
    sufficiently addressed. In particular, the challenge of
    interdisciplinary work, where there may not be common languages to
    describe concepts and there may be different common workflow practices
    will be a barrier to wider adoption of the primary thesis and criteria.

ANSWER: Maneage was precisely defined to address the problem of
publishing/collaborating on complete workflows. Hopefully with the
clarification to point 47 above, this should also become clear.

------------------------------





50. [Reviewer 5] Major figures currently working in this exact field do not
    have their work acknowledged in this work.

ANSWER: This was due to the strict word limit and the CiSE publication
policy (to not include a literature review because there is a limit of only
12 citations). But we had indeed done a comprehensive literature review and
the editors kindly agreed that we publish that review as appendices to the
main paper on arXiv and Zenodo.

------------------------------





51. [Reviewer 5] The popper convention: Making reproducible systems
    evaluation practical ... and the later revision that uses GitHub
    Actions, is largely the same as this work.

ANSWER: This work and the proposed criteria are very different from
Popper. A review of Popper has been given in Appendix B.

------------------------------





52. [Reviewer 5] The lack of attention to virtual machines and containers
    is highly problematic. While a reader cannot rely on DockerHub or a
    generic OS version label for a VM or container, these are some of the
    most promising tools for offering true reproducibility.

ANSWER: Containers and VMs have been more thoroughly discussed in the main
body and also extensively discussed in appendix A (that are now available
in the arXiv and Zenodo versions of this paper). As discussed (with many
cited examples), Contains and VMs are only good when they are themselves
reproducible (for example running the Dockerfile this year and next year
gives the same internal environment). However we show that this is not the
case in most solutions (a more comprehensive review would require its own
paper).

However with complete/robust environment builders like Maneage, Nix or GNU
Guix, the analysis environment within a container can be exactly reproduced
later. But even so, due to their binary nature and large storage volume,
they are not trusable sources for the long term (it is expensive to archive
them). We show several example in the paper of how projects that relied on
VMs in 2011 and 2014 are no longer active, and how even Dockerhub will be
deleting containers that are not used for more than 6 months in free
accounts (due to the large storage costs).

Raul: it would be interesting to mention here that Maneage has the criterion of
"Minimal complexity". This means that even if for any reason the project is not
able to be run in the future, the content, analysis scripts, etc. are accesible
for the interested reader (because it is in plain text). So, it is transparent
in any case and the interested reader can follow the analysis and study the
different decissions of each step (why and how the analysis was done).

------------------------------





53. [Reviewer 5] On the data side, containers have the promise to manage
    data sets and workflows completely [Lofstead J, Baker J, Younge A. Data
    pallets: containerizing storage for reproducibility and
    traceability. InInternational Conference on High Performance Computing
    2019 Jun 16 (pp. 36-45). Springer, Cham.] Taufer has picked up this
    work and has graduated a MS student working on this topic with a
    published thesis. See also Jimenez's P-RECS workshop at HPDC for
    additional work highly relevant to this paper.

ANSWER: Thank you for the interesting paper by Lofstead+2019 on Data
pallets. We have cited it in Appendix A as examples of how generic the
concept of containers is.

The topic of linking data to analysis is also a core result of the criteria
presented here, and is also discussed shortly in the paper.  There are
indeed many very interesting works on this topic. But the format of CiSE is
very short (a maximum of ~6000 words with 12 references), so we don't have
the space to go into this any further. But this is indeed a very
interesting aspect for follow up studies, especially as the usage of
Maneage incrases, and we have more example workflows by users to study the
linkage of data analysis.

------------------------------





54. [Reviewer 5] Some other systems that do similar things include:
    reprozip, occam, whole tale, snakemake.

ANSWER: All these tools have been reviewed in the newly added appendices.

------------------------------





55. [Reviewer 5] the paper needs to include the context of the current
    community development level to be a complete research paper. A revision
    that includes evaluation of (using the criteria) and comparison with
    the suggested systems and a related work section that seriously
    evaluates the work of the recommended authors, among others, would make
    this paper worthy for publication.

ANSWER: A thorough review of current low-level tools and and high-level
reproducible workflow management systems has been added in the extended
Appendix.

------------------------------





56. [Reviewer 5] Offers criteria any system that offers reproducibility
   should have.

ANSWER:

------------------------------





57. [Reviewer 5] Yet another example of a reproducible workflows project.

ANSWER: As the newly added thorough comparisons with existing systems
shows, these set of criteria and the proof-of-concept offer uniquely new
features. As another referee summarized: "This manuscript describes a new
reproducible workflow which doesn't require another new trendy high-level
software. The proposed workflow is only based on low-level tools already
widely known."

The fact that we don't define yet another workflow language and framework
and base the whole workflow on time-tested solutions in a framwork that
costs only ~100 kB to archive (in contrast to multi-GB containers or VMs)
is new.

------------------------------





58. [Reviewer 5] There are numerous examples, mostly domain specific, and
    this one is not the most advanced general solution.

ANSWER: As the comparisons in the appendices and clarifications above show,
there are many features in the proposed criteria and proof of concept that
are new.

------------------------------





59. [Reviewer 5] Lack of context in the field missing very relevant work
    that eliminates much, if not all, of the novelty of this work.

ANSWER: The newly added appendices thoroughly describe the context and
previous work that has been done in this field.

------------------------------