From: cise computer org To: mohammad akhlaghi org, infantesainz gmail com, boud astro uni torun pl, david valls-gabaud observatoiredeparis psl eu, rbaena iac es Received: Tue, 22 Sep 2020 15:28:21 -0400 Subject: Computing in Science and Engineering, CiSESI-2020-06-0048 major revision required -------------------------------------------------- Computing in Science and Engineering,CiSESI-2020-06-0048 "Towards Long-term and Archivable Reproducibility" manuscript type: Reproducible Research Dear Dr. Mohammad Akhlaghi, The manuscript that you submitted to Computing in Science and Engineering has completed the review process. After carefully examining the manuscript and reviews, we have decided that the manuscript needs major revisions before it can be considered for a second review. Your revision is due before 22-Oct-2020. Please note that if your paper was submitted to a special issue, this due date may be different. Contact the peer review administrator, Ms. Jessica Ingle, at cise computer.org if you have questions. The reviewer and editor comments are attached below for your reference. Please maintain our 6,250–word limit as you make your revisions. To upload your revision and summary of changes, log on to https://mc.manuscriptcentral.com/cise-cs, click on your Author Center, then "Manuscripts with Decisions." Under "Actions," choose "Create a Revision" next to the manuscript number. Highlight the changes to your manuscript by using the track changes mode in MS Word, the latexdiff package if using LaTex, or by using bold or colored text. When submitting your revised manuscript, you will need to respond to the reviewer comments in the space provided. If you have questions regarding our policies or procedures, please refer to the magazines' Author Information page linked from the Instructions and Forms (top right corner of the ScholarOne Manuscripts screen) or you can contact me. We look forward to receiving your revised manuscript. Sincerely, Dr. Lorena A. Barba George Washington University Mechanical and Aerospace Engineering Editor-in-Chief, Computing in Science and Engineering -------------------------------------------------- EiC comments: Some reviewers request additions, and overview of other tools, etc. In doing your revision, please remember space limitations: 6,250 words maximum, including all main body, abstract, keyword, bibliography (12 references or less), and biography text. See "Write For Us" section of the website: https://www.computer.org/csdl/magazine/cs Comments of the Associate Editor: Associate Editor Comments to the Author: Thank to the authors for your submission to the Reproducible Research department. Thanks to the reviewers for your careful and thoughtful reviews. We would appreciate it if you can make your reports available and share the DOI as soon as possible, per our original invitation e-mail. We will follow up our original invitation to obtain your review DOI, if you have not already included it in your review comments. Based on the review feedback, there are a number of major issues that require attention and many minor ones as well. Please take these into account as you prepare your major revision for another round of review. (See the actual review reports for details.) 1. In general, there are a number of presentation issues needing attention. There are general concerns about the paper lacking focus. Some terminology is not well-defined (e.g. longevity). In addition, the discussion of tools could benefit from some categorization to characterize their longevity. Background and related efforts need significant improvement. (See below.) 2. There is consistency among the reviews that related work is particularly lacking and not taking into account major works that have been written on this topic. See the reviews for details about work that could potentially be included in the discussion and how the current work is positioned with respect to this work. 3. The current work needs to do a better job of explaining how it deals with the nagging problem of running on CPU vs. different architectures. At least one review commented on the need to include a discussion of continuous integration (CI) and its potential to help identify problems running on different architectures. Is CI employed in any way in the work presented in this article? 4. The presentation of the Maneage tool is both lacking in clarity and consistency with the public information/documentation about the tool. While our review focus is on the article, it is important that readers not be confused when they visit your site to use your tools. 5. A significant question raised by one review is how this work compares to "executable" papers and Jupyter notebooks. Does this work embody similar/same design principles or expand upon the established alternatives? In any event, a discussion of this should be included in background/motivation and related work to help readers understand the clear need for a new approach, if this is being presented as new/novel. Reviews: Please note that some reviewers may have included additional comments in a separate file. If a review contains the note "see the attached file" under Section III A - Public Comments, you will need to log on to ScholarOne Manuscripts to view the file. After logging in, select the Author Center, click on the "Manuscripts with Decisions" queue and then click on the "view decision letter" link for this manuscript. You must scroll down to the very bottom of the letter to see the file(s), if any. This will open the file that the reviewer(s) or the Associate Editor included for you along with their review. -------------------------------------------------- Reviewer: 1 Recommendation: Author Should Prepare A Major Revision For A Second Review Comments: * Adding an explicit list of contributions would make it easier to the reader to appreciate these. * These are not mentioned/cited and are highly relevant to this paper (in no particular order): * Git flows, both in general and in particular for research. * Provenance work, in general and with git in particular * Reprozip: https://www.reprozip.org/ * OCCAM: https://occam.cs.pitt.edu/ * Popper: http://getpopper.io/ * Whole Tale: https://wholetale.org/ * Snakemake: https://github.com/snakemake/snakemake * CWL https://www.commonwl.org/ and WDL https://openwdl.org/ * Nextflow: https://www.nextflow.io/ * Sumatra: https://pythonhosted.org/Sumatra/ * Podman: https://podman.io * AppImage (https://appimage.org/), Flatpack (https://flatpak.org/), Snap (https://snapcraft.io/) * nbdev https://github.com/fastai/nbdev and jupytext * Bazel: https://bazel.build/ * Debian reproducible builds: https://wiki.debian.org/ReproducibleBuilds * Existing guidelines similar to the proposed "Criteria for longevity". Many articles of these in the form "10 simple rules for X", for example (not exhaustive list): * https://doi.org/10.1371/journal.pcbi.1003285 * https://arxiv.org/abs/1810.08055 * https://osf.io/fsd7t/ * A model project for reproducible papers: https://arxiv.org/abs/1401.2000 * Executable/reproducible paper articles and original concepts * Several claims in the manuscript are not properly justified, neither in the text nor via citation. Examples (not exhaustive list): * "it is possible to precisely identify the Docker “images” that are imported with their checksums, but that is rarely practiced in most solutions that we have surveyed [which ones?]" * "Other OSes [which ones?] have similar issues because pre-built binary files are large and expensive to maintain and archive." * "Researchers using free software tools have also already had some exposure to it" * "A popular framework typically falls out of fashion and requires significant resources to translate or rewrite every few years." * As mentioned in the discussion by the authors, not even Bash, Git or Make is reproducible, thus not even Maneage can address the longevity requirements. One possible alternative is the use of CI to ensure that papers are re-executable (several papers have been written on this topic). Note that CI is well-established technology (e.g. Jenkins is almost 10 years old). Additional Questions: 1. How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section.: Very Relevant 2. To what extent is this manuscript relevant to readers around the world?: The manuscript is of interest to readers throughout the world 1. Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical.: This article introduces desiderata for long-term archivable reproduciblity and presents Maneage, a system whose goal is to achieve these outlined properties. 2. Is the manuscript technically sound? Please explain your answer in the Detailed Comments section.: Partially 3. What do you see as this manuscript's contribution to the literature in this field?: Presentation of Maneage 4. What do you see as the strongest aspect of this manuscript?: A great summary of Maneage, as well as its implementaiton. 5. What do you see as the weakest aspect of this manuscript?: Criterion has been proposed previously. Maneage itself provides little novelty (see comments below). 1. Does the manuscript contain title, abstract, and/or keywords?: Yes 2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: Yes 3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: Important references are missing; more references are needed 4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Could be improved 5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Satisfactory 6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Satisfactory 7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Satisfactory 8. Please rate and comment on the readability of this manuscript in the Detailed Comments section.: Easy to read 9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are of limited interest to CiSE readers. Please rate the manuscript. Explain your choice in the Detailed Comments section.: Good -------------------------------------------------- Reviewer: 2 Recommendation: Accept If Certain Minor Revisions Are Made Comments: https://doi.org/10.22541/au.159724632.29528907 Operating System: Authors mention that Docker is usually used with an image of Ubuntu without precision about the version used. And Even if users take care about the version, the image is updated monthly thus the image used will have different OS components based on the generation time. This difference in OS components will interfere on the reproducibility. I agree on that, but I would like to add that it is a wrong habit of users. It is possible to generate reproducible Docker images by generating it from an ISO image of the OS. These ISO images are archived, at least for Ubuntu (http://old-releases.ubuntu.com/releases) and for Debian (https://cdimage.debian.org/mirror/cdimage/archive) thus allow users to generate an OS with identical components. Combined with the snapshot.debian.org service, it is even possible to update a Debian release to a specific time point up to 2005 and with a precision of six hours. With combination of both ISO image and snapshot.debian.org service it is possible to obtain an OS for Docker or for a VM with identical components even if users have to use the PM of the OS. Authors should add indication that using good practices it is possible to use Docker or VM to obtain identical OS usable for reproducible research. CPU architecture: The CPU architecture of the platform used to run the workflow is not discussed in the manuscript. During software integration in Debian, I have seen several software failing their unit tests due to different behavior from itself or from a library dependency. This not expected behavior was only present on non-x86 architectures, mainly because developers use a x86 machine for their developments and tests. Bug or feature? I don’t know, but nowadays, it is quite frequent to see computers with a non-x86 CPU. It would be annoying to fail the reproducibility step because of a different in CPU architecture. Authors should probably take into account the architecture used in their workflow or at least report it. POSIX dependency: I don’t understand the "no dependency beyond POSIX". Authors should more explained what they mean by this sentence. I completely agree that the dependency hell must be avoided and dependencies should be used with parsimony. Unfortunately, sometime we need proprietary or specialized software to read raw data. For example in genetics, micro-array raw data are stored in binary proprietary formats. To convert this data into a plain text format, we need the proprietary software provided with the measurement tool. Maneage: I was not able to properly set up a project with Maneage. The configuration step failed during the download of tools used in the workflow. This is probably due to a firewall/antivirus restriction out of my control. How frequent this failure happen to users? Moreover, the time to configure a new project is quite long because everything needs to be compiled. Authors should compare the time required to set up a project Maneage versus time used by other workflows to give an indication to the readers. Disclaimer: For the sake of transparency, it should be noted that I am involved in the development of Debian, thus my comments are probably oriented. Additional Questions: 1. How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section.: Relevant 2. To what extent is this manuscript relevant to readers around the world?: The manuscript is of interest to readers throughout the world 1. Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical.: The authors describe briefly the history of solutions proposed by researchers to generate reproducible workflows. Then, they report the problems with the current tools used to tackle the reproducible problem. They propose a set of criteria to develop new reproducible workflows and finally they describe their proof of concept workflow called "Maneage". This manuscript could help researchers to improve their workflow to obtain reproducible results. 2. Is the manuscript technically sound? Please explain your answer in the Detailed Comments section.: Yes 3. What do you see as this manuscript's contribution to the literature in this field?: The authors try to propose a simple answer to the reproducibility problem by defining new criteria. They also propose a proof of concept workflow which can be directly used by researchers for their projects. 4. What do you see as the strongest aspect of this manuscript?: This manuscript describes a new reproducible workflow which doesn't require another new trendy high-level software. The proposed workflow is only based on low-level tools already widely known. Moreover, the workflow takes into account the version of all software used in the chain of dependencies. 5. What do you see as the weakest aspect of this manuscript?: Authors don't discuss the problem of results reproducibility when analysis are performed using CPU with different architectures. Some libraries have different behaviors when they ran on different architectures and it could influence final results. Authors are probably talking about x86, but there is no reference at all in the manuscript. 1. Does the manuscript contain title, abstract, and/or keywords?: Yes 2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: Yes 3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: References are sufficient and appropriate 4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Yes 5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Satisfactory 6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Satisfactory 7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Satisfactory 8. Please rate and comment on the readability of this manuscript in the Detailed Comments section.: Easy to read 9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are of immediate and continuing interest to CiSE readers Please rate the manuscript. Explain your choice in the Detailed Comments section.: Good -------------------------------------------------- Reviewer: 3 Recommendation: Accept If Certain Minor Revisions Are Made Comments: Longevity of workflows in a project is one of the problems for reproducibility in different fields of computational research. Therefore, a proposal that seeks to guarantee this longevity becomes relevant for the entire community, especially when it is based on free software and is easy to access and implement. GOODMAN et al., 2016, BARBA, 2018 and PLESSER, 2018 observed in their research that the terms reproducibility and replicability are frequently found in the scientific literature and their use interchangeably ends up generating confusion due to the authors' lack of clarity. Thus, authors should define their use of the term briefly for their readers. The introduction is consistent with the proposal of the article, but deals with the tools separately, many of which can be used together to minimize some of the problems presented. The use of Ansible, Helm, among others, also helps in minimizing problems. When the authors use the Python example, I believe it is interesting to point out that today version 2 has been discontinued by the maintaining community, which creates another problem within the perspective of the article. Regarding the use of VM's and containers, I believe that the discussion presented by THAIN et al., 2015 is interesting to increase essential points of the current work. About the Singularity, the description article was missing (Kurtzer GM, Sochat V, Bauer MW, 2017). I also believe that a reference to FAIR is interesting (WILKINSON et al., 2016). In my opinion, the paragraph on IPOL seems to be out of context with the previous ones. This issue of end-to-end reproducibility of a publication could be better explored, which would further enrich the tool presented. The presentation of the longevity criteria was adequate in the context of the article and explored the points that were dealt with later. The presentation of the tool was consistent. On the project website, I suggest that the information contained in README-hacking be presented on the same page as the Tutorial. A topic breakdown is interesting, as the markdown reading may be too long to find information. Additional Questions: 1. How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section.: Relevant 2. To what extent is this manuscript relevant to readers around the world?: The manuscript is of interest to readers throughout the world 1. Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical.: In this article, the authors discuss the problem of the longevity of computational workflows, presenting what they consider to be criteria for longevity and an implementation based on these criteria, called Maneage, seeking to ensure a long lifespan for analysis projects. 2. Is the manuscript technically sound? Please explain your answer in the Detailed Comments section.: Yes 3. What do you see as this manuscript's contribution to the literature in this field?: In this article, the authors discuss the problem of the longevity of computational workflows, presenting what they consider to be criteria for longevity and an implementation based on these criteria, called Maneage, seeking to ensure a long lifespan for analysis projects. As a key point, the authors enumerate quite clear criteria that can guarantee the longevity of projects and present a free software-based way of achieving this objective. The method presented by the authors is not easy to implement for many end users, with low computer knowledge, but it can be easily implemented by users with average knowledge in the area. 4. What do you see as the strongest aspect of this manuscript?: One of the strengths of the manuscript is the implementation of Maneage entirely in free software and the search for completeness presented in the manuscript. The use of GNU software adds the guarantee of long maintenance by one of the largest existing software communities. In addition, the tool developed has already been tested in different publications, showing itself consistent in different scenarios. 5. What do you see as the weakest aspect of this manuscript?: For the proper functioning of the proposed tool, the user needs prior knowledge of LaTeX, GIT and the command line, which can keep inexperienced users away. Likewise, the tool is suitable for Unix users, keeping users away from Microsoft environments. Even though Unix-like environments are the majority in the areas of scientific computing, many users still perform their analysis in different areas on Windows computers or servers, with the assistance of package managers. 1. Does the manuscript contain title, abstract, and/or keywords?: Yes 2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: Yes 3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: Important references are missing; more references are needed 4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Could be improved 5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Satisfactory 6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Could be improved 7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Satisfactory 8. Please rate and comment on the readability of this manuscript in the Detailed Comments section.: Easy to read 9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are of immediate and continuing interest to CiSE readers Please rate the manuscript. Explain your choice in the Detailed Comments section.: Excellent -------------------------------------------------- Reviewer: 4 Recommendation: Author Should Prepare A Major Revision For A Second Review Comments: Overall evaluation - Good. This paper is in scope, and the topic is of interest to the readers of CiSE. However in its present form, I have concerns about whether the paper presents enough new contributions to the area in a way that can then be understood and reused by others. The main things I believe need addressing are: 1) Revisit the criteria, show how you have come to decide on them, give some examples of why they are important, and address potential missing criteria. 2) Clarify the discussion of challenges to adoption and make it clearer which tradeoffs are important to practitioners. 3) Be clearer about which sorts of research workflow are best suited to this approach. B2.Technical soundness: here I am discussing the soundness of the paper, rather than the soundness of the Maneage tool. There are some fundamental additional challenges to reproducibility that are not addressed. Although software library versions are addressed, there is also the challenge of mathematical reproducibility, particularly of the handling of floating point number, which might occur because of the way the code is written, and the hardware architecture (including if code is optimised / parallelised). This could obviously be addressed through a criterion around how code is written, but this will also come with a tradeoff against performance, which is never mentioned. Another tradeoff, which might affect Criterion 3 is time to result - people use popular frameworks because it is easier to use them. Regarding the discussion, I would liked to have seen explanation of how these challenges to adoption were identified: was this anecdotal, through surveys. participant observation? As a side note around the technical aspects of Maneage - it is using LaTeX which in turn is built on TeX which in turn has had many portability problems in the past due to being written using WEB / Tangle, though with web2c this is largely now resolved - potentially an interesting sidebar to investigate how LaTeX/TeX has ensured its longevity! C2. The title is not specific enough - it should refer to the reproducibility of workflows/projects. C4. As noted above, whilst the thesis stated is valid, it may not be useful to practitioners of computation science and engineering as it stands. C6. Manuscript focus. I would have liked a more focussed approach to the presentation of information in II. Longevity is not defined, and whilst various tools are discussed and discarded, no attempt is made to categorise the magnitude of longevity for which they are relevant. For instance, environment isolators are regarded by the software preservation community as adequate for timescale of the order of years, but may not be suitable for the timescale of decades where porting and emulation are used. The title of this section "Commonly used tools and their longevity" is also confusing - do you mean the longevity of the tools or the longevity of the workflows that can be produced using these tools? What happens if you use a combination of all four categories of tools? C8. Readability. I found it difficult to follow the description of how Maneage works. It wasn't clear to me if code was being run to generate the results and figures in a LaTeX paper that is part of a project in Maneage. It appears to be suggested this is the case, but Figure 1 doesn't show how this works - it just has the LaTeX files, the data files and the Makefiles. Is it being suggested that LaTeX itself is the programming language, using its macro functionality? I was a bit confused on how collaboration is handled as well - this appears to be using the Git branching model, and the suggestion that Maneage is keeping track of all components from all projects - but what happens if you are working with collaborators that are using their own Maneage instance? I would also liked to have seen a comparison between this approach and other "executable" paper approaches e.g. Jupyter notebooks, compared on completeness, time taken to write a "paper", ease of depositing in a repository, and ease of use by another researcher. Additional Questions: 1. How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section.: Relevant 2. To what extent is this manuscript relevant to readers around the world?: The manuscript is of interest to readers throughout the world 1. Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical.: This manuscript discusses the challenges of reproducibility of computational research workflows, suggests criteria for improving the "longevity" of workflows, describes the proof-of-concept tool, Maneage, that has been built to implement these criteria, and discusses the challenges to adoption. Of primary importance is the discussion of the challenges to adoption, as CiSE is about computational science which does not take place in a theoretical vacuum. Many of the identified challenges relate to the practice of computational science and the implementation of systems in the real world. 2. Is the manuscript technically sound? Please explain your answer in the Detailed Comments section.: Partially 3. What do you see as this manuscript's contribution to the literature in this field?: The manuscript makes a modest contribution to the literature through the description of the proof-of-concept, in particular its approach to integrating asset management, version control and build and the discussion of challenges to adoption. The proposed criteria have mostly been discussed at length in many other works looking at computational reproducibility and executable papers. 4. What do you see as the strongest aspect of this manuscript?: The strongest aspect is the discussion of difficulties for widespread adoption of this sort of approach. Because the proof-of-concept tool received support through the RDA, it was possible to get feedback from researchers who were likely to use it. This has highlighted and reinforced a number of challenges and caveats. 5. What do you see as the weakest aspect of this manuscript?: The weakest aspect is the assumption that research can be easily compartmentalized into simple and complete packages. Given that so much of research involves collaboration and interaction, this is not sufficiently addressed. In particular, the challenge of interdisciplinary work, where there may not be common languages to describe concepts and there may be different common workflow practices will be a barrier to wider adoption of the primary thesis and criteria. 1. Does the manuscript contain title, abstract, and/or keywords?: Yes 2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: No 3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: References are sufficient and appropriate 4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Could be improved 5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Satisfactory 6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Could be improved 7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Satisfactory 8. Please rate and comment on the readability of this manuscript in the Detailed Comments section.: Readable - but requires some effort to understand 9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are of immediate and continuing interest to CiSE readers Please rate the manuscript. Explain your choice in the Detailed Comments section.: Good -------------------------------------------------- Reviewer: 5 Recommendation: Author Should Prepare A Major Revision For A Second Review Comments: Major figures currently working in this exact field do not have their work acknowledged in this work. In no particular order: Victoria Stodden, Michael Heroux, Michela Taufer, and Ivo Jimenez. All of these authors have multiple publications that are highly relevant to this paper. In the case of Ivo Jimenez, his Popper work [Jimenez I, Sevilla M, Watkins N, Maltzahn C, Lofstead J, Mohror K, Arpaci-Dusseau A, Arpaci-Dusseau R. The popper convention: Making reproducible systems evaluation practical. In2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) 2017 May 29 (pp. 1561-1570). IEEE.] and the later revision that uses GitHub Actions, is largely the same as this work. The lack of attention to virtual machines and containers is highly problematic. While a reader cannot rely on DockerHub or a generic OS version label for a VM or container, these are some of the most promising tools for offering true reproducibility. On the data side, containers have the promise to manage data sets and workflows completely [Lofstead J, Baker J, Younge A. Data pallets: containerizing storage for reproducibility and traceability. InInternational Conference on High Performance Computing 2019 Jun 16 (pp. 36-45). Springer, Cham.] Taufer has picked up this work and has graduated a MS student working on this topic with a published thesis. See also Jimenez's P-RECS workshop at HPDC for additional work highly relevant to this paper. Some other systems that do similar things include: reprozip, occam, whole tale, snakemake. While the work here is a good start, the paper needs to include the context of the current community development level to be a complete research paper. A revision that includes evaluation of (using the criteria) and comparison with the suggested systems and a related work section that seriously evaluates the work of the recommended authors, among others, would make this paper worthy for publication. Additional Questions: 1. How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section.: Very Relevant 2. To what extent is this manuscript relevant to readers around the world?: The manuscript is of interest to readers throughout the world 1. Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical.: This paper describes the Maneage system for reproducibile workflows. It lays out a bit of the need, has very limited related work, and offers criteria any system that offers reproducibility should have, and finally describes how Maneage achieves these goals. 2. Is the manuscript technically sound? Please explain your answer in the Detailed Comments section.: Partially 3. What do you see as this manuscript's contribution to the literature in this field?: Yet another example of a reproducible workflows project. There are numerous examples, mostly domain specific, and this one is not the most advanced general solution. 4. What do you see as the strongest aspect of this manuscript?: Working code and published artifacts 5. What do you see as the weakest aspect of this manuscript?: Lack of context in the field missing very relevant work that eliminates much, if not all, of the novelty of this work. 1. Does the manuscript contain title, abstract, and/or keywords?: Yes 2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: Yes 3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: Important references are missing; more references are needed 4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Could be improved 5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Satisfactory 6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Could be improved 7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Could be improved 8. Please rate and comment on the readability of this manuscript in the Detailed Comaments section.: Easy to read 9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are likely to be of growing interest to CiSE readers over the next 12 months Please rate the manuscript. Explain your choice in the Detailed Comments section.: Fair