From: Computing in Science and Engineering To: mohammad akhlaghi org, infantesainz gmail com, boud astro uni torun pl, mkhellat ideal-information.com, david.valls-gabaud observatoiredeparis psl eu, rbaena iac es Cc: cise@computer.org, cise-rr@computer.org Received: Wed, 7 Apr 2021 19:39:59 +0000 Subject: Decision - Computing in Science and Engineering, CiSESI-2020-06-0048.R1 -------------------------------------------------- Computing in Science and Engineering, CiSESI-2020-06-0048.R1 "Towards Long-term and Archivable Reproducibility" manuscript type: Reproducible Research Dear Dr. Mohammad Akhlaghi: Congratulations! Your manuscript, "Towards Long-term and Archivable Reproducibility," CiSESI-2020-06-0048.R1, has been accepted for publication in an upcoming issue of Computing in Science and Engineering, subject to a final light copyedit. Do note the editors' comments below. Thank you, Lorena A. Barba Editor in Chief, Computing in Science and Engineering labarba gwu edu ********** Editor-in-Chief's Comments ********** - I am processing this as an "accept" to expedite, but please take notice/care of the following items before submitting your final files. - Note particularly that you have to edit any usage of a reference (like [1]) as part of speech in a sentence. Use alternatives like "Smith et al. [1]." The article template uses superscripts for references! - I strongly recommend that you deposit the appendices in arXiv, separately from the main preprint. This way, all the cited works will get their citation indexed by Google Scholar, which the authors will likely appreciate. - You may add the arXiv id for the manuscript and the appendices in your reproducibility statement at the end of the Abstract. - I have manually edited the due date for your final files for Friday April 9 because we are finalizing the next issue. If we received your files ASAP we can include your article in the next issue. Associate Editor Comments: ********** (There are no comments) Reviewers' Comments ********** Reviewer: 1 Recommendation: Accept With No Changes Comments: A more in depth evaluation of different options from a technical standpoint rather than just discussion would really strengthen the paper showing that the idea isn't just good enough to generate this paper that does not contain experimental code executions and benchmarking demonstrating different system characteristics. Deeper acknowledgement of the differences each system brings and over time as they are improved could be greatly expanded. This is an area of extreme concern. Additional Questions: 1. How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section.: Very Relevant 2. To what extent is this manuscript relevant to readers around the world?: The manuscript is of interest to readers throughout the world 1. Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical: This paper presents necessary requirements for reproducible work. This is aimed at being able to generate the same output from a workflow of inputs, test code, and data processing into a text report format. The concepts presented cover a wide variety of topics hitting on the vast majority of cases. The kinds of things not really addressed, such as minor hardware version differences not evident except in physical stamps on the part itself, but yielding slightly different behavior, are mentioned at a high level and left unaddressed. These concerns are part of the longevity discussion instead and limit the lifetime of an artifact. The topics presented cover the topic reasonably well and offer a good guide for people to think about how best to approach providing a reproducible scientific system. Most favorably, the paper itself is offered as an example with an embedded, machine generated version ID and link to the source materials. This is truly putting your money where your mouth is, so to speak. 2. Is the manuscript technically sound? Please explain your answer in the Detailed Comments section.: Yes 3. What do you see as this manuscript's contribution to the literature in this field?: Clearly defining what the longevity limitations are for reproducible work is crucial for us as a community to have effective discussions about what we mean for something to be reproducible. Unless we can agree what acceptable longevity is, we cannot agree if something is reproducible or not. The other factors listed for different components are important things to consider and cover basically everything necessary. 4. What do you see as the strongest aspect of this manuscript?: A simple, easy to follow discussion of what reproducibility really means and how to achieve it. 5. What do you see as the weakest aspect of this manuscript?: Few examples of the process itself and no testing/comparison of any of the variants in the supplemental materials. 1. Does the manuscript contain title, abstract, and/or keywords?: Yes 2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: Yes 3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: References are sufficient and appropriate 4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Yes 5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Satisfactory 6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Satisfactory 7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Satisfactory 8. Please rate and comment on the readability of this manuscript in the Detailed Comments section.: Easy to read 9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are of immediate and continuing interest to CiSE readers Please rate the manuscript. Explain your choice in the Detailed Comments section.: Good