diff options
Diffstat (limited to 'peer-review')
| -rw-r--r-- | peer-review/1-answer.txt | 154 |
1 files changed, 85 insertions, 69 deletions
diff --git a/peer-review/1-answer.txt b/peer-review/1-answer.txt index 5c27866..b837ce4 100644 --- a/peer-review/1-answer.txt +++ b/peer-review/1-answer.txt @@ -32,10 +32,9 @@ reader can easily access them. 2. [Associate Editor] There are general concerns about the paper lacking focus -ANSWER: We believe that by responding to the specific concerns raised -by the reviewers, as detailed below, we have tightened the focus of -the paper. - +ANSWER: With all the corrections/clarifications that have been done in this +review the focus of the paper should be clear now. We are very grateful to +the thorough listing of points by the referees. ------------------------------ @@ -46,9 +45,10 @@ the paper. 3. [Associate Editor] Some terminology is not well-defined (e.g. longevity). -ANSWER: Longevity has now been defined in the first paragraph of Section -II. With this definition, the main argument of the paper is clearer, -thank you (and thank you to the referees for highlighting this). +ANSWER: Reproducibility, Longevity and Usage have now been explicitly +defined in the first paragraph of Section II. With this definition, the +main argument of the paper is clearer, thank you (and thank you to the +referees for highlighting this). ------------------------------ @@ -133,14 +133,16 @@ future. is on the article, it is important that readers not be confused when they visit your site to use your tools. -ANSWER: Improving the consistency between this research paper and -the Maneage website is a useful recommendation. We have listed -this together with point 29 below at -https://savannah.nongnu.org/task/index.php?15823 -on the Maneage development task list. As indicated there, the -website is developed on a public git repository, so any specific -proposals for improvements can be handled efficiently and -transparently. +ANSWER: Thank you for raising this important point. We have broken down the +very long "About" page into multiple pages to help in readability: + +https://maneage.org/about.html + +Generally, the webpage will soon undergo major improvements to be even more +clear. The website is developed on a public git repository +(https://git.maneage.org/webpage.git), so any specific proposals for +improvements can be handled efficiently and transparently and we welcome +any feedback in this aspect. ------------------------------ @@ -602,9 +604,9 @@ level of peer-review control. Tutorial. A topic breakdown is interesting, as the markdown reading may be too long to find information. -ANSWER: Thank you for the very useful suggestion. We have listed this as -a task at https://savannah.nongnu.org/task/index.php?15823 . - +ANSWER: Thank you very much for this good suggestion, it has been +implemented: https://maneage.org/about.html . The webpage will continuously +be improved and such feedback is always very welcome. ------------------------------ @@ -696,18 +698,17 @@ highly modular and flexible nature of Makefiles run via 'Make'. which might occur because of the way the code is written, and the hardware architecture (including if code is optimised / parallelised). +ANSWER: Floating point errors and optimizations have been mentioned in the +discussion (Section V). The issue with parallelization has also been +discussed in Section IV, in the part on verification ("Where exact +reproducibility is not possible (for example due to parallelization), +values can be verified by any statistical means, specified by the project +authors."). -ANSWER: The authors of particular projects have to choose the level -floating point reproducibility that they judge viable. In section IV, -within the 6500-word limit, this is briefly described in the discussion -of the "verify.mk" rule file. The main paragraph is "Just before reaching ... -All project deliverables ... are verified ... with their checksums, to -automatically ensure exact reproducibility. .... [or] by any statistical -means, specified by the project authors." - -We have added a brief reference to zenodo.3951151, pointing out that -it illustrates an approach for statistical verifiability of -parallelised code using Maneage. +##################### +Find a good way to link to (Peper and Roukema: +https://doi.org/10.5281/zenodo.4062460) +##################### ------------------------------ @@ -719,26 +720,44 @@ parallelised code using Maneage. [reproducibility] ... will come with a tradeoff agianst performance, which is never mentioned. -ANSWER: The criteria we propose and the proof-of-concept with -Maneage do not force the choice of a tradeoff between exact bitwise -floating point reproducibility versus performance (e.g. speed). The -specific concepts of "verification" and "reproducibility" will vary -between domains of scientific computation, but we expect that the -criteria allow this wide range. We did not add text on this point. +ANSWER: The criteria we propose and the proof-of-concept with Maneage do +not force the choice of a tradeoff between exact bitwise floating point +reproducibility versus performance (e.g. speed). The specific concepts of +"verification" and "reproducibility" will vary between domains of +scientific computation, but we expect that the criteria allow this wide +range. +Performance is indeed an important issue for _immediate_ reproducibility +and we would have liked to discuss it. But due to the strict word-count, we +feel that adding it to the discussion points, without having adequate space +to elaborate, can confuse the readers away from the focus of this paper +(which is focused on long term usability). It has therefore not been added. ------------------------------ + + + + 38. [Reviewer 4] Tradeoff, which might affect Criterion 3 is time to result, people use popular frameworks because it is easier to use them. -ANSWER: Section IV includes some quantified examples of timing -involved in the Maneage implementation of the criteria of our -paper. It is true that the initial build time of a Maneage install -may discourage some scientists; but a serious scientific research -project is never started and completed on a time scale of a few -hours. +ANSWER: That is true. In section IV, we have given the time it takes to +build Maneage (only once on each computer) to be around 1.5 hours on an +8-core CPU (a typical machine that may be used for data analysis). We +therefore conclude that when the analysis is complex (and thus taking many +hours or days to complete), this time is negligible. +But if the project's full analysis takes 10 minutes or less (like the +extremely simple analysis done in this paper which takes a fraction of a +second). Indeed, the 1.5 hour building time is significant. In those cases, +as discussed in the main body, the project can be built once in a Docker +image and easily moved to other computers. + +Generally, it is true that the initial configuration time (only once on +each computer) of a Maneage install may discourage some scientists; but a +serious scientific research project is never started and completed on a +time scale of a few hours. ------------------------------ @@ -772,14 +791,16 @@ already written. 40. [Reviewer 4] Potentially an interesting sidebar to investigate how LaTeX/TeX has ensured its longevity! +ANSWER: That is indeed a very interesting subject to study (an obvious link +is that LaTeX/TeX is very strongly based on plain text files). We have been +in touch with Karl Berry (one of the core people behind TeX Live, who also +plays a prominent role in GNU) and have whitnessed the TeX Live community's +efforts to become more and more portable and longer-lived. -ANSWER: We agree that this would be interesting; an obvious link is -that LaTeX/TeX is very strongly based on plain text files, making user -hacking easy, provided that the user is willing to experiment and -search and read through the source files. However, as the reviewer states, -this would be a sidebar, and we are constrained for space. - - +However, as the reviewer states, this would be a sidebar, and we are +constrained for space, so we couldn't find a place to highlight this. But +it is indeed a subject worthy of a full paper (that can be very useful for +many software projects0.. ------------------------------ @@ -790,19 +811,17 @@ this would be a sidebar, and we are constrained for space. 41. [Reviewer 4] The title is not specific enough - it should refer to the reproducibility of workflows/projects. -ANSWER: A problem here is that "workflow" and "project" taken in -isolation risk being vague for wider audiences. Also, we aim at -covering a wider range of aspects of a project than just than the -workflow alone; in the other direction, the word "project" could be -seen as too broad, including the funding, principal investigator, -and team coordination. +ANSWER: A problem here is that "workflow" and "project" taken in isolation +risk being vague for wider audiences. Also, we aim at covering a wider +range of aspects of a project than just than the workflow alone; in the +other direction, the word "project" could be seen as too broad, including +the funding, principal investigator, and team coordination. -A specific title that might be appropriate could be, for example, -"Towards long-term and archivable reproducibility of scientific -computational research projects". Using a term proposed by one of -our reviewers, "Towards long-term and archivable end-to-end -reproducibility of scientific computational research projects" -might also be appropriate. +A specific title that might be appropriate could be, for example, "Towards +long-term and archivable reproducibility of scientific computational +research projects". Using a term proposed by one of our reviewers, "Towards +long-term and archivable end-to-end reproducibility of scientific +computational research projects" might also be appropriate. Nevertheless, we feel that in the context of an article published in CiSE, our current short title is sufficient. @@ -872,7 +891,6 @@ longevity; same as supported CPU architectures." longevity of the workflows that can be produced using these tools? What happens if you use a combination of all four categories of tools? - ANSWER: We have changed the section title to "Longevity of existing tools" to clarify that we refer to longevity of the tools. @@ -950,14 +968,12 @@ effort, without any major difficulties. write a "paper", ease of depositing in a repository, and ease of use by another researcher. - -ANSWER: This type of sociological survey will make sense once the - number of projects run with Maneage is sufficiently high. The - time taken to write a paper should be measurably automatically, - from the git history. The other parameters suggested would - require cooperation from the scientists in responding to - the survey, or will have to be collected anecdotally in the - short term. +ANSWER: This type of sociological survey will make sense once the number of +projects run with Maneage is sufficiently high. The time taken to write a +paper should be measurable automatically: from the git history. The other +parameters suggested would require cooperation from the scientists in +responding to the survey, or will have to be collected anecdotally in the +short term. ------------------------------ |