diff options
Diffstat (limited to 'peer-review')
| -rw-r--r-- | peer-review/1-answer.txt | 102 |
1 files changed, 68 insertions, 34 deletions
diff --git a/peer-review/1-answer.txt b/peer-review/1-answer.txt index 5e612f8..e0b0da1 100644 --- a/peer-review/1-answer.txt +++ b/peer-review/1-answer.txt @@ -32,9 +32,10 @@ reader can easily access them. 2. [Associate Editor] There are general concerns about the paper lacking focus -########################### -ANSWER: -########################### +ANSWER: With all the corrections/clarifications that have been done in this +review the focus of the paper should be clear now. We are very grateful to +the thorough listing of points by the referees. + ------------------------------ @@ -45,9 +46,10 @@ ANSWER: 3. [Associate Editor] Some terminology is not well-defined (e.g. longevity). -ANSWER: Longevity has now been defined in the first paragraph of Section -II. With this definition, the main argument of the paper is clearer, -thank you (and thank you to the referees for highlighting this). +ANSWER: Reproducibility, Longevity and Usage have now been explicitly +defined in the first paragraph of Section II. With this definition, the +main argument of the paper is clearer, thank you (and thank you to the +referees for highlighting this). ------------------------------ @@ -132,10 +134,13 @@ future. is on the article, it is important that readers not be confused when they visit your site to use your tools. -########################### -ANSWER [NOT COMPLETE]: We should separate the various sections of the -README-hacking.md webpage into smaller pages that can be entered. -########################### +ANSWER: Thank you for raising this important point. We have broken down the +very long "About" page into multiple pages to help in readability: + +https://maneage.org/about.html + +Generally, the webpage will soon undergo major improvements to be even more +clear. ------------------------------ @@ -597,9 +602,9 @@ level of peer-review control. Tutorial. A topic breakdown is interesting, as the markdown reading may be too long to find information. -##################################### -ANSWER: -##################################### +ANSWER: Thank you very much for this good suggestion, it has been +implemented: https://maneage.org/about.html . The webpage will continuously +be improved and such feedback is always very welcome. ------------------------------ @@ -691,9 +696,12 @@ highly modular and flexible nature of Makefiles run via 'Make'. which might occur because of the way the code is written, and the hardware architecture (including if code is optimised / parallelised). -################################ -ANSWER: -################################ +ANSWER: Floating point errors and optimizations have been mentioned in the +discussion (Section V). The issue with parallelization has also been +discussed in Section IV, in the part on verification ("Where exact +reproducibility is not possible (for example due to paralleliza- tion), +values can be verified by any statistical means, specified by the project +authors."). ------------------------------ @@ -703,18 +711,32 @@ ANSWER: 37. [Reviewer 4] Performance ... is never mentioned -################################ -ANSWER: -################################ +ANSWER: Performance is indeed an important issue for _immediate_ +reproducibility and we would have liked to discuss it. But due to the +strict word-count, we feel that adding it to the discussion points, without +having adequate space to elaborate, can confuse the readers of this paper +(which is focused on long term usability). ------------------------------ + + + + 38. [Reviewer 4] Tradeoff, which might affect Criterion 3 is time to result, people use popular frameworks because it is easier to use them. -################################ -ANSWER: -################################ +ANSWER: That is true. In section IV, we have given the time it takes to +build Maneage (only once for a project on each computer) to be around 1.5 +hours on an 8-core CPU (a typical machine that may be used for data +analysis). We therefore conclude that when the analysis is complex (and +thus taking many hours or days to complete), this time is negligible. + +But if the project's full analysis takes 10 minutes or less (like the +extremely simple analysis done in this paper which takes a fraction of a +second). Indeed, the 1.5 hour building time is significant. In those cases, +as discussed in the main body, the project can be built once in a Docker +image and easily moved to other computers. ------------------------------ @@ -747,9 +769,13 @@ there. 40. [Reviewer 4] Potentially an interesting sidebar to investigate how LaTeX/TeX has ensured its longevity! -############################## -ANSWER: -############################## +ANSWER: That is indeed a very interesting subject to study. We have been in +touch with Karl Berry (one of the core people behind TeX Live, who also +plays a prominent role in GNU) and have whitnessed the TeX Live community's +efforts to become more and more portable and longer-lived. But after +looking at the strict word limit, we couldn't find a place to highlight +this. But it is indeed a subject worthy of a full paper (that can be very +useful for many software projects0.. ------------------------------ @@ -760,9 +786,11 @@ ANSWER: 41. [Reviewer 4] The title is not specific enough - it should refer to the reproducibility of workflows/projects. -############################## -ANSWER: -############################## +ANSWER: Since this journal is focused on "Computing in Science and +Engineering", the fact that it relates to computational workflows will be +clear to any reader. Since the other referees didn't complain about this, +we will keep it as it was, but of course, we are open to the suggestions of +the editors in the final title. ------------------------------ @@ -820,9 +848,16 @@ determined by the host kernel, usually a decade", for Python packages: longevity of the workflows that can be produced using these tools? What happens if you use a combination of all four categories of tools? -########################## -ANSWER: -########################## +ANSWER: Thank you for highlighting this. The title has been shortend and +the section immediately starts with definitions. + +The aspects of the tools discussed in this section are orthogonal to each +other. For example a VM/container, package manager, notebook: some projects +may have any different combinations of the three. In some aspects using +them together can improve the operations, but for example building a +VM/container with or without a package manager makes no difference on the +main issue we raise about containers (that they are large binary blobs that +don't necessarily contain how the environment within them was built). ------------------------------ @@ -888,9 +923,8 @@ project. This can be generalized to any Git based collaboration model. ease of depositing in a repository, and ease of use by another researcher. -####################### -ANSWER: -####################### +ANSWER: These have been highlighted in various parts of the text (also +reviewed in previous points). ------------------------------ |