1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
|
From: Computing in Science and Engineering <onbehalfof@manuscriptcentral.com>
To: mohammad akhlaghi org,
infantesainz gmail com,
boud astro uni torun pl,
mkhellat ideal-information.com,
david.valls-gabaud observatoiredeparis psl eu,
rbaena iac es
Cc: cise@computer.org,
cise-rr@computer.org
Received: Wed, 7 Apr 2021 19:39:59 +0000
Subject: Decision - Computing in Science and Engineering, CiSESI-2020-06-0048.R1
--------------------------------------------------
Computing in Science and Engineering, CiSESI-2020-06-0048.R1
"Towards Long-term and Archivable Reproducibility"
manuscript type: Reproducible Research
Dear Dr. Mohammad Akhlaghi:
Congratulations! Your manuscript, "Towards Long-term and Archivable
Reproducibility," CiSESI-2020-06-0048.R1, has been accepted for publication
in an upcoming issue of Computing in Science and Engineering, subject to a
final light copyedit. Do note the editors' comments below.
Thank you,
Lorena A. Barba
Editor in Chief, Computing in Science and Engineering
labarba gwu edu
**********
Editor-in-Chief's Comments
**********
- I am processing this as an "accept" to expedite, but please take
notice/care of the following items before submitting your final files.
- Note particularly that you have to edit any usage of a reference (like
[1]) as part of speech in a sentence. Use alternatives like "Smith et
al. [1]." The article template uses superscripts for references!
- I strongly recommend that you deposit the appendices in arXiv, separately
from the main preprint. This way, all the cited works will get their
citation indexed by Google Scholar, which the authors will likely
appreciate.
- You may add the arXiv id for the manuscript and the appendices in your
reproducibility statement at the end of the Abstract.
- I have manually edited the due date for your final files for Friday April
9 because we are finalizing the next issue. If we received your files
ASAP we can include your article in the next issue.
Associate Editor Comments:
**********
(There are no comments)
Reviewers' Comments
**********
Reviewer: 1
Recommendation: Accept With No Changes
Comments:
A more in depth evaluation of different options from a technical standpoint
rather than just discussion would really strengthen the paper showing that
the idea isn't just good enough to generate this paper that does not
contain experimental code executions and benchmarking demonstrating
different system characteristics. Deeper acknowledgement of the differences
each system brings and over time as they are improved could be greatly
expanded. This is an area of extreme concern.
Additional Questions:
1. How relevant is this manuscript to the readers of this periodical?
Please explain your rating in the Detailed Comments section.: Very Relevant
2. To what extent is this manuscript relevant to readers around the world?:
The manuscript is of interest to readers throughout the world
1. Please summarize what you view as the key point(s) of the manuscript and
the importance of the content to the readers of this periodical: This
paper presents necessary requirements for reproducible work. This is aimed
at being able to generate the same output from a workflow of inputs, test
code, and data processing into a text report format.
The concepts presented cover a wide variety of topics hitting on the vast
majority of cases. The kinds of things not really addressed, such as minor
hardware version differences not evident except in physical stamps on the
part itself, but yielding slightly different behavior, are mentioned at a
high level and left unaddressed. These concerns are part of the longevity
discussion instead and limit the lifetime of an artifact.
The topics presented cover the topic reasonably well and offer a good guide
for people to think about how best to approach providing a reproducible
scientific system. Most favorably, the paper itself is offered as an
example with an embedded, machine generated version ID and link to the
source materials. This is truly putting your money where your mouth is, so
to speak.
2. Is the manuscript technically sound? Please explain your answer in the
Detailed Comments section.: Yes
3. What do you see as this manuscript's contribution to the literature in
this field?: Clearly defining what the longevity limitations are for
reproducible work is crucial for us as a community to have effective
discussions about what we mean for something to be reproducible. Unless we
can agree what acceptable longevity is, we cannot agree if something is
reproducible or not. The other factors listed for different components are
important things to consider and cover basically everything necessary.
4. What do you see as the strongest aspect of this manuscript?: A simple,
easy to follow discussion of what reproducibility really means and how to
achieve it.
5. What do you see as the weakest aspect of this manuscript?: Few examples
of the process itself and no testing/comparison of any of the variants in
the supplemental materials.
1. Does the manuscript contain title, abstract, and/or keywords?: Yes
2. Are the title, abstract, and keywords appropriate? Please elaborate in
the Detailed Comments section.: Yes
3. Does the manuscript contain sufficient and appropriate references
(maximum 12-unless the article is a survey or tutorial in scope)? Please
elaborate in the Detailed Comments section.: References are sufficient and
appropriate
4. Does the introduction clearly state a valid thesis? Please explain your
answer in the Detailed Comments section.: Yes
5. How would you rate the organization of the manuscript? Please elaborate
in the Detailed Comments section.: Satisfactory
6. Is the manuscript focused? Please elaborate in the Detailed Comments
section.: Satisfactory
7. Is the length of the manuscript appropriate for the topic? Please
elaborate in the Detailed Comments section.: Satisfactory
8. Please rate and comment on the readability of this manuscript in the
Detailed Comments section.: Easy to read
9. Please rate and comment on the timeliness and long term interest of this
manuscript to CiSE readers in the Detailed Comments section. Select all
that apply.: Topic and content are of immediate and continuing interest to
CiSE readers
Please rate the manuscript. Explain your choice in the Detailed Comments
section.: Good
|
