From e9dfc5bf433fbf61b97179e3243a20e06a4b01dc Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Wed, 2 Dec 2020 15:41:29 +0000 Subject: Minor edits in newly added parts on statistical verification Thanks a lot Boud for adding that script in your own project and linking it here. Since the raw file (without context of the whole project) is very hard to understand for the users, I switched the URL to the navigable URL the link is actually on the filename. It will always show the most recent version of this script, not the particular snapshot of now. But infact that is better, since we can make it better and improve it over time. Maybe even by the end of this paper's referee review will be able to include it in Maneage's core branch. I also removed the link to this discussion at the first paragraph of Section IV (proof of concept). Since that is just the introduction, and going into this level of detail there could be confusing for the readers. Having the name of the script in the proper place is more direct and understandable for the readers. Thanks again Boud for the nice work on this ;-). --- paper.tex | 5 +++-- peer-review/1-answer.txt | 12 ++++++------ 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/paper.tex b/paper.tex index 98498ad..f300fb9 100644 --- a/paper.tex +++ b/paper.tex @@ -287,7 +287,7 @@ In such cases, it is best to immediately convert the data upon collection, and a \section{Proof of concept: Maneage} With the longevity problems of existing tools outlined above, a proof-of-concept tool is presented here via an implementation that has been tested in published papers \cite{akhlaghi19, infante20}. -\new{Since the initial submission of this paper, it has also been used in \href{https://doi.org/10.5281/zenodo.3951151}{zenodo.3951151} (on the COVID-19 pandemic) and \href{https://doi.org/10.5281/zenodo.4062460}{zenodo.4062460} (which \href{https://archive.softwareheritage.org/swh:1:cnt:4217e24e4a474ba43a4d30abfb0a42b823ef4640}{illustrates statistical reproducibility}, e.g., for parallelised code).} +\new{Since the initial submission of this paper, it has also been used in \href{https://doi.org/10.5281/zenodo.3951151}{zenodo.3951151} (on the COVID-19 pandemic) and \href{https://doi.org/10.5281/zenodo.4062460}{zenodo.4062460}.} It was also awarded a Research Data Alliance (RDA) adoption grant for implementing the recommendations of the joint RDA and World Data System (WDS) working group on Publishing Data Workflows \cite{austin17}, from the researchers' perspective. The tool is called Maneage, for \emph{Man}aging data Lin\emph{eage} (the ending is pronounced as in ``lineage''), hosted at \url{https://maneage.org}. @@ -408,7 +408,8 @@ Other built files (intermediate analysis steps) cascade down in the lineage to o Just before reaching the ultimate target (\inlinecode{paper.pdf}), the lineage reaches a bottleneck in \inlinecode{verify.mk} to satisfy the verification criteria (this step was not yet available in \cite{akhlaghi19, infante20}). All project deliverables (macro files, plot or table data and other datasets) are verified at this stage, with their checksums, to automatically ensure exact reproducibility. -Where exact reproducibility is not possible \new{(for example, due to parallelization)}, values can be verified by \new{\href{https://archive.softwareheritage.org/swh:1:cnt:4217e24e4a474ba43a4d30abfb0a42b823ef4640}{a statistical method specified}} by the project authors. +Where exact reproducibility is not possible \new{(for example, due to parallelization)}, values can be verified by the project authors. +\new{For example see \new{\href{https://archive.softwareheritage.org/browse/origin/content/?branch=refs/heads/postreferee_corrections&origin_url=https://codeberg.org/boud/elaphrocentre.git&path=reproduce/analysis/bash/verify-parameter-statistically.sh}{verify-parameter-statistically.sh}} of \href{https://doi.org/10.5281/zenodo.4062460}{zenodo.4062460}.} \begin{figure*}[t] \begin{center} \includetikz{figure-branching}{scale=1}\end{center} diff --git a/peer-review/1-answer.txt b/peer-review/1-answer.txt index 76c574e..ae28c5f 100644 --- a/peer-review/1-answer.txt +++ b/peer-review/1-answer.txt @@ -703,15 +703,15 @@ discussed in Section IV, in the part on verification ("Where exact reproducibility is not possible (for example due to parallelization), values can be verified by a statistical method specified by the project authors."). We have linked keywords in the latter sentence to a Software -Heritage URI [swh] with the specific file in the Peper and Roukema -Maneage'd paper that illustrates an example of how statistical -verification of parallelised code can work in practice. +Heritage URI [1] with the specific file in a Maneage'd paper that +illustrates an example of how statistical verification of parallelised code +can work in practice (Peper & Roukema 2020; zenodo.4062460). We would be interested to hear if any other papers already exist that use -automatic statistical verification of parallelised code as has been done -in this Maneage'd paper. +automatic statistical verification of parallelised code as has been done in +this Maneage'd paper. -[swh] https://archive.softwareheritage.org/swh:1:cnt:4217e24e4a474ba43a4d30abfb0a42b823ef4640 +[1] https://archive.softwareheritage.org/browse/origin/content/?branch=refs/heads/postreferee_corrections&origin_url=https://codeberg.org/boud/elaphrocentre.git&path=reproduce/analysis/bash/verify-parameter-statistically.sh ------------------------------ -- cgit v1.2.1