aboutsummaryrefslogtreecommitdiff
path: root/reproduce/analysis/bash/download-multi-try
diff options
context:
space:
mode:
authorMohammad Akhlaghi <mohammad@akhlaghi.org>2019-04-15 01:47:58 +0100
committerMohammad Akhlaghi <mohammad@akhlaghi.org>2019-04-15 02:24:09 +0100
commit313b936b502d22b6a2ff43f560dee0bb51fd01d0 (patch)
tree70f884b91b393be4d3c6b7cfaeaf3412900bd16f /reproduce/analysis/bash/download-multi-try
parent4722ea598edd6b630227404c48c1c09ac527e9b8 (diff)
New architecture to separate software-building and analysis steps
Until now, the software building and analysis steps of the pipeline were intertwined. However, these steps (of how to build a software, and how to use it) are logically completely independent. Therefore with this commit, the pipeline now has a new architecture (particularly in the `reproduce' directory) to emphasize this distinction: The `reproduce' directory now has the two `software' and `analysis' subdirectories and the respective parts of the previous architecture have been broken up between these two based on their function. There is also no more `src' directory. The `config' directory for software and analysis is now mixed with the language-specific directories. Also, some of the software versions were also updated after some checks with their webpages. This new architecture will allow much more focused work on each part of the pipeline (to install the software and to run them for an analysis).
Diffstat (limited to 'reproduce/analysis/bash/download-multi-try')
-rwxr-xr-xreproduce/analysis/bash/download-multi-try112
1 files changed, 112 insertions, 0 deletions
diff --git a/reproduce/analysis/bash/download-multi-try b/reproduce/analysis/bash/download-multi-try
new file mode 100755
index 0000000..1fd7497
--- /dev/null
+++ b/reproduce/analysis/bash/download-multi-try
@@ -0,0 +1,112 @@
+# Attempt downloading multiple times before crashing whole project. From
+# the top project directory (for the shebang above), this script must be
+# run like this:
+#
+# $ /path/to/download-multi-try downloader lockfile input-url downloaded-name
+#
+# NOTE: The `downloader' must contain the option to specify the output name
+# in its end. For example "wget -O". Any other option can also be placed in
+# the middle.
+#
+# Due to temporary network problems, a download may fail suddenly, but
+# succeed in a second try a few seconds later. Without this script that
+# temporary glitch in the network will permanently crash the project and
+# it can't continue. The job of this script is to be patient and try the
+# download multiple times before crashing the whole project.
+#
+# LOCK FILE: Since there is ultimately only one network port to the outside
+# world, downloading is done much faster in serial, not in parallel. But
+# the project's processing may be done in parallel (with multiple threads
+# needing to download different files at the same time). Therefore, this
+# script uses the `flock' program to only do one download at a time. To
+# benefit from it, any call to this script must be given the same lock
+# file.
+#
+# Copyright (C) 2019 Mohammad Akhlaghi <mohammad@akhlaghi.org>
+#
+# This script is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This script is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
+# Public License for more details. See <http://www.gnu.org/licenses/>.
+
+
+
+
+
+# Script settings
+# ---------------
+# Stop the script if there are any errors.
+set -e
+
+
+
+
+
+# Input arguments and necessary sanity checks.
+inurl="$3"
+outname="$4"
+lockfile="$2"
+downloader="$1"
+if [ "x$downloader" = x ]; then
+ echo "$0: downloader (first argument) not given."; exit 1;
+fi
+if [ "x$lockfile" = x ]; then
+ echo "$0: lock file (second argument) not given."; exit 1;
+fi
+if [ "x$inurl" = x ]; then
+ echo "$0: full input URL (third argument) not given."; exit 1;
+fi
+if [ "x$outname" = x ]; then
+ echo "$0: output name (fourth argument) not given."; exit 1;
+fi
+
+
+
+
+
+# Try downloading multiple times before crashing.
+counter=0
+maxcounter=10
+while [ ! -f "$outname" ]; do
+
+ # Increment the counter. We need the `counter=' part here because
+ # without it the evaluation of arithmetic expression will be like and
+ # error and the script is set to crash on errors.
+ counter=$((counter+1))
+
+ # If we have passed a maximum number of trials, just exit with
+ # a failed code.
+ if (( counter > maxcounter )); then
+ echo
+ echo "Failed $maxcounter download attempts: $outname"
+ echo
+ exit 1
+ fi
+
+ # If this isn't the first attempt print a notice and wait a little for
+ # the next trail.
+ if (( counter > 1 )); then
+ tstep=$((counter*5))
+ echo "Download trial $counter for '$outname' in $tstep seconds."
+ sleep $tstep
+ fi
+
+ # Attempt downloading the file (one-at-a-time). Note that the
+ # `downloader' ends with the respective option to specify the output
+ # name. For example "wget -O" (so `outname', that comes after it) will
+ # be the name of the downloaded file.
+ flock "$lockfile" bash -c \
+ "if ! $downloader $outname $inurl; then rm -f $outname; fi"
+done
+
+
+
+
+
+# Return successfully
+exit 0