aboutsummaryrefslogtreecommitdiff
path: root/reproduce/src/bash/download-multi-try
diff options
context:
space:
mode:
authorMohammad Akhlaghi <mohammad@akhlaghi.org>2019-03-27 19:53:18 +0000
committerMohammad Akhlaghi <mohammad@akhlaghi.org>2019-03-28 09:27:41 +0000
commit98d31767a965ec75f4920b666f236cbb6baa91ab (patch)
treebe68dd523f434ad2138abf6f837447099fc240a7 /reproduce/src/bash/download-multi-try
parent356d998b87d93cc86dc44642ae95fed7478bc788 (diff)
flock is now built in configure, to allow serial downloads
Until now, we were using `flock' (file-lock) for downloading the input datasets in series. But we couldn't do this when downloading the software tarballs because `flock' wasn't yet available. Generally, unlike processing, downloading is much better done in series than in parallel. To enable serial downloads of the software also, with this commit we are installing `flock' in the configure script (not in a Makefile). As a result, besides `flock', we can also benefit from the other good features of the `reproduce/src/bash/download-multi-try' script *(for example attempting download again after some time). Some GNU mirrors may have problems at the time of download, so with this commit, we are using the main GNU FTP server for GNU programs.
Diffstat (limited to 'reproduce/src/bash/download-multi-try')
-rwxr-xr-xreproduce/src/bash/download-multi-try15
1 files changed, 10 insertions, 5 deletions
diff --git a/reproduce/src/bash/download-multi-try b/reproduce/src/bash/download-multi-try
index 1be2ffa..cdeca8e 100755
--- a/reproduce/src/bash/download-multi-try
+++ b/reproduce/src/bash/download-multi-try
@@ -1,11 +1,13 @@
-#!.local/bin/bash
-#
# Attempt downloading multiple times before crashing whole pipeline. From
# the top project directory (for the shebang above), this script must be
# run like this:
#
# $ /path/to/download-multi-try downloader lockfile input-url downloaded-name
#
+# NOTE: The `downloader' must contain the option to specify the output name
+# in its end. For example "wget -O". Any other option can also be placed in
+# the middle.
+#
# Due to temporary network problems, a download may fail suddenly, but
# succeed in a second try a few seconds later. Without this script that
# temporary glitch in the network will permanently crash the pipeline and
@@ -70,7 +72,7 @@ fi
-# Try downloading multiple times before crashing
+# Try downloading multiple times before crashing.
counter=0
maxcounter=10
while [ ! -f "$outname" ]; do
@@ -97,9 +99,12 @@ while [ ! -f "$outname" ]; do
sleep $tstep
fi
- # Attempt downloading the file (one-at-a-time).
+ # Attempt downloading the file (one-at-a-time). Note that the
+ # `downloader' ends with the respective option to specify the output
+ # name. For example "wget -O" (so `outname', that comes after it) will
+ # be the name of the downloaded file.
flock "$lockfile" bash -c \
- "if ! $downloader -O$outname $inurl; then rm -f $outname; fi"
+ "if ! $downloader $outname $inurl; then rm -f $outname; fi"
done