flock is now built in configure, to allow serial downloads

Until now, we were using `flock' (file-lock) for downloading the input
datasets in series. But we couldn't do this when downloading the software
tarballs because `flock' wasn't yet available. Generally, unlike
processing, downloading is much better done in series than in parallel.

To enable serial downloads of the software also, with this commit we are
installing `flock' in the configure script (not in a Makefile). As a
result, besides `flock', we can also benefit from the other good features
of the `reproduce/src/bash/download-multi-try' script *(for example
attempting download again after some time).

Some GNU mirrors may have problems at the time of download, so with this
commit, we are using the main GNU FTP server for GNU programs.

1 files changed, 61 insertions, 2 deletions

diff --git a/configure b/configure
index c8e2b63..c34a68a 100755
--- a/configure
+++ b/configure
@@ -418,8 +418,12 @@ if [ $rewritepconfig = yes ]; then
         $pconf.in >> $pconf
 else
     # Read the values from existing configuration file.
-    inbdir=$(awk     '$1=="BDIR"             {print $3}' $pconf)
-    downloader=$(awk '$1=="DOWNLOADER"       {print $3}' $pconf)
+    inbdir=$(awk     '$1=="BDIR"                     {print $3}' $pconf)
+
+    # The downloader command may contain multiple elements, so we'll just
+    # change the (in memory) first and second tokens to empty space and
+    # write the full line (the original file is unchanged).
+    downloader=$(awk '$1=="DOWNLOADER" {$1=""; $2=""; print $0}' $pconf)
 
     # Make sure all necessary variables have a value
     err=0
@@ -676,6 +680,61 @@ fi
 
 
 
+# Build `flock' as first program
+# ------------------------------
+#
+# Flock (or file-lock) is a unique program in the pipeline that is
+# necessary to serialize the (generally parallel) processing of make when
+# necessary. GNU/Linux machines have it as part of their `util-linux'
+# programs. But to be consistent, we will be using our own build.
+#
+# The reason its sepecial is that we need it to serialize the download
+# process of the dependency tarballs.
+flockversion=$(awk '/flock-version/{print $3}' \
+                   reproduce/config/pipeline/dependency-versions.mk)
+flocktar=flock-$flockversion.tar.gz
+flockurl=http://github.com/discoteq/flock/releases/download/v$flockversion/
+
+# Prepare/download the tarball.
+if ! [ -f $tardir/$flocktar ]; then
+    if [ -f $ddir/$flocktar ]; then
+        cp $ddir/$flocktar $tardir/$flocktar
+    else
+        if ! $downloader $tardir/$flocktar $flockurl/$flocktar; then
+            rm -f $tardir/$flocktar;
+            echo
+            echo "DOWNLOAD ERROR: Couldn't download the 'flock' tarball:"
+            echo "  $flockurl"
+            echo
+            echo "You can manually place it in '$ddir' to avoid downloading."
+            exit 1
+        fi
+    fi
+fi
+
+# If the tarball is newer than the (possibly existing) program, then delete
+# the program.
+if [ -f .local/bin/flock ]; then
+    if [ $tardir/$flocktar -nt .local/bin/flock ]; then
+        rm .local/bin/flock
+    fi
+fi
+
+# Build `flock' if necessary.
+if ! [ -f .local/bin/flock ]; then
+    cd $depdir
+    tar xf $tardir/$flocktar
+    cd flock-$flockversion
+    ./configure --prefix=$instdir
+    make; make install
+    cd $topdir
+    rm -rf $depdir/flock-$flockversion
+fi
+
+
+
+
+
 # Build Basic dependencies
 # ------------------------
 #