Core software build before using Make to build other software

Until now, Maneage would only build Flock before building everything else
using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid
parallel downloads during the building of software (which could cause
network problems). But after recently trying Maneage on FreeBSD (which is
not yet complete, see bug #58465), we noticed that the BSD implemenation of
Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq'
parts) and its shell also had some peculiarities.

It was thus decided to also install our own minimalist shell, Make and
compressor program before calling 'basic.mk'. In this way, 'basic.mk' can
now assume the same GNU Make features that high-level.mk and python.mk
assume. The pre-make building of software is now organized in
'reproduce/software/shell/pre-make-build.sh'.

Another nice feature of this commit is for macOS users: until now the
default macOS Make had problems for parallel building of software, so
'basic.mk' was built in one thread. But now that we can build the core
tools with GNU Make on macOS too, it uses all threads. Furthermore, since
we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have
to finish every line of a long rule with a backslash to keep variables and
such.

Generally, the pre-make software are now organized like this: first we
build Lzip before anything else: it is downloaded as a simple '.tar' file
that is not compressed (only ~400kb). Once Lzip is built, the pre-make
phase continues with building GNU Make, Dash (a minimalist shell) and
Flock. All of their tarballs are in '.tar.lz'. Maneage then enters
'basic.mk' and the first program it builds is GNU Gzip (itself packaged as
'.tar.lz'). Once Gzip is built, we build all the other compression software
(all downloaded as '.tar.gz'). Afterwards, any compression standard for
other software is fine because we have it.

In the process, a bug related to using backup servers was found in
'reproduce/analysis/bash/download-multi-try' for calling outside of
'basic.mk' and removed Bash-specific features. As a result of that bug-fix,
because we now have multiple servers for software tarballs, the backup
servers now have their own configuration file in
'reproduce/software/config/servers-backup.conf'. This makes it much easier
to maintain the backup server list across the multiple places that we need
it.

Some other minor fixes:

 - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'.

 - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD
   (which doesn't have GCC).

 - We are now using 'uname -s' to specify if we are on a Linux kernel or
   not, if not, we are still using the old 'on_mac_os' variable.

 - While I was trying to build on FreeBSD, I noticed some further
   corrections that could help. For example the 'makelink' Make-function
   now takes a third argument which can be a different name compared to the
   actual program (used for examle to make a link to '/usr/bin/cc' from
   'gcc'.

 - Until now we didn't know if the host's Make implementation supports
   placing a '@' at the start of the recipe (to avoid printing the actual
   commands to standard output). Especially in the tarball download phase,
   there are many lines that are printed for each download which was really
   annoying. We already used '@' in 'high-level.mk' and 'python.mk' before,
   but now that we also know that 'basic.mk' is called with our custom GNU
   Make, we can use it at the start for a cleaner stdout.

 - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a
   system where we can't install GCC (or has activated the '--host-cc'
   option), it may not be present and the project shouldn't break because
   of this. So with this commit, when a Fortran compiler isn't present,
   WCSLIB will be built with the '--disable-fortran' configuration option.

This commit (task #15667) was completed with help/checks by Raul
Infante-Sainz and Boud Roukema.

1 files changed, 249 insertions, 0 deletions

diff --git a/reproduce/software/shell/pre-make-build.sh b/reproduce/software/shell/pre-make-build.sh
new file mode 100755
index 0000000..e2ac789
--- /dev/null
+++ b/reproduce/software/shell/pre-make-build.sh
@@ -0,0 +1,249 @@
+#!/bin/sh
+#
+# Very basic tools necessary to start Maneage's default building.
+#
+# Copyright (C) 2020 Mohammad Akhlaghi <mohammad@akhlaghi.org>
+#
+# This script is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This script is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this script.  If not, see <http://www.gnu.org/licenses/>.
+
+
+
+
+
+# Script settings
+# ---------------
+# Stop the script if there are any errors.
+set -e
+
+
+
+
+
+# Input arguments.
+bdir=$1
+ddir=$2
+downloader="$3"
+
+
+
+
+
+# Basic directories/files
+topdir=$(pwd)
+sdir=$bdir/software
+tardir=$sdir/tarballs
+instdir=$sdir/installed
+tmpblddir=$sdir/build-tmp
+confdir=reproduce/software/config
+ibidir=$instdir/version-info/proglib
+downloadwrapper=reproduce/analysis/bash/download-multi-try
+
+# Derived directories
+bindir=$instdir/bin
+versionsfile=$confdir/versions.conf
+checksumsfile=$confdir/checksums.conf
+backupfile=$confdir/servers-backup.conf
+
+
+
+
+# Set the system to first look into our newly installed programs.
+export PATH="$bindir:$PATH"
+
+
+
+
+
+# Load the backup servers
+backupservers=$(awk '!/^#/{printf "%s ", $1}' $backupfile)
+
+
+
+
+
+# Download the necessary tarball.
+download_tarball() {
+  # Basic definitions
+  maneagetar=$tardir/$tarball
+
+  # See if the tarball already exists in Maneage.
+  if [ -f "$maneagetar" ]; then
+    just_a_place_holder=1
+  else
+    ucname=$tardir/$tarball.unchecked
+
+    # See if it is in the input software directory.
+    if [ -f "$ddir/$tarball" ]; then
+      cp $ddir/$tarball $ucname
+    else
+      $downloadwrapper "$downloader" nolock $url/$tarball $ucname \
+                       "$backupservers"
+    fi
+
+    # Make sure this is the correct tarball.
+    if type sha512sum > /dev/null 2> /dev/null; then
+      checksum=$(sha512sum "$ucname" | awk '{print $1}')
+      expectedchecksum=$(awk '/^'$progname'-checksum/{print $3}' $checksumsfile)
+      if [ x$checksum = x$expectedchecksum ]; then mv "$ucname" "$maneagetar"
+      else
+        echo "ERROR: Non-matching checksum for '$tarball'."
+        echo "Checksum should be: $expectedchecksum"
+        echo "Checksum is:        $checksum"
+        exit 1
+      fi;
+    else mv "$ucname" "$maneagetar"
+    fi
+  fi
+
+  # If the tarball is newer than the (possibly existing) program (the version
+  # has changed), then delete the program.
+  if [ -f $ibidir/$progname ]; then
+      if [ $maneagetar -nt $ibidir/$progname ]; then
+          rm $ibidir/$progname
+      fi
+  fi
+}
+
+
+
+
+
+# Build the program from the tarball
+build_program() {
+  if ! [ -f $ibidir/$progname ]; then
+
+    # Go into the temporary building directory.
+    cd $tmpblddir
+    unpackdir="$progname"-"$version"
+
+    # Some implementations of 'tar' don't recognize Lzip, so we need to
+    # manually call Lzip first, then call tar afterwards.
+    csuffix=$(echo $tarball | sed -e's/\./ /g' | awk '{print $NF}')
+    rm -rf $unpackdir
+    if [ x$csuffix = xlz ]; then
+      intarrm=1
+      intar=$(echo $tarball | sed -e's/.lz//')
+      lzip -c -d $tardir/$tarball > $intar
+    else
+      intarrm=0
+      intar=$tardir/$tarball
+    fi
+
+    # Unpack the tarball and build the program.
+    tar xf $intar
+    if [ x$intarrm = x1 ]; then rm $intar; fi
+    cd $unpackdir
+    ./configure --prefix=$instdir
+    make
+    make install
+    cd $topdir
+    rm -rf $tmpblddir/$unpackdir
+    echo "$progname_tex $version" > $ibidir/$progname
+  fi
+}
+
+
+
+
+
+# Lzip
+# ----
+#
+# Lzip is a compression program that is the first built program in Maneage
+# because the sources of all other programs (including other compression
+# softwaer) are compressed. Lzip has the advantage that it is very small
+# (without compression it is just ~400Kb). So we use its '.tar' file and
+# won't rely on the host's compression tools at all.
+progname="lzip"
+progname_tex="Lzip"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar
+download_tarball
+build_program
+
+
+
+
+
+# GNU Make
+# --------
+#
+# The job orchestrator of Maneage is GNU Make. Although it is not
+# impossible to account for all the differences between various Make
+# implementations, its much easier (for reading the code and
+# writing/debugging it) if we can count on a special implementation. So
+# before going into the complex job orchestration in building high-level
+# software, we start by building GNU Make.
+progname="make"
+progname_tex="GNU Make"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar.lz
+download_tarball
+build_program
+
+
+
+
+
+# Dash
+# ----
+#
+# Dash is a shell (http://gondor.apana.org.au/~herbert/dash). Having it in
+# this phase will allow us to have a fixed/identical shell for 'basic.mk'
+# (which builds GNU Bash).
+progname="dash"
+progname_tex="Dash"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar.lz
+download_tarball
+build_program
+
+# If the 'sh' symbolic link isn't set yet, set it to point to Dash.
+if [ -f $bindir/sh ]; then just_a_place_holder=1
+else ln -sf $bindir/dash $bindir/sh;
+fi
+
+
+
+
+
+# Flock
+# -----
+#
+# Flock (or file-lock) is necessary to serialize operations when
+# necessary. GNU/Linux machines have it as part of their `util-linux'
+# programs. But to be consistent in non-GNU/Linux systems, we will be using
+# our own build.
+#
+# The reason that `flock' is built here is that generally the building of
+# software is done in parallel, but we need it to serialize the download
+# process of the software tarballs to avoid network complications when too
+# many simultaneous download commands are called.
+progname="flock"
+progname_tex="Discoteq flock"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar.lz
+download_tarball
+build_program
+
+
+
+
+
+# Finish this script successfully
+exit 0