Imported recent updated in Maneage, minor conflict fixed

The minor conflict was with 'reproduce/software/make/high-level.mk', and in
particular because we implemented the fix to Maneage's Task #15664 in this
project first. After it was moved to the main Maneage branch some minor
stylistic corrections were done to it, thus causing the conflict. To
resolve the conflict, I simply imported the full Maneage version of the
file with this command:

  git checkout maneage -- reproduce/software/make/high-level.mk

The other conflicts were due to the deleted files (that were resolved as
described in 'README-hacking.md') and the LaTeX files that I had told
'.gitattributes' to ignore from the Maneage branch.

3 files changed, 320 insertions, 154 deletions

diff --git a/reproduce/software/shell/bashrc.sh b/reproduce/software/shell/bashrc.sh
index 69bbba6..e846d6a 100755
--- a/reproduce/software/shell/bashrc.sh
+++ b/reproduce/software/shell/bashrc.sh
@@ -31,12 +31,15 @@
 #
 # Copyright (C) 2019-2020 Mohammad Akhlaghi <mohammad@akhlaghi.org>
 #
-# This script is part of Maneage. Maneage is free software: you can
-# redistribute it and/or modify it under the terms of the GNU General
-# Public License as published by the Free Software Foundation, either
-# version 3 of the License, or (at your option) any later version.
-#
-# Maneage is distributed in the hope that it will be useful, but WITHOUT
-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
-# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
-# more details. See <http://www.gnu.org/licenses/>.
+# This script is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This script is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this script.  If not, see <http://www.gnu.org/licenses/>.
diff --git a/reproduce/software/shell/configure.sh b/reproduce/software/shell/configure.sh
index 71b07cc..882570e 100755
--- a/reproduce/software/shell/configure.sh
+++ b/reproduce/software/shell/configure.sh
@@ -1,18 +1,21 @@
-#! /bin/sh
+#!/bin/sh
 #
 # Necessary preparations/configurations for the reproducible project.
 #
 # Copyright (C) 2018-2020 Mohammad Akhlaghi <mohammad@akhlaghi.org>
 #
-# This script is part of Maneage. Maneage is free software: you can
-# redistribute it and/or modify it under the terms of the GNU General
-# Public License as published by the Free Software Foundation, either
-# version 3 of the License, or (at your option) any later version.
+# This script is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
 #
-# Maneage is distributed in the hope that it will be useful, but WITHOUT
-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
-# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
-# more details. See <http://www.gnu.org/licenses/>.
+# This script is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this script.  If not, see <http://www.gnu.org/licenses/>.
 
 
 
@@ -316,57 +319,27 @@ static_build=no
 
 
 
-# Check for GNU gettext
-# ---------------------
+# See if we are on a Linux-based system
+# --------------------------------------
 #
-# Some of the basic sofware need GNU gettext which we don't yet install.
-has_gettext=0
-if type msgfmt > /dev/null 2>/dev/null; then has_gettext=1; fi
-if [ $has_gettext = 0 ]; then
-    cat <<EOF
-______________________________________________________
-!!!!!!!          GNU gettext NOT FOUND         !!!!!!!
-
-Some of the core programs in Maneage require a running GNU gettext on the
-host. Unfortuantely we haven't had the time yet to include its installation
-in Maneage, but it is defined as task #15616 so we don't forget. Generally,
-GNU gettext is already available on many systems, so it hasn't been
-reported too commonly. But please consider helping in this if you have
-time and interest.
-
-Until we include gettext's installation in Maneage, please use your package
-manager to install GNU gettext, then re-run the configuration to
-continue. Here are the relevant commands on some common GNU/Linux
-distributions (you'll only need one of them, or a similar command, if you
-use another package manager):
-
-  apt install gettext
-  yum install gettext
-  pacman -S gettext
-
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-
-EOF
-    exit 1
+# Some features are tailored to GNU/Linux systems, while the BSD-based
+# behavior is different. Initially we only tested macOS (hence the name of
+# the variable), but as FreeBSD is also being inlucded in our tests. As
+# more systems get used, we need to tailor these kinds of things better.
+kernelname=$(uname -s)
+if [ x$kernelname = xLinux ]; then
+    on_mac_os=no
+else
+    host_cc=1
+    on_mac_os=yes
 fi
 
 
 
 
 
-# If we are on a Mac OS system
-# ----------------------------
-#
-# For the time being, we'll use the existance of `otool' to see if we are
-# on a Mac OS system or not. Some tools (for example OpenSSL) need to know
-# this.
-#
-# On Mac OS, the building of GCC crashes sometimes while building libiberty
-# with CLang's `g++'. Until we find a solution, we'll just use the host's C
-# compiler.
-if type otool > /dev/null 2>/dev/null; then
-    host_cc=1
-    on_mac_os=yes
+# Print warning if the host CC is to be used.
+if [ x$host_cc = x1 ]; then
     cat <<EOF
 
 ______________________________________________________
@@ -382,14 +355,11 @@ ______________________________________________________
 
 EOF
     sleep 5
-else
-    on_mac_os=no
 fi
 
 
 
 
-
 # Necessary C library element positions
 # -------------------------------------
 #
@@ -984,7 +954,7 @@ version that this project was designed to use in '$depverfile'
 ($gversion). Please re-run after removing the former file:
 
    $ rm $glconf
-   $ ./project configure
+   $ ./project configure -e
 
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
@@ -1177,98 +1147,29 @@ fi
 
 
 
-# Number of threads for basic builds
-# ----------------------------------
+# Number of threads to build software
+# -----------------------------------
 #
-# Since the system might not have GNU Make at this stage, and other Make
-# implementations can't deal with parallel build properly, we'll just
-# default to 1 thread. This is because some versions of Make complain about
-# not having enough 'pipe' (memory) on some systems. After some searching,
-# I found out its because of too many threads. GNU Make will be present on
-# GNU systems (that have `nproc', part of GNU Coreutils). So to simplify
-# the test for GNU Make, we'll just try running `nproc'.
-if which nproc &> /dev/null; then
-    if [ $jobs = 0 ]; then
+# If the user hasn't manually specified the number of threads, see if we
+# can deduce it from the host:
+#  - On systems with GNU Coreutils we have 'nproc'.
+#  - On BSD-based systems (for example FreeBSD and macOS), we have a
+#    'hw.ncpu' in the output of 'sysctl'.
+#  - When none of the above work, just set the number of threads to 1.
+if [ $jobs = 0 ]; then
+    if type nproc > /dev/null 2> /dev/null; then
         numthreads=$(nproc --all);
     else
-        numthreads=$jobs
+        numthreads=$(sysctl -a | awk '/^hw\.ncpu/{print $2}')
+        if [ x"$numthreads" = x ]; then numthreads=1; fi
     fi
 else
-    numthreads=1;
-fi
-
-
-
-
-
-# Build `flock' before other program
-# ----------------------------------
-#
-# Flock (or file-lock) is a unique program that is necessary to serialize
-# the (generally parallel) processing of make when necessary. GNU/Linux
-# machines have it as part of their `util-linux' programs. But to be
-# consistent in non-GNU/Linux systems, we will be using our own build.
-#
-# The reason that `flock' is sepecial is that we need it to serialize the
-# download process of the software tarballs.
-flockversion=$(awk '/flock-version/{print $3}' $depverfile)
-flockchecksum=$(awk '/flock-checksum/{print $3}' $depshafile)
-flocktar=flock-$flockversion.tar.gz
-flockurl=http://github.com/discoteq/flock/releases/download/v$flockversion/
-
-# Prepare/download the tarball.
-if ! [ -f $tardir/$flocktar ]; then
-    flocktarname=$tardir/$flocktar
-    ucname=$flocktarname.unchecked
-    if [ -f $ddir/$flocktar ]; then
-        cp $ddir/$flocktar $ucname
-    else
-        if ! $downloader $ucname $flockurl/$flocktar; then
-            rm -f $ucname;
-            echo
-            echo "DOWNLOAD ERROR: Couldn't download the 'flock' tarball:"
-            echo "  $flockurl"
-            echo
-            echo "You can manually place it in '$ddir' to avoid downloading."
-            exit 1
-        fi
-    fi
-
-    # Make sure this is the correct tarball.
-    if type sha512sum > /dev/null 2>/dev/null; then
-        checksum=$(sha512sum "$ucname" | awk '{print $1}')
-        if [ x$checksum = x$flockchecksum ]; then mv "$ucname" "$flocktarname"
-        else echo "ERROR: Non-matching checksum for '$flocktar'."; exit 1
-        fi;
-    else mv "$ucname" "$flocktarname"
-    fi
-fi
-
-# If the tarball is newer than the (possibly existing) program (the version
-# has changed), then delete the program.
-if [ -f .local/bin/flock ]; then
-    if [ $tardir/$flocktar -nt $ibidir/flock ]; then
-        rm $ibidir/flock
-    fi
-fi
-
-# Build `flock' if necessary.
-if ! [ -f $ibidir/flock ]; then
-    cd $tmpblddir
-    tar xf $tardir/$flocktar
-    cd flock-$flockversion
-    ./configure --prefix=$instdir
-    make
-    make install
-    cd $topdir
-    rm -rf $tmpblddir/flock-$flockversion
-    echo "Discoteq flock $flockversion" > $ibidir/flock
+    numthreads=$jobs
 fi
 
 
 
 
-
 # Paths needed by the host compiler (only for `basic.mk')
 # -------------------------------------------------------
 #
@@ -1295,14 +1196,28 @@ fi
 
 
 
-# Build basic software
-# --------------------
+# Build core tools for project
+# ----------------------------
+#
+# Here we build the core tools that 'basic.mk' depends on: Lzip
+# (compression program), GNU Make (that 'basic.mk' is written in), Dash
+# (minimal Bash-like shell) and Flock (to lock files and enable serial
+# download).
+./reproduce/software/shell/pre-make-build.sh \
+    "$bdir" "$ddir" "$downloader"
+
+
+
+
+
+# Build other basic tools our own GNU Make
+# ----------------------------------------
 #
 # When building these software we don't have our own un-packing software,
 # Bash, Make, or AWK. In this step, we'll install such low-level basic
 # tools, but we have to be very portable (and use minimal features in all).
 echo; echo "Building necessary software (if necessary)..."
-make -k -f reproduce/software/make/basic.mk \
+.local/bin/make -k -f reproduce/software/make/basic.mk \
      sys_library_path=$sys_library_path \
      rpath_command=$rpath_command \
      static_build=$static_build \
@@ -1374,11 +1289,10 @@ analysis, you can safely ignore this warning and continue.
 
 If you later have internet access and would like to add TeX live to your
 project, please delete the respective files, then re-run configure as shown
-below. Within configure, answer 'n' (for "no") when asked to re-write the
-configuration files.
+below.
 
     rm .local/version-info/tex/texlive-ready-tlmgr
-    ./project configure
+    ./project configure -e
 
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
diff --git a/reproduce/software/shell/pre-make-build.sh b/reproduce/software/shell/pre-make-build.sh
new file mode 100755
index 0000000..e2ac789
--- /dev/null
+++ b/reproduce/software/shell/pre-make-build.sh
@@ -0,0 +1,249 @@
+#!/bin/sh
+#
+# Very basic tools necessary to start Maneage's default building.
+#
+# Copyright (C) 2020 Mohammad Akhlaghi <mohammad@akhlaghi.org>
+#
+# This script is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This script is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this script.  If not, see <http://www.gnu.org/licenses/>.
+
+
+
+
+
+# Script settings
+# ---------------
+# Stop the script if there are any errors.
+set -e
+
+
+
+
+
+# Input arguments.
+bdir=$1
+ddir=$2
+downloader="$3"
+
+
+
+
+
+# Basic directories/files
+topdir=$(pwd)
+sdir=$bdir/software
+tardir=$sdir/tarballs
+instdir=$sdir/installed
+tmpblddir=$sdir/build-tmp
+confdir=reproduce/software/config
+ibidir=$instdir/version-info/proglib
+downloadwrapper=reproduce/analysis/bash/download-multi-try
+
+# Derived directories
+bindir=$instdir/bin
+versionsfile=$confdir/versions.conf
+checksumsfile=$confdir/checksums.conf
+backupfile=$confdir/servers-backup.conf
+
+
+
+
+# Set the system to first look into our newly installed programs.
+export PATH="$bindir:$PATH"
+
+
+
+
+
+# Load the backup servers
+backupservers=$(awk '!/^#/{printf "%s ", $1}' $backupfile)
+
+
+
+
+
+# Download the necessary tarball.
+download_tarball() {
+  # Basic definitions
+  maneagetar=$tardir/$tarball
+
+  # See if the tarball already exists in Maneage.
+  if [ -f "$maneagetar" ]; then
+    just_a_place_holder=1
+  else
+    ucname=$tardir/$tarball.unchecked
+
+    # See if it is in the input software directory.
+    if [ -f "$ddir/$tarball" ]; then
+      cp $ddir/$tarball $ucname
+    else
+      $downloadwrapper "$downloader" nolock $url/$tarball $ucname \
+                       "$backupservers"
+    fi
+
+    # Make sure this is the correct tarball.
+    if type sha512sum > /dev/null 2> /dev/null; then
+      checksum=$(sha512sum "$ucname" | awk '{print $1}')
+      expectedchecksum=$(awk '/^'$progname'-checksum/{print $3}' $checksumsfile)
+      if [ x$checksum = x$expectedchecksum ]; then mv "$ucname" "$maneagetar"
+      else
+        echo "ERROR: Non-matching checksum for '$tarball'."
+        echo "Checksum should be: $expectedchecksum"
+        echo "Checksum is:        $checksum"
+        exit 1
+      fi;
+    else mv "$ucname" "$maneagetar"
+    fi
+  fi
+
+  # If the tarball is newer than the (possibly existing) program (the version
+  # has changed), then delete the program.
+  if [ -f $ibidir/$progname ]; then
+      if [ $maneagetar -nt $ibidir/$progname ]; then
+          rm $ibidir/$progname
+      fi
+  fi
+}
+
+
+
+
+
+# Build the program from the tarball
+build_program() {
+  if ! [ -f $ibidir/$progname ]; then
+
+    # Go into the temporary building directory.
+    cd $tmpblddir
+    unpackdir="$progname"-"$version"
+
+    # Some implementations of 'tar' don't recognize Lzip, so we need to
+    # manually call Lzip first, then call tar afterwards.
+    csuffix=$(echo $tarball | sed -e's/\./ /g' | awk '{print $NF}')
+    rm -rf $unpackdir
+    if [ x$csuffix = xlz ]; then
+      intarrm=1
+      intar=$(echo $tarball | sed -e's/.lz//')
+      lzip -c -d $tardir/$tarball > $intar
+    else
+      intarrm=0
+      intar=$tardir/$tarball
+    fi
+
+    # Unpack the tarball and build the program.
+    tar xf $intar
+    if [ x$intarrm = x1 ]; then rm $intar; fi
+    cd $unpackdir
+    ./configure --prefix=$instdir
+    make
+    make install
+    cd $topdir
+    rm -rf $tmpblddir/$unpackdir
+    echo "$progname_tex $version" > $ibidir/$progname
+  fi
+}
+
+
+
+
+
+# Lzip
+# ----
+#
+# Lzip is a compression program that is the first built program in Maneage
+# because the sources of all other programs (including other compression
+# softwaer) are compressed. Lzip has the advantage that it is very small
+# (without compression it is just ~400Kb). So we use its '.tar' file and
+# won't rely on the host's compression tools at all.
+progname="lzip"
+progname_tex="Lzip"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar
+download_tarball
+build_program
+
+
+
+
+
+# GNU Make
+# --------
+#
+# The job orchestrator of Maneage is GNU Make. Although it is not
+# impossible to account for all the differences between various Make
+# implementations, its much easier (for reading the code and
+# writing/debugging it) if we can count on a special implementation. So
+# before going into the complex job orchestration in building high-level
+# software, we start by building GNU Make.
+progname="make"
+progname_tex="GNU Make"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar.lz
+download_tarball
+build_program
+
+
+
+
+
+# Dash
+# ----
+#
+# Dash is a shell (http://gondor.apana.org.au/~herbert/dash). Having it in
+# this phase will allow us to have a fixed/identical shell for 'basic.mk'
+# (which builds GNU Bash).
+progname="dash"
+progname_tex="Dash"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar.lz
+download_tarball
+build_program
+
+# If the 'sh' symbolic link isn't set yet, set it to point to Dash.
+if [ -f $bindir/sh ]; then just_a_place_holder=1
+else ln -sf $bindir/dash $bindir/sh;
+fi
+
+
+
+
+
+# Flock
+# -----
+#
+# Flock (or file-lock) is necessary to serialize operations when
+# necessary. GNU/Linux machines have it as part of their `util-linux'
+# programs. But to be consistent in non-GNU/Linux systems, we will be using
+# our own build.
+#
+# The reason that `flock' is built here is that generally the building of
+# software is done in parallel, but we need it to serialize the download
+# process of the software tarballs to avoid network complications when too
+# many simultaneous download commands are called.
+progname="flock"
+progname_tex="Discoteq flock"
+url=http://akhlaghi.org/src
+version=$(awk '/^'$progname'-version/{print $3}' $versionsfile)
+tarball=$progname-$version.tar.lz
+download_tarball
+build_program
+
+
+
+
+
+# Finish this script successfully
+exit 0