path: root/reproduce/software/shell/configure.sh
diff options
authorMohammad Akhlaghi <mohammad@akhlaghi.org>2020-06-01 03:05:37 +0100
committerMohammad Akhlaghi <mohammad@akhlaghi.org>2020-06-02 22:36:03 +0100
commit3d8aa5953c4e0b79278ab2e27ec4e1051310d04f (patch)
treeba90256edba224edc5300d24e512b24f964ad757 /reproduce/software/shell/configure.sh
parent213dac320133571bca871275c1bf6446ddee33cb (diff)
Core software build before using Make to build other software
Until now, Maneage would only build Flock before building everything else using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid parallel downloads during the building of software (which could cause network problems). But after recently trying Maneage on FreeBSD (which is not yet complete, see bug #58465), we noticed that the BSD implemenation of Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq' parts) and its shell also had some peculiarities. It was thus decided to also install our own minimalist shell, Make and compressor program before calling 'basic.mk'. In this way, 'basic.mk' can now assume the same GNU Make features that high-level.mk and python.mk assume. The pre-make building of software is now organized in 'reproduce/software/shell/pre-make-build.sh'. Another nice feature of this commit is for macOS users: until now the default macOS Make had problems for parallel building of software, so 'basic.mk' was built in one thread. But now that we can build the core tools with GNU Make on macOS too, it uses all threads. Furthermore, since we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have to finish every line of a long rule with a backslash to keep variables and such. Generally, the pre-make software are now organized like this: first we build Lzip before anything else: it is downloaded as a simple '.tar' file that is not compressed (only ~400kb). Once Lzip is built, the pre-make phase continues with building GNU Make, Dash (a minimalist shell) and Flock. All of their tarballs are in '.tar.lz'. Maneage then enters 'basic.mk' and the first program it builds is GNU Gzip (itself packaged as '.tar.lz'). Once Gzip is built, we build all the other compression software (all downloaded as '.tar.gz'). Afterwards, any compression standard for other software is fine because we have it. In the process, a bug related to using backup servers was found in 'reproduce/analysis/bash/download-multi-try' for calling outside of 'basic.mk' and removed Bash-specific features. As a result of that bug-fix, because we now have multiple servers for software tarballs, the backup servers now have their own configuration file in 'reproduce/software/config/servers-backup.conf'. This makes it much easier to maintain the backup server list across the multiple places that we need it. Some other minor fixes: - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'. - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD (which doesn't have GCC). - We are now using 'uname -s' to specify if we are on a Linux kernel or not, if not, we are still using the old 'on_mac_os' variable. - While I was trying to build on FreeBSD, I noticed some further corrections that could help. For example the 'makelink' Make-function now takes a third argument which can be a different name compared to the actual program (used for examle to make a link to '/usr/bin/cc' from 'gcc'. - Until now we didn't know if the host's Make implementation supports placing a '@' at the start of the recipe (to avoid printing the actual commands to standard output). Especially in the tarball download phase, there are many lines that are printed for each download which was really annoying. We already used '@' in 'high-level.mk' and 'python.mk' before, but now that we also know that 'basic.mk' is called with our custom GNU Make, we can use it at the start for a cleaner stdout. - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a system where we can't install GCC (or has activated the '--host-cc' option), it may not be present and the project shouldn't break because of this. So with this commit, when a Fortran compiler isn't present, WCSLIB will be built with the '--disable-fortran' configuration option. This commit (task #15667) was completed with help/checks by Raul Infante-Sainz and Boud Roukema.
Diffstat (limited to 'reproduce/software/shell/configure.sh')
1 files changed, 49 insertions, 99 deletions
diff --git a/reproduce/software/shell/configure.sh b/reproduce/software/shell/configure.sh
index 69097c2..5cf813b 100755
--- a/reproduce/software/shell/configure.sh
+++ b/reproduce/software/shell/configure.sh
@@ -1,4 +1,4 @@
-#! /bin/sh
# Necessary preparations/configurations for the reproducible project.
@@ -319,19 +319,27 @@ static_build=no
-# If we are on a Mac OS system
-# ----------------------------
-# For the time being, we'll use the existance of `otool' to see if we are
-# on a Mac OS system or not. Some tools (for example OpenSSL) need to know
-# this.
+# See if we are on a Linux-based system
+# --------------------------------------
-# On Mac OS, the building of GCC crashes sometimes while building libiberty
-# with CLang's `g++'. Until we find a solution, we'll just use the host's C
-# compiler.
-if type otool > /dev/null 2>/dev/null; then
+# Some features are tailored to GNU/Linux systems, while the BSD-based
+# behavior is different. Initially we only tested macOS (hence the name of
+# the variable), but as FreeBSD is also being inlucded in our tests. As
+# more systems get used, we need to tailor these kinds of things better.
+kernelname=$(uname -s)
+if [ x$kernelname = xLinux ]; then
+ on_mac_os=no
+# Print warning if the host CC is to be used.
+if [ x$host_cc = x1 ]; then
cat <<EOF
@@ -347,14 +355,11 @@ ______________________________________________________
sleep 5
- on_mac_os=no
# Necessary C library element positions
# -------------------------------------
@@ -1143,98 +1148,29 @@ fi
-# Number of threads for basic builds
-# ----------------------------------
+# Number of threads to build software
+# -----------------------------------
-# Since the system might not have GNU Make at this stage, and other Make
-# implementations can't deal with parallel build properly, we'll just
-# default to 1 thread. This is because some versions of Make complain about
-# not having enough 'pipe' (memory) on some systems. After some searching,
-# I found out its because of too many threads. GNU Make will be present on
-# GNU systems (that have `nproc', part of GNU Coreutils). So to simplify
-# the test for GNU Make, we'll just try running `nproc'.
-if which nproc &> /dev/null; then
- if [ $jobs = 0 ]; then
+# If the user hasn't manually specified the number of threads, see if we
+# can deduce it from the host:
+# - On systems with GNU Coreutils we have 'nproc'.
+# - On BSD-based systems (for example FreeBSD and macOS), we have a
+# 'hw.ncpu' in the output of 'sysctl'.
+# - When none of the above work, just set the number of threads to 1.
+if [ $jobs = 0 ]; then
+ if type nproc > /dev/null 2> /dev/null; then
numthreads=$(nproc --all);
- numthreads=$jobs
+ numthreads=$(sysctl -a | awk '/^hw\.ncpu/{print $2}')
+ if [ x"$numthreads" = x ]; then numthreads=1; fi
- numthreads=1;
-# Build `flock' before other program
-# ----------------------------------
-# Flock (or file-lock) is a unique program that is necessary to serialize
-# the (generally parallel) processing of make when necessary. GNU/Linux
-# machines have it as part of their `util-linux' programs. But to be
-# consistent in non-GNU/Linux systems, we will be using our own build.
-# The reason that `flock' is sepecial is that we need it to serialize the
-# download process of the software tarballs.
-flockversion=$(awk '/flock-version/{print $3}' $depverfile)
-flockchecksum=$(awk '/flock-checksum/{print $3}' $depshafile)
-# Prepare/download the tarball.
-if ! [ -f $tardir/$flocktar ]; then
- flocktarname=$tardir/$flocktar
- ucname=$flocktarname.unchecked
- if [ -f $ddir/$flocktar ]; then
- cp $ddir/$flocktar $ucname
- else
- if ! $downloader $ucname $flockurl/$flocktar; then
- rm -f $ucname;
- echo
- echo "DOWNLOAD ERROR: Couldn't download the 'flock' tarball:"
- echo " $flockurl"
- echo
- echo "You can manually place it in '$ddir' to avoid downloading."
- exit 1
- fi
- fi
- # Make sure this is the correct tarball.
- if type sha512sum > /dev/null 2>/dev/null; then
- checksum=$(sha512sum "$ucname" | awk '{print $1}')
- if [ x$checksum = x$flockchecksum ]; then mv "$ucname" "$flocktarname"
- else echo "ERROR: Non-matching checksum for '$flocktar'."; exit 1
- fi;
- else mv "$ucname" "$flocktarname"
- fi
-# If the tarball is newer than the (possibly existing) program (the version
-# has changed), then delete the program.
-if [ -f .local/bin/flock ]; then
- if [ $tardir/$flocktar -nt $ibidir/flock ]; then
- rm $ibidir/flock
- fi
-# Build `flock' if necessary.
-if ! [ -f $ibidir/flock ]; then
- cd $tmpblddir
- tar xf $tardir/$flocktar
- cd flock-$flockversion
- ./configure --prefix=$instdir
- make
- make install
- cd $topdir
- rm -rf $tmpblddir/flock-$flockversion
- echo "Discoteq flock $flockversion" > $ibidir/flock
+ numthreads=$jobs
# Paths needed by the host compiler (only for `basic.mk')
# -------------------------------------------------------
@@ -1261,14 +1197,28 @@ fi
-# Build basic software
-# --------------------
+# Build core tools for project
+# ----------------------------
+# Here we build the core tools that 'basic.mk' depends on: Lzip
+# (compression program), GNU Make (that 'basic.mk' is written in), Dash
+# (minimal Bash-like shell) and Flock (to lock files and enable serial
+# download).
+./reproduce/software/shell/pre-make-build.sh \
+ "$bdir" "$ddir" "$downloader"
+# Build other basic tools our own GNU Make
+# ----------------------------------------
# When building these software we don't have our own un-packing software,
# Bash, Make, or AWK. In this step, we'll install such low-level basic
# tools, but we have to be very portable (and use minimal features in all).
echo; echo "Building necessary software (if necessary)..."
-make -k -f reproduce/software/make/basic.mk \
+.local/bin/make -k -f reproduce/software/make/basic.mk \
sys_library_path=$sys_library_path \
rpath_command=$rpath_command \
static_build=$static_build \