From c0a512e0631a024f18bec847125a9a03e6aa23a1 Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Fri, 22 May 2020 23:35:21 +0100 Subject: Corrected copyright notices to fit GPL suggested format In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf --- reproduce/analysis/bash/download-multi-try | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) (limited to 'reproduce/analysis/bash/download-multi-try') diff --git a/reproduce/analysis/bash/download-multi-try b/reproduce/analysis/bash/download-multi-try index 4136df4..8d10bf4 100755 --- a/reproduce/analysis/bash/download-multi-try +++ b/reproduce/analysis/bash/download-multi-try @@ -28,15 +28,18 @@ # # Copyright (C) 2019-2020 Mohammad Akhlaghi # -# This script is part of Maneage. Maneage is free software: you can -# redistribute it and/or modify it under the terms of the GNU General -# Public License as published by the Free Software Foundation, either -# version 3 of the License, or (at your option) any later version. +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. # -# Maneage is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. See . +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . -- cgit v1.2.1 From 3d8aa5953c4e0b79278ab2e27ec4e1051310d04f Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Mon, 1 Jun 2020 03:05:37 +0100 Subject: Core software build before using Make to build other software Until now, Maneage would only build Flock before building everything else using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid parallel downloads during the building of software (which could cause network problems). But after recently trying Maneage on FreeBSD (which is not yet complete, see bug #58465), we noticed that the BSD implemenation of Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq' parts) and its shell also had some peculiarities. It was thus decided to also install our own minimalist shell, Make and compressor program before calling 'basic.mk'. In this way, 'basic.mk' can now assume the same GNU Make features that high-level.mk and python.mk assume. The pre-make building of software is now organized in 'reproduce/software/shell/pre-make-build.sh'. Another nice feature of this commit is for macOS users: until now the default macOS Make had problems for parallel building of software, so 'basic.mk' was built in one thread. But now that we can build the core tools with GNU Make on macOS too, it uses all threads. Furthermore, since we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have to finish every line of a long rule with a backslash to keep variables and such. Generally, the pre-make software are now organized like this: first we build Lzip before anything else: it is downloaded as a simple '.tar' file that is not compressed (only ~400kb). Once Lzip is built, the pre-make phase continues with building GNU Make, Dash (a minimalist shell) and Flock. All of their tarballs are in '.tar.lz'. Maneage then enters 'basic.mk' and the first program it builds is GNU Gzip (itself packaged as '.tar.lz'). Once Gzip is built, we build all the other compression software (all downloaded as '.tar.gz'). Afterwards, any compression standard for other software is fine because we have it. In the process, a bug related to using backup servers was found in 'reproduce/analysis/bash/download-multi-try' for calling outside of 'basic.mk' and removed Bash-specific features. As a result of that bug-fix, because we now have multiple servers for software tarballs, the backup servers now have their own configuration file in 'reproduce/software/config/servers-backup.conf'. This makes it much easier to maintain the backup server list across the multiple places that we need it. Some other minor fixes: - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'. - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD (which doesn't have GCC). - We are now using 'uname -s' to specify if we are on a Linux kernel or not, if not, we are still using the old 'on_mac_os' variable. - While I was trying to build on FreeBSD, I noticed some further corrections that could help. For example the 'makelink' Make-function now takes a third argument which can be a different name compared to the actual program (used for examle to make a link to '/usr/bin/cc' from 'gcc'. - Until now we didn't know if the host's Make implementation supports placing a '@' at the start of the recipe (to avoid printing the actual commands to standard output). Especially in the tarball download phase, there are many lines that are printed for each download which was really annoying. We already used '@' in 'high-level.mk' and 'python.mk' before, but now that we also know that 'basic.mk' is called with our custom GNU Make, we can use it at the start for a cleaner stdout. - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a system where we can't install GCC (or has activated the '--host-cc' option), it may not be present and the project shouldn't break because of this. So with this commit, when a Fortran compiler isn't present, WCSLIB will be built with the '--disable-fortran' configuration option. This commit (task #15667) was completed with help/checks by Raul Infante-Sainz and Boud Roukema. --- reproduce/analysis/bash/download-multi-try | 57 ++++++++++++++++++------------ 1 file changed, 34 insertions(+), 23 deletions(-) (limited to 'reproduce/analysis/bash/download-multi-try') diff --git a/reproduce/analysis/bash/download-multi-try b/reproduce/analysis/bash/download-multi-try index 8d10bf4..e4ccd26 100755 --- a/reproduce/analysis/bash/download-multi-try +++ b/reproduce/analysis/bash/download-multi-try @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # # Attempt downloading multiple times before crashing whole project. From # the top project directory (for the shebang above), this script must be @@ -90,50 +90,61 @@ counter=0 maxcounter=10 while [ ! -f "$outname" ]; do - # Increment the counter. We need the `counter=' part here because - # without it the evaluation of arithmetic expression will be like and - # error and the script is set to crash on errors. - counter=$((counter+1)) + # Increment the counter. + counter=$(echo $counter | awk '{print $1+1}') # If we have passed a maximum number of trials, just exit with # a failed code. - if (( counter > maxcounter )); then - echo + reachedmax=$(echo $counter \ + | awk '{if($1>'$maxcounter') print "yes"; else print "no";}') + if [ x$reachedmax = xyes ]; then + echo "" echo "Failed $maxcounter download attempts: $outname" - echo + echo "" exit 1 fi # If this isn't the first attempt print a notice and wait a little for # the next trail. - if (( counter > 1 )); then - tstep=$((counter*5)) + if [ x$counter = x1 ]; then + just_a_place_holder=1 + else + tstep=$(echo $counter | awk '{print $1*5}') echo "Download trial $counter for '$outname' in $tstep seconds." sleep $tstep fi - # Attempt downloading the file (one-at-a-time). Note that the - # `downloader' ends with the respective option to specify the output - # name. For example "wget -O" (so `outname', that comes after it) will - # be the name of the downloaded file. + # Attempt downloading the file. Note that the `downloader' ends with + # the respective option to specify the output name. For example "wget + # -O" (so `outname', that comes after it) will be the name of the + # downloaded file. if [ x"$lockfile" = xnolock ]; then if ! $downloader $outname $inurl; then rm -f $outname; fi else # Try downloading from the requested URL. - flock "$lockfile" bash -c \ + flock "$lockfile" sh -c \ "if ! $downloader $outname $inurl; then rm -f $outname; fi" + fi + + # If the download failed, try the backup server(s). + if [ ! -f "$outname" ]; then + if [ x"$backupservers" != x ]; then + for bs in $backupservers; do - # If it failed, try the backup server(s). - if [ ! -f "$outname" ]; then - if [ x"$backupservers" != x ]; then - for bs in "$backupservers"; do - flock "$lockfile" bash -c \ + # Use this backup server. + if [ x"$lockfile" = xnolock ]; then + if ! $downloader $outname $bs/$urlfile; then rm -f $outname; fi + else + flock "$lockfile" sh -c \ "if ! $downloader $outname $bs/$urlfile; then rm -f $outname; fi" - done - fi + fi + + # If the file was downloaded, break out of the loop that + # parses over the backup servers. + if [ -f "$outname" ]; then break; fi + done fi fi - done -- cgit v1.2.1