diff options
| -rw-r--r-- | README | 15 | ||||
| -rw-r--r-- | README.md | 63 | ||||
| -rw-r--r-- | reproduce/src/make/dependencies-basic.mk | 54 |
3 files changed, 73 insertions, 59 deletions
@@ -27,14 +27,15 @@ Running the pipeline 2. Configure the environment (top-level directories in particular) and build all the necessary software for use in the next step. It is - recommended to modify the directories. Please read the comments (lines - starting with an `#') in the file that is opened by `./configure' and - set the directories accordingly. + recommended to modify the directories. Please read the description of + each necessary input clearly and set the best value. Note that the + configure scripts also downloads, builds and locally installs many + programs (pipeline dependencies). So it may take a while to complete. $ ./configure -3. Run the following command to reproduce all the analysis and build the - final `paper.pdf' on 8 threads. If your CPU has a different number of - threads, change the number. +3. Run the following command (local build of Make) to reproduce all the + analysis and build the final `paper.pdf' on *8* threads. If your CPU has + a different number of threads, change the number. - $ .local/bin/make -j8 + $ ./.local/bin/make -j8 @@ -2,7 +2,8 @@ Introduction ============ This description is for *creators* of the reproduction pipeline. See -`README` for instructions on running it. +`README` for instructions on running it (in short, just download/clone it, +then run `./configure` and `./.local/bin/make -j8`). This project contains a **fully working template** for a high-level research reproduction pipeline, or reproducible paper, as defined in the @@ -238,7 +239,8 @@ In order to adopt this pipeline to your research, it is important to first understand its architecture so you can navigate your way in the directories and understand how to implement your research project within its framework. But before reading this theoretical discussion, please run the -pipeline without any change, just to see how it works. +pipeline (described in `README`: first run `./configure`, then +`./.local/bin/make -j8`) without any change, just to see how it works. In order to obtain a reproducible result it is important to have an identical environment (for example same versions the programs that it will @@ -246,17 +248,25 @@ use). This also has the added advantage that in your separate research projects, you can use different versions of a single software and they won't interfere. Therefore, the pipeline builds its own dependencies during the `./configure` step. Building of the dependencies is managed by -`reproduce/src/make/dependencies.mk`. So later, if you add a new -program/library for your processing, don't forget to include a rule on how -to build it, in this file. - -When you run `.local/bin/make` to start the processing, the first file that -is read is the top-level `Makefile`. Therefore, we'll start our -navigation/discussion with this file. This file is relatively short and -heavily commented so hopefully the descriptions in each comment will be -enough to understand the general details. As you read this section, please -also look at the contents of the mentioned files and directories to fully -understand what is being said. +`reproduce/src/make/dependencies-basic.mk` and +`reproduce/src/make/dependencies.mk`. These Makefiles are called by the +`./configure` script. The first is intended for downloading and building +the most basic tools like GNU Bash, GNU Make, and GNU Tar. Therefore it +must only contain very basic and portable Make and shell features. The +second is called after the first, thus enabling usage of the modern and +advanced features of GNU Bash and GNU Make, similar to the rest of the +pipeline. Later, if you add a new program/library for your research, you +will need to include a rule on how to download and build it (in +`reproduce/src/make/dependencies.mk`). + +After configuring, the `.local/bin/make` command will start the processing +with the custom version of Make that was locally installed during +configuration. The first file that is read is the top-level +`Makefile`. Therefore, we'll start our navigation/discussion with this +file. This file is relatively short and heavily commented so hopefully the +descriptions in each comment will be enough to understand the general +details. As you read this section, please also look at the contents of the +mentioned files and directories to fully understand what is going on. Before starting to look into the top `Makefile`, it is important to recall that Make defines dependencies by files. Therefore, the input and output of @@ -283,18 +293,21 @@ completely (to define dependencies and etc) and starts its execution after that. So it is fine to define the rule to build `paper.pdf` at a later stage (this is one beauty of Make!). -Having defined the top target, we will include all the necessary -Makefiles. First we include all `reproduce/config/pipeline/*.mk`. The -configuration of each logical step of the pipeline is placed here as a -separate file. These Makefiles must only contain raw Make variables -(pipeline configurations). By raw we mean that the Make variables in these -files must not depend on any other variables because we don't want to -assume any order in reading them. It is very important to *not* define any -rule or other Make construct in any of these _configuration-Makefiles_ (see -the next paragraph for Makefiles with rules). This will enable you to set -the respective files in this directory as a prerequisite to any target that -depends on their variable values. Therefore, if you change any of the -values, all targets that depend on those values will be re-built. +Having defined the top target, our next step is to include all the other +necessary Makefiles. First we include all Makefiles that satisfy this +wildcard: `reproduce/config/pipeline/*.mk`. These Makefiles don't actually +have any rules, they just have values for various free parameters +throughout the pipeline. Open a few of them to see for your self. These +Makefiles must only contain raw Make variables (pipeline +configurations). By raw we mean that the Make variables in these files must +not depend on variables in any other Makefile. This is because we don't +want to assume any order in reading them. It is very important to *not* +define any rule or other Make construct in any of these +_configuration-Makefiles_ (see the next paragraph for Makefiles with +rules). This will enable you to set the respective Makefiles in this +directory as a prerequisite to any target that depends on their variable +values. Therefore, if you change any of their values, all targets that +depend on those values will be re-built. Once all the raw variables have been imported into the top Makefile, we are ready to import the Makefiles containing the details of the processing diff --git a/reproduce/src/make/dependencies-basic.mk b/reproduce/src/make/dependencies-basic.mk index dcc0b59..0372d7d 100644 --- a/reproduce/src/make/dependencies-basic.mk +++ b/reproduce/src/make/dependencies-basic.mk @@ -88,7 +88,7 @@ all: $(foreach p, $(top-level-programs), $(ibdir)/$(p)) tarballs = $(foreach t, bash-$(bash-version).tar.gz \ binutils-$(binutils-version).tar.lz \ bzip2-$(bzip2-version).tar.gz \ - gzip-$(gzip-version).tar.lz \ + gzip-$(gzip-version).tar.gz \ lzip-$(lzip-version).tar.gz \ make-$(make-version).tar.lz \ tar-$(tar-version).tar.gz \ @@ -133,10 +133,17 @@ $(tarballs): $(tdir)/%: +# GNU Gzip. +$(ibdir)/gzip: $(tdir)/gzip-$(gzip-version).tar.gz + $(call gbuild,$(subst $(tdir)/,,$<), gzip-$(gzip-version), static) + + + + + # GNU Lzip: For a static build, the `-static' flag should be given to # LDFLAGS on the command-line (not from the environment). $(ibdir)/lzip: $(tdir)/lzip-$(lzip-version).tar.gz - echo; echo $(static_build); echo; ifeq ($(static_build),yes) $(call gbuild,$(subst $(tdir)/,,$<), lzip-$(lzip-version), , \ LDFLAGS="-static") @@ -148,15 +155,6 @@ endif -# GNU Gzip. -$(ibdir)/gzip: $(tdir)/gzip-$(gzip-version).tar.lz \ - $(ibdir)/lzip - $(call gbuild,$(subst $(tdir)/,,$<), gzip-$(gzip-version), static) - - - - - # Zlib: its `./configure' doesn't use Autoconf's configure script, it just # accepts a direct `--static' option. $(ildir)/libz.a: $(tdir)/zlib-$(zlib-version).tar.gz @@ -195,33 +193,34 @@ $(ibdir)/bzip2: $(tdir)/bzip2-$(bzip2-version).tar.gz -# GNU Binutils: -$(ibdir)/nm: $(tdir)/binutils-$(binutils-version).tar.lz \ - $(ibdir)/lzip \ - $(ildir)/libz.a - $(call gbuild,$(subst $(tdir)/,,$<), binutils-$(binutils-version), \ - static) - - - - - # GNU Tar: When built statically, tar gives a segmentation fault on # unpacking Bash. So we'll build it dynamically. $(ibdir)/tar: $(tdir)/tar-$(tar-version).tar.gz \ $(ibdir)/bzip2 \ - $(ibdir)/gzip \ $(ibdir)/lzip \ - $(ibdir)/xz \ - $(ibdir)/nm + $(ibdir)/gzip \ + $(ibdir)/xz $(call gbuild,$(subst $(tdir)/,,$<), tar-$(tar-version)) +# GNU Binutils: +$(ibdir)/nm: $(tdir)/binutils-$(binutils-version).tar.lz \ + $(ibdir)/tar \ + $(ildir)/libz.a + $(call gbuild,$(subst $(tdir)/,,$<), binutils-$(binutils-version), \ + static) + + + + + # GNU Which: -$(ibdir)/which: $(tdir)/which-$(which-version).tar.gz +$(ibdir)/which: $(tdir)/which-$(which-version).tar.gz \ + $(ibdir)/tar \ + $(ibdir)/nm $(call gbuild,$(subst $(tdir)/,,$<), which-$(which-version), static) @@ -234,7 +233,8 @@ $(ibdir)/which: $(tdir)/which-$(which-version).tar.gz # `--disable-load', but unfortunately I don't know any way to fix the # second. So, we'll have to build it dynamically for now. $(ibdir)/make: $(tdir)/make-$(make-version).tar.lz \ - $(ibdir)/tar + $(ibdir)/tar \ + $(ibdir)/nm $(call gbuild,$(subst $(tdir)/,,$<), make-$(make-version)) |