From 2264c0fea5668cd8f827cdbd89004fd390b0b14f Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Thu, 23 Jan 2020 18:54:29 +0000 Subject: IMPORTANT: Project preparation is now also done with project make Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system. --- README-hacking.md | 33 ++++++--------- README.md | 16 +------ project | 82 ++++++++++-------------------------- reproduce/software/bash/configure.sh | 4 +- 4 files changed, 37 insertions(+), 98 deletions(-) diff --git a/README-hacking.md b/README-hacking.md index 83c341b..324ada4 100644 --- a/README-hacking.md +++ b/README-hacking.md @@ -279,28 +279,20 @@ example if you run `.local/bin/ls` you will be using the `ls` of the template, which is problably different from your system's `ls` (run them both with `--version` to check). -Once the project is configured for your system, `./project prepare` and -`./project make` will do the basic preparations and run the project's -analysis with the custom version of software. The `project` script is just -a wrapper, and with the commands above, it will call `top-prepare.mk` and -`top-make.mk` (both are in the `reproduce/analysis/make` directory). - -In the template, no particular preparation is necessary, so it will -immediately finish and instruct you to run `./project make`. But in some -projects, it can be very useful to do some very basic preparatory steps on -the input data that can greatly optimize running of `./project make`. For -example, you may need to query a server, to find how many input files there -are. Once that number is known in the preparation phase, `./project make` -can parallelize the analysis much more effectively. +Once the project is configured for your system, `./project make` will do +the basic preparations and run the project's analysis with the custom +version of software. The `project` script is just a wrapper, and with the +`make` argument, it will first call `top-prepare.mk` and `top-make.mk` +(both are in the `reproduce/analysis/make` directory). In terms of organization, `top-prepare.mk` and `top-make.mk` have an -identical design, only a minor difference. So, let's continue the -template's architecture with `top-make.mk`. Once you understand that, -you'll clearly understand `top-prepare.mk` also. These very high-level -files are relatively short and heavily commented so hopefully the -descriptions in each comment will be enough to understand the general -details. As you read this section, please also look at the contents of the -mentioned files and directories to fully understand what is going on. +identical design, only minor differences. So, let's continue the template's +architecture with `top-make.mk`. Once you understand that, you'll clearly +understand `top-prepare.mk` also. These very high-level files are +relatively short and heavily commented so hopefully the descriptions in +each comment will be enough to understand the general details. As you read +this section, please also look at the contents of the mentioned files and +directories to fully understand what is going on. Before starting to look into the top `top-make.mk`, it is important to recall that Make defines dependencies by files. Therefore, the @@ -587,7 +579,6 @@ First custom commit ```shell $ ./project configure # Build the project's software environment (can take an hour or so). - $ ./project prepare # Pre-processing preparations (doing nothing in the raw template). $ ./project make # Do the processing and build paper (just a simple demo in the template). # Open 'paper.pdf' and see if everything is ok. diff --git a/README.md b/README.md index 3796d2f..f89f89a 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,6 @@ received this source from arXiv, please see the respective section below. $ git clone XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX $ cd XXXXXXXXXXXXXXXXXX $ ./project configure -$ ./project prepare $ ./project make ``` @@ -77,20 +76,7 @@ requiring root/administrator permissions. $ ./project configure ``` -3. In some cases, the project's analysis may need some preparations to - optimize its processing. This is usually mainly related to input data, - and some very basic calculations that can help the management of the - overall lproject in the main/next step. To do the basic preparations, - please run this command to do the preparation on `8` threads. If your - CPU has a different number of threads, change the number (you can see - the number of threads available to your operating system by running - `./.local/bin/nproc`) - - ```shell - $ ./project prepare -j8 - ``` - -4. Run the following command to reproduce all the analysis and build the +3. Run the following command to reproduce all the analysis and build the final `paper.pdf` on `8` threads. If your CPU has a different number of threads, change the number (you can see the number of threads available to your operating system by running `./.local/bin/nproc`) diff --git a/project b/project index 5d161be..d994b09 100755 --- a/project +++ b/project @@ -66,14 +66,12 @@ print_help() { # Print the output. cat < $finaltarget # The configuration is now complete, we can inform the user on the next # step(s) to take. if [ x$reproducible_paper_group_name = x ]; then - buildcommand="./project prepare -j8" + buildcommand="./project make -j8" else - buildcommand="./project prepare --group=$reproducible_paper_group_name -j8" + buildcommand="./project make --group=$reproducible_paper_group_name -j8" fi cat <