aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorMohammad Akhlaghi <mohammad@akhlaghi.org>2020-01-23 18:54:29 +0000
committerMohammad Akhlaghi <mohammad@akhlaghi.org>2020-01-23 19:16:16 +0000
commit2264c0fea5668cd8f827cdbd89004fd390b0b14f (patch)
tree99af64b55557bfb69ea630a653bb8bf07586c63a
parentcf0427009decbcad33a269898a428341e9c0ceff (diff)
IMPORTANT: Project preparation is now also done with project make
Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system.
-rw-r--r--README-hacking.md33
-rw-r--r--README.md16
-rwxr-xr-xproject82
-rwxr-xr-xreproduce/software/bash/configure.sh4
4 files changed, 37 insertions, 98 deletions
diff --git a/README-hacking.md b/README-hacking.md
index 83c341b..324ada4 100644
--- a/README-hacking.md
+++ b/README-hacking.md
@@ -279,28 +279,20 @@ example if you run `.local/bin/ls` you will be using the `ls` of the
template, which is problably different from your system's `ls` (run them
both with `--version` to check).
-Once the project is configured for your system, `./project prepare` and
-`./project make` will do the basic preparations and run the project's
-analysis with the custom version of software. The `project` script is just
-a wrapper, and with the commands above, it will call `top-prepare.mk` and
-`top-make.mk` (both are in the `reproduce/analysis/make` directory).
-
-In the template, no particular preparation is necessary, so it will
-immediately finish and instruct you to run `./project make`. But in some
-projects, it can be very useful to do some very basic preparatory steps on
-the input data that can greatly optimize running of `./project make`. For
-example, you may need to query a server, to find how many input files there
-are. Once that number is known in the preparation phase, `./project make`
-can parallelize the analysis much more effectively.
+Once the project is configured for your system, `./project make` will do
+the basic preparations and run the project's analysis with the custom
+version of software. The `project` script is just a wrapper, and with the
+`make` argument, it will first call `top-prepare.mk` and `top-make.mk`
+(both are in the `reproduce/analysis/make` directory).
In terms of organization, `top-prepare.mk` and `top-make.mk` have an
-identical design, only a minor difference. So, let's continue the
-template's architecture with `top-make.mk`. Once you understand that,
-you'll clearly understand `top-prepare.mk` also. These very high-level
-files are relatively short and heavily commented so hopefully the
-descriptions in each comment will be enough to understand the general
-details. As you read this section, please also look at the contents of the
-mentioned files and directories to fully understand what is going on.
+identical design, only minor differences. So, let's continue the template's
+architecture with `top-make.mk`. Once you understand that, you'll clearly
+understand `top-prepare.mk` also. These very high-level files are
+relatively short and heavily commented so hopefully the descriptions in
+each comment will be enough to understand the general details. As you read
+this section, please also look at the contents of the mentioned files and
+directories to fully understand what is going on.
Before starting to look into the top `top-make.mk`, it is important to
recall that Make defines dependencies by files. Therefore, the
@@ -587,7 +579,6 @@ First custom commit
```shell
$ ./project configure # Build the project's software environment (can take an hour or so).
- $ ./project prepare # Pre-processing preparations (doing nothing in the raw template).
$ ./project make # Do the processing and build paper (just a simple demo in the template).
# Open 'paper.pdf' and see if everything is ok.
diff --git a/README.md b/README.md
index 3796d2f..f89f89a 100644
--- a/README.md
+++ b/README.md
@@ -21,7 +21,6 @@ received this source from arXiv, please see the respective section below.
$ git clone XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$ cd XXXXXXXXXXXXXXXXXX
$ ./project configure
-$ ./project prepare
$ ./project make
```
@@ -77,20 +76,7 @@ requiring root/administrator permissions.
$ ./project configure
```
-3. In some cases, the project's analysis may need some preparations to
- optimize its processing. This is usually mainly related to input data,
- and some very basic calculations that can help the management of the
- overall lproject in the main/next step. To do the basic preparations,
- please run this command to do the preparation on `8` threads. If your
- CPU has a different number of threads, change the number (you can see
- the number of threads available to your operating system by running
- `./.local/bin/nproc`)
-
- ```shell
- $ ./project prepare -j8
- ```
-
-4. Run the following command to reproduce all the analysis and build the
+3. Run the following command to reproduce all the analysis and build the
final `paper.pdf` on `8` threads. If your CPU has a different number of
threads, change the number (you can see the number of threads available
to your operating system by running `./.local/bin/nproc`)
diff --git a/project b/project
index 5d161be..d994b09 100755
--- a/project
+++ b/project
@@ -66,14 +66,12 @@ print_help() {
# Print the output.
cat <<EOF
Usage: $scriptname configure [OPTIONS]
- $scriptname prepare [OPTIONS]
$scriptname make [OPTIONS]
Top-level script to manage the reproducible project. The high-level
operation is defined by the (mandatory) second argument:
configure - Configure project for this machine (e.g., build software).
- prepare - Low-level preparations to optimize building with 'make'.
make - Run the project (do analysis and build outputs).
RECOMMENDATION: If this is the first time you are configuring this
@@ -151,7 +149,6 @@ do
case $1 in
# Main operation.
configure) func_operation_set $1; shift;;
- prepare) func_operation_set $1; shift;;
make) func_operation_set $1; shift;;
@@ -294,44 +291,6 @@ controlled_env() {
-# Error messages
-# --------------
-#
-# Having the error messages here helps the over-all process be more
-# readable.
-print_error_abort() {
- case $1 in
- prepare)
- cat <<EOF
-
-The project isn't configured for this system, or the configuration wasn't
-successful. To configure the project, please use this command:
-
- $ ./project configure
-
-(TIP: if you have already ran this command once, run it with '-e' to use
-the previous configuration, run with '--help' for more info)
-
-EOF
- exit 1;
- ;;
- make)
- cat <<EOF
-
-The project preparation hasn't been completed, or it wasn't successful. To
-prepare the project prior to building it, please use this command:
-
- $ ./project prepare
-
-EOF
- exit 1;
- ;;
- esac
-}
-
-
-
-
# Do requested operation
# ----------------------
perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x"
@@ -398,32 +357,30 @@ case $operation in
- # Run the input management.
- prepare)
+ # Run the project.
+ make)
# Make sure the configure script has been completed properly
# (`configuration-done.txt' exists).
if ! [ -f .build/software/configuration-done.txt ]; then
- print_error_abort $operation
- fi
-
- # Run input-preparations in control environment
- controlled_env reproduce/analysis/make/top-prepare.mk
- ;;
-
-
+ cat <<EOF
+The project isn't configured for this system, or the configuration wasn't
+successful. To configure the project, please use this command:
+ $ ./project configure
- # Run the project
- make)
+[[ TIP: If you have already configured it once, run it with '-e' to use the
+previous configuration. Run with '--help' for more info ]]
- # Make sure the configure script has been completed properly
- # (`configuration-done.txt' exists).
- if ! [ -f .build/software/preparation-done.mk ]; then
- print_error_abort $operation
+EOF
+ exit 1
fi
+ # Run input-preparations (optionally build Makefiles with special
+ # values for optimizing the main `top-make.mk').
+ controlled_env reproduce/analysis/make/top-prepare.mk
+
# Run the actual project.
controlled_env reproduce/analysis/make/top-make.mk
;;
@@ -434,9 +391,14 @@ case $operation in
# Operation not specified.
*)
- echo "No operation defined."
- echo "Please run with '--help' for more information."
- echo "Available operations are: 'configure', 'prepare', or 'make')."
+ cat <<EOF
+
+No operation defined!
+
+Please run with '--help' for more information.
+(TIP: available operations are: 'configure' or 'make').
+
+EOF
exit 1
;;
esac
diff --git a/reproduce/software/bash/configure.sh b/reproduce/software/bash/configure.sh
index 60f69b9..17e3048 100755
--- a/reproduce/software/bash/configure.sh
+++ b/reproduce/software/bash/configure.sh
@@ -1387,9 +1387,9 @@ echo `.local/bin/date` > $finaltarget
# The configuration is now complete, we can inform the user on the next
# step(s) to take.
if [ x$reproducible_paper_group_name = x ]; then
- buildcommand="./project prepare -j8"
+ buildcommand="./project make -j8"
else
- buildcommand="./project prepare --group=$reproducible_paper_group_name -j8"
+ buildcommand="./project make --group=$reproducible_paper_group_name -j8"
fi
cat <<EOF