diff options
author | Mohammad Akhlaghi <mohammad@akhlaghi.org> | 2020-01-23 18:54:29 +0000 |
---|---|---|
committer | Mohammad Akhlaghi <mohammad@akhlaghi.org> | 2020-01-23 19:16:16 +0000 |
commit | 2264c0fea5668cd8f827cdbd89004fd390b0b14f (patch) | |
tree | 99af64b55557bfb69ea630a653bb8bf07586c63a | |
parent | cf0427009decbcad33a269898a428341e9c0ceff (diff) |
IMPORTANT: Project preparation is now also done with project make
Until now, the main commands to run the project were these: `./project
configure' (to build the software), `./project prepare' (to possibly
arrange input datasets and build special configuration Makefiles) and
finally `./project make' to run the project.
The main logic behind the "prepare" phase `top-prepare.mk' is to build
configuration files that can be fed into the "make" step and optimize its
operation. For example when the total number of necessary inputs for the
majority of the analysis is not as large as the total number of
inputs. With "prepare" (when necessary), you go through the raw inputs,
select the ones that are necessary for the rest of the project. The output
of `top-prepare.mk' is a configuration file (a Make variable) that keeps
the IDs (numbers, names, etc). That configuration file would then be used
in the `top-make.mk' to identify the lower level targets and allow optimal
project organization and management.
But the last two are both part of the analysis, and while they indeed need
different calls to Make to be executed, many projects don't actually need a
preparation phase: ultimately, its an implementation choice by the project
developers and doesn't concern the project users (or the developers when
they are running it).
To avoid confusing the users, or simply annoying them when a projet doesn't
need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk'
Makefiles are called with the single `./project make' command and
`./project prepare' has been dropped. I noticed this while writing the
paper on this system.
-rw-r--r-- | README-hacking.md | 33 | ||||
-rw-r--r-- | README.md | 16 | ||||
-rwxr-xr-x | project | 82 | ||||
-rwxr-xr-x | reproduce/software/bash/configure.sh | 4 |
4 files changed, 37 insertions, 98 deletions
diff --git a/README-hacking.md b/README-hacking.md index 83c341b..324ada4 100644 --- a/README-hacking.md +++ b/README-hacking.md @@ -279,28 +279,20 @@ example if you run `.local/bin/ls` you will be using the `ls` of the template, which is problably different from your system's `ls` (run them both with `--version` to check). -Once the project is configured for your system, `./project prepare` and -`./project make` will do the basic preparations and run the project's -analysis with the custom version of software. The `project` script is just -a wrapper, and with the commands above, it will call `top-prepare.mk` and -`top-make.mk` (both are in the `reproduce/analysis/make` directory). - -In the template, no particular preparation is necessary, so it will -immediately finish and instruct you to run `./project make`. But in some -projects, it can be very useful to do some very basic preparatory steps on -the input data that can greatly optimize running of `./project make`. For -example, you may need to query a server, to find how many input files there -are. Once that number is known in the preparation phase, `./project make` -can parallelize the analysis much more effectively. +Once the project is configured for your system, `./project make` will do +the basic preparations and run the project's analysis with the custom +version of software. The `project` script is just a wrapper, and with the +`make` argument, it will first call `top-prepare.mk` and `top-make.mk` +(both are in the `reproduce/analysis/make` directory). In terms of organization, `top-prepare.mk` and `top-make.mk` have an -identical design, only a minor difference. So, let's continue the -template's architecture with `top-make.mk`. Once you understand that, -you'll clearly understand `top-prepare.mk` also. These very high-level -files are relatively short and heavily commented so hopefully the -descriptions in each comment will be enough to understand the general -details. As you read this section, please also look at the contents of the -mentioned files and directories to fully understand what is going on. +identical design, only minor differences. So, let's continue the template's +architecture with `top-make.mk`. Once you understand that, you'll clearly +understand `top-prepare.mk` also. These very high-level files are +relatively short and heavily commented so hopefully the descriptions in +each comment will be enough to understand the general details. As you read +this section, please also look at the contents of the mentioned files and +directories to fully understand what is going on. Before starting to look into the top `top-make.mk`, it is important to recall that Make defines dependencies by files. Therefore, the @@ -587,7 +579,6 @@ First custom commit ```shell $ ./project configure # Build the project's software environment (can take an hour or so). - $ ./project prepare # Pre-processing preparations (doing nothing in the raw template). $ ./project make # Do the processing and build paper (just a simple demo in the template). # Open 'paper.pdf' and see if everything is ok. @@ -21,7 +21,6 @@ received this source from arXiv, please see the respective section below. $ git clone XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX $ cd XXXXXXXXXXXXXXXXXX $ ./project configure -$ ./project prepare $ ./project make ``` @@ -77,20 +76,7 @@ requiring root/administrator permissions. $ ./project configure ``` -3. In some cases, the project's analysis may need some preparations to - optimize its processing. This is usually mainly related to input data, - and some very basic calculations that can help the management of the - overall lproject in the main/next step. To do the basic preparations, - please run this command to do the preparation on `8` threads. If your - CPU has a different number of threads, change the number (you can see - the number of threads available to your operating system by running - `./.local/bin/nproc`) - - ```shell - $ ./project prepare -j8 - ``` - -4. Run the following command to reproduce all the analysis and build the +3. Run the following command to reproduce all the analysis and build the final `paper.pdf` on `8` threads. If your CPU has a different number of threads, change the number (you can see the number of threads available to your operating system by running `./.local/bin/nproc`) @@ -66,14 +66,12 @@ print_help() { # Print the output. cat <<EOF Usage: $scriptname configure [OPTIONS] - $scriptname prepare [OPTIONS] $scriptname make [OPTIONS] Top-level script to manage the reproducible project. The high-level operation is defined by the (mandatory) second argument: configure - Configure project for this machine (e.g., build software). - prepare - Low-level preparations to optimize building with 'make'. make - Run the project (do analysis and build outputs). RECOMMENDATION: If this is the first time you are configuring this @@ -151,7 +149,6 @@ do case $1 in # Main operation. configure) func_operation_set $1; shift;; - prepare) func_operation_set $1; shift;; make) func_operation_set $1; shift;; @@ -294,44 +291,6 @@ controlled_env() { -# Error messages -# -------------- -# -# Having the error messages here helps the over-all process be more -# readable. -print_error_abort() { - case $1 in - prepare) - cat <<EOF - -The project isn't configured for this system, or the configuration wasn't -successful. To configure the project, please use this command: - - $ ./project configure - -(TIP: if you have already ran this command once, run it with '-e' to use -the previous configuration, run with '--help' for more info) - -EOF - exit 1; - ;; - make) - cat <<EOF - -The project preparation hasn't been completed, or it wasn't successful. To -prepare the project prior to building it, please use this command: - - $ ./project prepare - -EOF - exit 1; - ;; - esac -} - - - - # Do requested operation # ---------------------- perms="u+r,u+w,g+r,g+w,o-r,o-w,o-x" @@ -398,32 +357,30 @@ case $operation in - # Run the input management. - prepare) + # Run the project. + make) # Make sure the configure script has been completed properly # (`configuration-done.txt' exists). if ! [ -f .build/software/configuration-done.txt ]; then - print_error_abort $operation - fi - - # Run input-preparations in control environment - controlled_env reproduce/analysis/make/top-prepare.mk - ;; - - + cat <<EOF +The project isn't configured for this system, or the configuration wasn't +successful. To configure the project, please use this command: + $ ./project configure - # Run the project - make) +[[ TIP: If you have already configured it once, run it with '-e' to use the +previous configuration. Run with '--help' for more info ]] - # Make sure the configure script has been completed properly - # (`configuration-done.txt' exists). - if ! [ -f .build/software/preparation-done.mk ]; then - print_error_abort $operation +EOF + exit 1 fi + # Run input-preparations (optionally build Makefiles with special + # values for optimizing the main `top-make.mk'). + controlled_env reproduce/analysis/make/top-prepare.mk + # Run the actual project. controlled_env reproduce/analysis/make/top-make.mk ;; @@ -434,9 +391,14 @@ case $operation in # Operation not specified. *) - echo "No operation defined." - echo "Please run with '--help' for more information." - echo "Available operations are: 'configure', 'prepare', or 'make')." + cat <<EOF + +No operation defined! + +Please run with '--help' for more information. +(TIP: available operations are: 'configure' or 'make'). + +EOF exit 1 ;; esac diff --git a/reproduce/software/bash/configure.sh b/reproduce/software/bash/configure.sh index 60f69b9..17e3048 100755 --- a/reproduce/software/bash/configure.sh +++ b/reproduce/software/bash/configure.sh @@ -1387,9 +1387,9 @@ echo `.local/bin/date` > $finaltarget # The configuration is now complete, we can inform the user on the next # step(s) to take. if [ x$reproducible_paper_group_name = x ]; then - buildcommand="./project prepare -j8" + buildcommand="./project make -j8" else - buildcommand="./project prepare --group=$reproducible_paper_group_name -j8" + buildcommand="./project make --group=$reproducible_paper_group_name -j8" fi cat <<EOF |