diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 400 |
1 files changed, 400 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..1a52571 --- /dev/null +++ b/README.md @@ -0,0 +1,400 @@ +Introduction +============ + +This description is for *creators* of the reproduction pipeline. See +`README` on instructions for running it. + +This project contains a **fully working template** for a high-level +research reproduction pipeline as defined in the link below. If this page +is inaccessible at the time of reading, please see the end of this file +which contains a portion of the introduction in this webpage. + + http://akhlaghi.org/reproducible-science.html + +This template is created with the aim of supporting reproducible research +by making it easy to start a project in this framework. It is very easy to +customize this template pipeline for any particular research/job and expand +it as it starts and evolves. It can be run with no modification (as +described in `README`) as a demonstration and customized by editing the +existing rules and adding new rules as well as adding new Makefiles as the +research/project grows. + +This file will continue with a discussion of why Make is the perfect +language/framework for a research reproduction pipeline and how to master +Make easily. Afterwards, a checklist of actions that are necessary to +customize this pipeline for your research is provided. The main body of +this text will finish with some tips and guidelines on how to manage or +extend it as your research grows. Please share your thoughts and +suggestions on this pipeline so we can implement them and make it even more +easier to use and more robust. + + +Why Make? +--------- + +When batch processing is necessary (no manual intervention, as in a +reproduction pipeline), shell scripts are usually the first solution that +comes to mind. However, the problem with scripts for a scientific +reproduction pipeline is the complexity and non-linearity. A script will +start from the top/start every time it is run. So if you have gone through +90% of a research project and want to run the remaining 10% that you have +newly added, you have to run the whole script from the start again and wait +until you see the effects of the last few steps (for the possible errors, +or better solutions and etc). It is possible to manually ignore/comment +parts of a script to only do a special part, but such checks/comments will +only add to the complexity of the script and they is prone to very serious +bugs in the end (when trying to reproduce from scratch). Such bugs are very +hard to notice when adding the checks or commenting-code in a script. + +The Make paradigm, on the other hand, starts from the end: the final +target. It builds a dependency tree internally, and finds where it should +actually start each time it is run. Therefore, in the scenario above, a +researcher that has just added the final 10% of steps of her research to +her Makefile, will only have run those extra steps. As commonly happens in +a research context, due to its paradigm Make, it is also trivial to change +the processing of any intermediate (already written) rule/step in the +middle of an already written pipeline: the next time Make is run, only +rules affected by the changes/additions will be re-run. + +This greatly speeds up the processing (enabling creative changes), while +keeping all the dependencies clearly documented (as part of the Make +language), and most importantly, enabling full reproducibility from scratch +with no changes in the pipeline code that was working during the +research. Since the dependencies are also clearly demarcated, Make can +identify independent steps and run them in parallel (further speeding up +the process). Make was designed for this purpose and it is how huge +projects like all Unix-like operating systems (including GNU/Linux or Mac +OS operating systems) and their core components are built. Therefore, Make +is a highly mature paradigm/system with robust and highly efficient +implementations in various operating systems perfectly suited for a complex +non-linear research project. + +Make is a small language with the aim of defining "rules" containing +"targets", "prerequisites" and "recipes". It comes with some cool features +like functions or automatic-variables to greatly facilitate the management +of any of those constructs. For a more detailed (yet still general) +introduction see Wikipedia: + + https://en.wikipedia.org/wiki/Make_(software) + +Many implementations of Make exist and all should be usable with this +pipeline. This pipeline has been created and tested mainly with GNU Make +which is the most common implementation. But if you see parts specific to +GNU Make, please inform us to correct it. + + +How can I learn Make? +--------------------- + +The best place to learn Make from scratch is the GNU Make manual. It is an +excellent and non-technical (in its first chapters) book to help get +started. It is freely available and always up to date with the current +release. It also clearly explains which features are specific to GNU Make +and which are general in all implementations. So the first few chapters +regarding the generalities are useful for all implementations. + +The first link below has links to the GNU Make manual in various formats +and in the second, you can get it in PDF (which may be easier to read in +the first time). + + https://www.gnu.org/software/make/manual/ + + https://www.gnu.org/software/make/manual/make.pdf + +If you use GNU Make, you also have the whole GNU Make manual on the +command-line with the following command (you can come out of the "info" +environment by pressing `q`, if you don't know "Info", we strongly +recommend running "$ info info" to learn it easily, it greatly simplifies +your access to many manuals that are installed on your system). + +```shell + $ info make +``` + +If you use the Emacs text editor, you will find the Info version of the +Make manual there also. + + + + +Checklist to customize the pipeline +=================================== + +Take the following steps to fully customize this pipeline for your research +project. After finishing the list, be sure to run `./configure` and `make` +to see if everything works correctly before expanding it. If you notice +anything missing or any in-correct part (probably a change that has not +been explained here), please let us know to correct it. + + - **Get this repository** (if you don't already have it): Arguably the + easiest way to start is to clone this repository as shown below: + + ```shell + $ git clone https://gitlab.com/makhlaghi/reproduction-pipeline-template.git + ``` + + - **Copyright**, **name** and **date**: Go over the following files and + correct the copyright, names and dates in their first few lines: + `README`, `Makefile` and `reproduce/src/make/*.mk`. When making new + files, always remember to add a similar copyright statement at the top + of the tile. + + - **Gnuastro**: GNU Astronomy Utilities (Gnuastro) is currently a + dependency of the pipeline and without it, the pipeline will complain + and abort. The main reason for this is to demonstrate how critically + important it is to version your software. If you don't want to install + Gnuastro please follow the instructions in the list below. If you do + have Gnuastro (or have installed it to check this pipeline), then + after an initial check, try un-commenting the `onlyversion` line and + running the pipeline to see the respective error. Such features in a + software makes it easy to design a robust pipeline like this. If you + have tried it and don't need Gnuastro in your pipeline, also follow + this list: + + - Delete the description about Gnuastro in `README`. + - Delete everything about Gnuastro in `reproduce/src/make/initialize.mk` + - Delete `and Gnuastro \gnuastrover` from `tex/preamble-style` + + - **Initiate a new Git repo**: You don't want to mix the history of this + template reproduction pipeline with your own reproduction + pipeline. You have already made some small changes in the previous + step, so let's re-initiate history before continuing. But before doing + that, keep the output of `git describe` in a place and write it in + your first commit message to document what point in this pipeline's + history you started from. Since the pipeline is highly integrated with + your particular research, it may not be easy to merge the changes + later. Having the commit in this history that you started from, will + allow you to check and manually apply any changes that don't interfere + with your implemented pipeline. After this step, you can commit your + changes into your newly initiated history as you like. + + ```shell + $ git describe # The point in this history you started from. + $ git clean -fxd # Remove any possibly created extra files. + $ rm -rf .git # Completely remove this history. + $ git init # Initiate a new history. + $ git add --all # Stage everything that is here. + $ git commit # Make your first commit (mention the first output) + ``` + + - **`README`**: Go through this top-level instruction file and make it fit + to your pipeline: update the text and etc. Don't forget that your + colleagues or anyone else, will first be drawn to read this file, so + make it as easy as possible for them to understand your + work. Therefore, also check and update `README` one last time when you + are ready to publish your work (and its reproduction pipeline). + + - **First input dataset**: The user manages the top-level directory of the + input data through the variables set in + `reproduce/config/pipeline/DIRECTORIES.mk.in` (the user actually edits + a `DIRECTORIES.mk` file that is copied from the `.mk.in` file, but the + `.mk` file is not under version control). So open this file and + replace `SURVEY` with the name of your input survey or dataset (all in + capital letters), for example if you are working on data from the XDF + survey, replace `SURVEY` with `XDF`. But don't change the value, just + the name. Afterwards, change any occurrence of `SURVEY` in the whole + pipeline with the new name. You can find the occurrences with a simple + command like the ones shown below. We follow the Make convention here + that all `ONLY-CAPITAL` variables are those directly set by the user + and all `small-caps` variables are set by the pipeline designer. All + variables that also depend on this survey have a `survey` in their + name. Hence, also correct all these occurrences to your new name in + small-caps. Of course, ignore those occurrences that are irrelevant, + like those in this file. Note that in the raw version of this template + no target depends on these files, so they are ignored. Afterwards, set + the webpage and correct the filenames in + `reproduce/src/make/download.mk' if necessary. + + ```shell + $ grep -r SURVEY ./ + $ grep -r survey ./ + ``` + + - **Other input datasets**: Add any other input datasets that may be + necessary for your research to the pipeline based on the example + above. + + +Tips on using the pipeline +========================== + +The following is a list of points, tips, or recommendations that have been +learned after some experience with this pipeline. Please don't hesitate to +share any experience you gain after using this pipeline with us. In this +way, we can add it here for others to also benefit. + + - **Modularity**: Modularity is the key to easy and clean growth of a + project. So it is always best to break up a job into as many + sub-components as reasonable. Here are some tips to stay modular. + + - *Short recipes*: if you see the recipe of a rule becoming more than a + few lines, it probably a good sign that you should break up the job. + + - *Context-based (many) Makefiles*: This pipeline is designed to allow + the easy inclusion of many Makefiles (in `reproduce/src/make/*.mk`) + for maximal modularity. So keep the rules for closely related parts + of the processing in separate Makefiles. + + - *Clear file names*: Be very clear and descriptive with the naming of + the files and the variables because a few months after the processing + it will be very hard to remember what each one does. Also this helps + others (your collaborators or other people reading the pipeline after + it is published) to more easily understand your work and find their + way around. + + - *Standard naming*: As the project grows, following a good standard in + naming the files is very useful. Try best to use multiple word + filenames for anything that is non-trivial (separating the words with + a `-`). For example if you have a Makefile for creating a catalog and + another two for processing it under models A and B, you can name them + like this: `catalog-create.mk`, `catalog-modela.mk` and + `catalog-modelb.mk`. In this way, when listing the contents of + `reproduce/src/make` to see all the Makefiles, those related to the + catalog will all be close to each other and thus easily found. This + also helps in auto-completions by the shell or text editors like + Emacs. + + - *Source directories*: If you need to add scripts (shell, Python, AWK + or any other language), keep them in a separate directory under + `reproduce/src`, with the appropriate name. + + - *Configuration files*: If your research uses special programs as part + of the processing, put all their configuration files in a devoted + directory (with the program's name) within + `reproduce/config`. Similar to the `reproduce/config/gnuastro` + directory (which is put in the template as a demo in case you use GNU + Astronomy Utilities). It is much cleaner and readable (thus less + buggy) to avoid mixing the configuration files, even if there is no + technical necessity. + + + - **Contents**: It is good practice to follow the following + recommendations on the contents of your files, whether they are source + code for a program, Makefiles, scripts or configuration files + (copyrights aren't necessary for the latter). + + - *Copyright*: Always start a file containing programming constructs + with a copyright statement like the ones that this template starts + with (for example in the top level `Makefile`). + + - *Comments*: Comments are vital for readability (by yourself in two + months or others). Describe everything you can about why you are + doing something, how you are doing it, and what you expect the result + to be. Write the comments as if it was what you would say to describe + the variable, recipe or rule to a friend sitting beside you. When + writing the pipeline it is very tempting to just steam ahead, but be + patient and write comments before the rules or recipes. This will + also allow you to think more about what you should be doing. Also, in + several months when you come back to the code, you will appreciate + the effort of writing them. Just don't forget to also read and update + the comment first if you later want to make changes to the variable, + recipe or rule. As a general rule of thumb: first the comments, then + the code. + + + - **Make programming**: Here are some experiences that we have come to + learn over the years in using Make and are useful/handy in research + contexts. + + - *Automatic variables*: These are wonderful and very useful Make + constructs that greatly shrink the text, while help in read-ability, + robustness (less bugs in typos for example) and generalization. For + example even when a rule only has one target or one prerequisite, + always use `$@` instead of the target's name, `$<` instead of the + first prerequisite, `$^` instead of the full list of prerequisites + and etc. You can see the full list of automatic variables + [here](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html). If + you use GNU Make, you can also see this page on your command-line: + + ```shell + $ info make "automatic variables + ``` + + - *Large files*: If you are dealing with very large files (thus having + multiple copies of them for intermediate steps is not possible), one + solution is the following strategy. Set a small plain text file as + the actual target and delete the large file when it is no longer + needed by the pipeline (in the last rule that needs it). Below is a + simple demonstration of doing this, where we use Gnuastro's + Arithmetic program to add all pixels of the input image with 2 and + create `large1.fits`. We then subtract 2 from `large1.fits` to create + `large2.fits` and delete `large1.fits` in the same rule (when its no + longer needed). We can later do the same with `large2.fits` when it + is no longer needed and so on. + ``` + large1.fits.txt: input.fits + astarithmetic $< 2 + --output=$(subst .txt,,$@) + echo "done" > $@ + large2.fits.txt: large1.fits.txt + astarithmetic $(subst .txt,,$<) 2 - --output=$(subst .txt,,$@) + rm $(subst .txt,,$<) + echo "done" > $@ + ``` + A more advanced Make programmer will use [Make's call + function](https://www.gnu.org/software/make/manual/html_node/Call-Function.html) + to define a wrapper in `reproduce/src/make/initialize.mk`. This + wrapper will replace `$(subst .txt,,XXXXX)`. Therefore, it will be + possible to greatly simplify this repetitive statement and make the + code even more readable throughout the whole pipeline. + + + - **Dependencies**: It is critically important to exactly document, keep + and check the versions of the programs you are using in the pipeline. + + - *Check versions*: In `reproduce/src/make/initialize.mk`, check the + versions of the programs you are using. + + - *Keep the source tarball of dependencies*: keep a tarball of the + necessary version of all your dependencies (and also a copy of the + higher-level libraries they depend on). Software evolves very fast + and only in a few years, a feature might be changed or removed from + the mainstream version or the software server might go down. To be + safe, keep a copy of the tarballs (they are hardly ever over a few + megabytes, very insignificant compared to the data). If you intend to + release the pipeline in a place like Zenodo, then you can create your + submission early (before public release) and upload/keep all the + necessary tarballs (and data) there. + + - *Keep your input data*: The input data is also critical to the + pipeline, so like the above for software, make sure you have a backup + of them + + + + + + + +Appendix: Introduction to this concept from link above +====================================================== + +In case [the link above](http://akhlaghi.org/reproducible-science.html) is +not accessible at the time of reading, here is a copy of the introduction +of that link, describing the necessity for a reproduction pipeline like +this (copied on February 7th, 2018): + +The most important element of a "scientific" statement/result is the fact +that others should be able to falsify it. The Tsunami of data that has +engulfed astronomers in the last two decades, combined with faster +processors and faster internet connections has made it much more easier to +obtain a result. However, these factors have also increased the complexity +of a scientific analysis, such that it is no longer possible to describe +all the steps of an analysis in the published paper. Citing this +difficulty, many authors suffice to describing the generalities of their +analysis in their papers. + +However, It is impossible to falsify (or even study) a result if you can't +exactly reproduce it. The complexity of modern science makes it vitally +important to exactly reproduce the final result. Because even a small +deviation can be due to many different parts of an analysis. Nature is +already a black box which we are trying so hard to comprehend. Not letting +other scientists see the exact steps taken to reach a result, or not +allowing them to modify it (do experiments on it) is a self-imposed black +box, which only exacerbates our ignorance. + +Other scientists should be able to reproduce, check and experiment on the +results of anything that is to carry the "scientific" label. Any result +that is not reproducible (due to incomplete information by the author) is +not scientific: the readers have to have faith in the subjective experience +of the authors in the very important choice of configuration values and +order of operations: this is contrary to the scientific spirit.
\ No newline at end of file |