From 5a6e6e667b6263424135eec34e9d49da51449936 Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Sun, 25 Apr 2021 23:52:22 +0100 Subject: README.md: edited steps to only build software env in Docker image Until now, while the series of steps mentioned in 'README.md' were complete, they had some implicit thing in them that made it a little hard to run as a checklist (the commands to do some basic things weren't included). Also, it was recommending to run a long 'docker run ...' command, which wasn't too user friendly. With this commit, the series of steps is now a complete checklist, containing every step. Also, the checklist now recommends putting the long 'docker run' command inside a script called 'docker-run' that will also do a 'sudo' internally (thus making things very easy for a first-time user). Also, since the 'docker-run' script contains host OS-specific directory names, it should not be under control, so it has been added to the '.gitignore' file in case users decide to keep this same name (which is recommended). --- README.md | 224 +++++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 149 insertions(+), 75 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 07f900f..66c1d53 100644 --- a/README.md +++ b/README.md @@ -459,6 +459,8 @@ docker cp CONTAINER:/file/path/within/container /host/path/target + + #### Only software environment in the Docker image You can set the docker image to only contain the software environment and @@ -468,96 +470,168 @@ image to a minimum (only containing the built software environment) to easily move it from one computer to another. Below we'll summarize the steps. -1. Get your user ID with this command: `id -u`. + 1. Get your user ID with this command: `id -u`. -2. Put the following lines into a `Dockerfile` of an otherwise empty -directory. Just replacing `UID` with your user ID (found in the step -above). This will build the basic directory structure. for the next steps. + 2. Make a new (empty) directory called `docker` temporarily (will be + deleted later). -```shell -FROM debian:stable-slim -RUN apt-get update && apt-get install -y gcc g++ wget -RUN useradd -ms /bin/sh --uid UID maneager -USER maneager -WORKDIR /home/maneager -RUN mkdir build -``` + ```shell + mkdir docker + cd docker + ``` -3. Create an image based on the `Dockerfile` above. Just replace `PROJECT` -with your desired name. + 3. Make a `Dockerfile` (within the new/empty directory) with the + following contents. Just replacing `UID` with your user ID (found in + step 1 above). -```shell -docker build -t PROJECT ./ -``` + ``` + FROM debian:stable-slim + RUN apt-get update && apt-get install -y gcc g++ wget + RUN useradd -ms /bin/sh --uid UID maneager + USER maneager + WORKDIR /home/maneager + RUN mkdir build + ``` -4. Run the command below to create a container based on the image and mount -the desired directories on your host into the special directories of your -container. Just don't forget to replace `PROJECT` and set the `/PATH`s to -the respective paths in your host operating system. + 4. Create a Docker image based on the `Dockerfile` above. Just replace + `MANEAGEBASE` with your desired name (this won't be your final image, + so you can safely use a name like `maneage-base`). Note that you need + to have root/administrator previlages when running it, so -```shell -docker run -v /PATH/TO/PROJECT/SOURCE:/home/maneager/source \ - -v /PATH/TO/PROJECT/ANALYSIS/OUTPUTS:/home/maneager/build/analysis \ - -v /PATH/TO/SOFTWARE/SOURCE/CODE/DIR:/home/maneager/software \ - -v /PATH/TO/RAW/INPUT/DATA:/home/maneager/data \ - -it PROJECT -``` + ```shell + sudo su + docker build -t MANEAGEBASE ./ + exit + ``` -5. After running the command above, you are within the container. Go into -the project source directory and run these commands to build the software -environment. + 5. You don't need the temporary directory any more (the docker image is + saved in Docker's own location, and accessible from anywhere). -```shell -cd /home/maneager/source -./project configure --build-dir=/home/maneager/build \ - --software-dir=/home/maneager/software \ - --input-dir=/home/maneager/data -``` + ```shell + cd .. + rm -rf docker + ``` -6. After the configuration finishes successfully, it will say so and ask -you to run `./project make`. But don't do that yet. Keep this Docker -container open and don't exit the container or terminal. Open a new -terminal, and follow the steps described in the sub-section above to -preserve the built container as a Docker image. Let's assume you call it -`PROJECT-ENV`. After the new image is made, you should be able to see the -new image in the list of images with this command (in the same terminal -that you created the image): + 6. Put the following contents into a newly created plain-text file called + `docker-run`, while setting the initial variables based on your system + (the `software_dir` and `data_dir` can point to empty directories: if + you don't already have the necessary software or data, they + will/should be downloaded automatically). -```shell -docker image list # In the other terminal. -``` + ``` + #!/bin/sh + # Create Docker container from existing image. This script be run in the + # top project source directory (that has 'README.md' and 'paper.tex'). If + # not, replace the '$(pwd)' with the project source directory. + docker_name=MANEAGEBASE + data_dir=/PATH/TO/DATA/DIRECTORY + analysis_dir=/PATH/TO/ANALYSIS/DIRECTORY + software_dir=/PATH/TO/SOFTWARE/DIRECTORY + sudo docker run -v $(pwd):/home/maneager/source \ + -v $analysis_dir:/home/maneager/build/analysis \ + -v $software_dir:/home/maneager/software \ + -v $data_dir:/home/maneager/data \ + -it $docker_name + ``` -7. Now you can run `./project make` in the initial container. You will see -that all the built products (temporary or final datasets or PDFs), will be -written in the `/PATH/TO/PROJECT/ANALYSIS/OUTPUTS` directory of your -host. You can even change the source of your project on your host operating -system an re-run Make to see the effect on the outputs and add/commit the -changes to your Git history within your host. You can also exit the -container any time. You can later load the `PROJECT-ENV` environment image -into a new container with the same `docker run -v ...` command above, just -use `PROJECT-ENV` instead of `PROJECT`. + 7. Make the `docker-run` script executable and **put its name in your + `.gitignore`**. It is important that this file doesn't go into your + project's history because it contains directory names only for this + particular system (you can always recreate it and update the directory + values for another system, by looking at this `README.md` and + copy-pasting). If you use the standard `docker-run` name for this tiny + script, it is already included in Maneage's `.gitignore`, so you don't + need to re-insert it there. -8. In case you want to store the image as a single file as backup or to -move to another computer, you can run the commands below. They will produce -a single `project-env.tar.gz` file. + ```shell + chmod docker-run + emacs .gitignore + ``` -```shell -docker save -o project-env.tar PROJECT-ENV -gzip --best project-env.tar -``` + 8. You can now start the Docker image by executing your newly added + script like below (it will ask for your root password). You will + notice that you are in the Docker container with the changed prompt. + + ```shell + ./docker-run + ``` + + 9. You are now within the container. Go into the project source directory + and run these commands to build the software environment. + + ```shell + cd source + ./project configure --build-dir=/home/maneager/build \ + --software-dir=/home/maneager/software \ + --input-dir=/home/maneager/data + ``` + + 10. After the configuration finishes successfully, it will say so. It will + then ask you to run `./project make`. **But don't do that yet**. Keep + this Docker container open and don't exit the container or + terminal. Open a new terminal, and follow the steps described in the + sub-section above to preserve (or "commit") the built container as a + Docker image. Let's assume you call it `MY-PROJECT-ENV`. After the new + image is made, you should be able to see the new image in the list of + images with this command (in yet another terminal): + + ```shell + docker image list # In the other terminal. + ``` + + 11. Now that you have safely "committed" your current Docker container + into a separate Docker image, you can **exit the container** safely + with the `exit` command. Don't worry, you won't loose the built + software environment: it is all now saved separately within the Docker + image. + + 12. Re-open your `docker-run` script and change `MANEAGEBASE` to + `MY-PROJECT-ENV` (or any other name you set for the environment you + committed above). + + ```shell + emacs docker-run + ``` + + 13. That is it! You can now always easily enter your container (only for + the software environemnt) with the command below. Within the + container, any file you save/edit in the `source` directory of the + docker container is the same file on your host OS and any file you + build in your `build/analysis` directory (within the Maneage'd + project) will be on your host OS. You can even use your container's + Git to store the history of your project in your host OS. See the next + step in case you want to move your built software environment to + another computer. + + ```shell + ./docker-run + ``` + + 14. In case you want to store the image as a single file as backup or to + move to another computer, you can run the commands below. They will + produce a single `project-env.tar.gz` file. + + ```shell + docker save -o my-project-env.tar MY-PROJECT-ENV + gzip --best project-env.tar + ``` + + 15. To load the tarball above into a clean docker environment (for example + on another system) copy the `my-project-env.tar.gz` file there and run + the command below. You can then create the `docker-run` script for + that system and run it to enter. Just don't forget that if your + `analysis_dir` directory is empty on the new/clean system. So you + should first run the same `./project configure ...` command above in + the docker image so it connects the environment to your source. Don't + worry, it won't build any software and should finish in a second or + two. Afterwards, you can safely run `./project make` and continue + working like you did on the old system. + + ```shell + docker load --input my-project-env.tar.gz + ``` -9. To load the tarball above into a clean docker environment (either on the -same system or in another system), and create a new container from the -image like above (the `docker run -v ...` command). Just don't forget that -if your `/PATH/TO/PROJECT/ANALYSIS/OUTPUTS` directory is empty on the -new/clean system, you should first run `./project configure -e` in the -docker image so it builds the core file structure there. Don't worry, it -won't build any software and should finish in a second or two. Afterwards, -you can safely run `./project make`. -```shell -docker load --input project-env.tar.gz -``` -- cgit v1.2.1 From 8463df97c6f26ec4d22cd5828bb0574fd5e450d2 Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Mon, 4 Oct 2021 02:51:45 +0200 Subject: IMPORTANT: Updates to almost all software This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2 --- README.md | 232 ++++++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 174 insertions(+), 58 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 66c1d53..250f807 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ Reproducible source for XXXXXXXXXXXXXXXXX ------------------------------------------------------------------------- -Copyright (C) 2018-2021 Mohammad Akhlaghi \ +Copyright (C) 2018-2022 Mohammad Akhlaghi \ See the end of the file for license conditions. This is the reproducible project source for the paper titled "**XXX XXXXX @@ -188,6 +188,55 @@ analysis and finally create the final paper). +### Building on ARM + +As of 2021-10-13, very little testing of Maneage has been done on arm64 +(tested in [aarch64](https://en.wikipedia.org/wiki/AArch64)). However, +_some_ testing has been done on [the +PinePhone](https://en.wikipedia.org/wiki/PinePhone), running +[Debian/Mobian](https://wiki.mobian-project.org/doku.php?id=pinephone). In +principle default Maneage branch (not all high-level software have been +tested) should run fully (configure + make) from the raw source to the +final verified pdf. Some issues that you might need to be aware of are +listed below. + +#### Older packages + +In old packages that may be still needed and that have an old +`config.guess` file (e.g. from 2002, such as fftw2-2.1.5-4.2, that are not +in the base Maneage branch) may crash during the build. A workaround is to +provide an updated (e.g. 2018) 'config.guess' file (automake --add-missing +--force-missing --copy) in 'reproduce/software/patches/' and copy it over +the old file during the build of the package. + +#### An un-killable running job + +Vampires may be a problem on the pinephone/aarch64. A "vampire" is defined +here as a job that is in the "R" (running) state, using nearly 95-100% of a +cpu, for an extremely long time (hours), without producing any output to +its log file, and is immune to being killed by the user or root with 'kill +-9'. A reboot and relaunch of the './project configure --existing-conf' +command is the only solution currently known (as of 2021-10-13) for +vampires. These are known to have occurred with linux-image-5.13-sunxi64. + + +#### RAM/swap space + +Adding atleast 3 Gb of swap space (man swapon, man mkswap, man dd) on the +eMMC may help to reduce the chance of having errors due to the lack of RAM. + + +#### Time scale + +On the PinePhone v1.2b, apart from the time wasted by vampires, expect +roughly 24 hours' wall time in total for the full 'configure' phase. The +default 'maneage' example calculations, diagrams and pdf production are +light and should be very fast. + + + + + ### Building in Docker containers Docker containers are a common way to build projects in an independent @@ -251,8 +300,9 @@ MB), not the full TeXLive collection! ```shell FROM debian:stable-slim - RUN apt-get update && apt-get install -y gcc g++ wget + RUN apt update && apt install -y gcc g++ wget RUN useradd -ms /bin/sh maneager + RUN printf '123\n123' | passwd root USER maneager WORKDIR /home/maneager RUN mkdir build @@ -285,22 +335,22 @@ MB), not the full TeXLive collection! ```shell # C and C++ compiler. - RUN apt-get update && apt-get install -y gcc g++ + RUN apt update && apt install -y gcc g++ # Uncomment this if you don't have 'software-XXXX.tar.gz' (below). - #RUN apt-get install -y wget + #RUN apt install -y wget ``` 3. **Define a user:** Some core software packages will complain if you try to install them as the default (root) user. Generally, it is also good - practice to avoid being the root user. After building the Docker image, - you can always run it as root with this command: `docker run -u 0 -it - XXXXXXX` (where `XXXXXXX` is the image identifier). Hence with the - commands below we define a `maneager` user and activate it for the next - steps. + practice to avoid being the root user. Hence with the commands below we + define a `maneager` user and activate it for the next steps. But just + in case root access is necessary temporarily, with the `passwd` + command, we are setting the root password to `123`. ```shell RUN useradd -ms /bin/sh maneager + RUN printf '123\n123' | passwd root USER maneager WORKDIR /home/maneager ``` @@ -476,21 +526,24 @@ steps. deleted later). ```shell - mkdir docker - cd docker + mkdir docker-tmp + cd docker-tmp ``` 3. Make a `Dockerfile` (within the new/empty directory) with the - following contents. Just replacing `UID` with your user ID (found in - step 1 above). + following contents. Just replace `UID` with your user ID (found in + step 1 above). Note that we are manually setting the `maneager` (user) + password to `123` and the root password to '456' (both should be + repeated because they must be confirmed by `passwd`). ``` FROM debian:stable-slim - RUN apt-get update && apt-get install -y gcc g++ wget - RUN useradd -ms /bin/sh --uid UID maneager + RUN useradd -ms /bin/sh --uid UID maneager; \ + printf '123\n123' | passwd maneager; \ + printf '456\n456' | passwd root USER maneager WORKDIR /home/maneager - RUN mkdir build + RUN mkdir build; mkdir build/analysis ``` 4. Create a Docker image based on the `Dockerfile` above. Just replace @@ -499,9 +552,7 @@ steps. to have root/administrator previlages when running it, so ```shell - sudo su - docker build -t MANEAGEBASE ./ - exit + sudo docker build -t MANEAGEBASE ./ ``` 5. You don't need the temporary directory any more (the docker image is @@ -509,43 +560,87 @@ steps. ```shell cd .. - rm -rf docker + rm -rf docker-tmp ``` 6. Put the following contents into a newly created plain-text file called - `docker-run`, while setting the initial variables based on your system - (the `software_dir` and `data_dir` can point to empty directories: if - you don't already have the necessary software or data, they - will/should be downloaded automatically). + `docker-run`, while setting the mandatory variables based on your + system. The name `docker-run` is already inside Maneage's `.gitignore` + file, so you don't have to worry about mistakenly commiting this file + (which contains private information: directories in this computer). ``` #!/bin/sh - # Create Docker container from existing image. This script be run in the - # top project source directory (that has 'README.md' and 'paper.tex'). If - # not, replace the '$(pwd)' with the project source directory. + # + # Create a Docker container from an existing image of the built + # software environment, but with the source, data and build (analysis) + # directories directly within the host file system. This script should + # be run in the top project source directory (that has 'README.md' and + # 'paper.tex'). If not, replace the '$(pwd)' part with the project + # source directory. + + # MANDATORY: Name of Docker container docker_name=MANEAGEBASE - data_dir=/PATH/TO/DATA/DIRECTORY - analysis_dir=/PATH/TO/ANALYSIS/DIRECTORY - software_dir=/PATH/TO/SOFTWARE/DIRECTORY - sudo docker run -v $(pwd):/home/maneager/source \ - -v $analysis_dir:/home/maneager/build/analysis \ - -v $software_dir:/home/maneager/software \ - -v $data_dir:/home/maneager/data \ - -it $docker_name + + # MANDATORY: Location of "build" directory on this system (to host the + # 'analysis' sub-directory for output data products and possibly others). + build_dir=/PATH/TO/THIS/PROJECT/S/BUILD/DIR + + # OPTIONAL: Location of project's input data in this system. If not + # present, a 'data' directory under the build directory will be created. + data_dir=/PATH/TO/THIS/PROJECT/S/DATA/DIR + + # OPTIONAL: Location of software tarballs to use in building Maneage's + # internal software environment. + software_dir=/PATH/TO/SOFTWARE/TARBALL/DIR + + + + + + # Internal proceessing + # -------------------- + # + # Sanity check: Make sure that the build directory actually exists. + if ! [ -d $build_dir ]; then + echo "ERROR: '$build_dir' doesn't exist"; exit 1; + fi + + # If the host operating system has '/dev/shm', then give Docker access + # to it also for improved speed in some scenarios (like configuration). + if [ -d /dev/shm ]; then shmopt="-v /dev/shm:/dev/shm"; + else shmopt=""; fi + + # If the 'analysis' and 'data' directories (that are mounted), don't exist, + # then create them (otherwise Docker will create them as 'root' before + # creating the container, and we won't have permission to write in them. + analysis_dir="$build_dir"/analysis + if ! [ -d $analysis_dir ]; then mkdir $analysis_dir; fi + + # If the data or software directories don't exist, put them in the build + # directory (they will remain empty, but this helps in simplifiying the + # mounting command!). + if ! [ -d $data_dir ]; then + data_dir="$build_dir"/data + if ! [ -d $data_dir ]; then mkdir $data_dir; fi + fi + if ! [ -d $software_dir ]; then + software_dir="$build_dir"/tarballs-software + if ! [ -d $software_dir ]; then mkdir $software_dir; fi + fi + + # Run the Docker image while setting up the directories. + sudo docker run -v "$software_dir":/home/maneager/tarballs-software \ + -v "$analysis_dir":/home/maneager/build/analysis \ + -v "$data_dir":/home/maneager/data \ + -v "$(pwd)":/home/maneager/source \ + $shmopt -it $docker_name ``` - 7. Make the `docker-run` script executable and **put its name in your - `.gitignore`**. It is important that this file doesn't go into your - project's history because it contains directory names only for this - particular system (you can always recreate it and update the directory - values for another system, by looking at this `README.md` and - copy-pasting). If you use the standard `docker-run` name for this tiny - script, it is already included in Maneage's `.gitignore`, so you don't - need to re-insert it there. + 7. Make the `docker-run` script executable. ```shell - chmod docker-run - emacs .gitignore + chmod +x docker-run ``` 8. You can now start the Docker image by executing your newly added @@ -556,19 +651,40 @@ steps. ./docker-run ``` - 9. You are now within the container. Go into the project source directory - and run these commands to build the software environment. + 9. You are now within the container. First, we'll add the GNU C and C++ + compilers (which are necessary to build our own programs in Maneage) + and the GNU WGet downloader (which may be necessary if you don't have + a core software's tarball already). Maneage will build pre-defined + versions of both and will use them. But for the very first packages, + they are necessary. In the process, by setting the `PS1` environment + variable, we'll define a color-coding for the interactive shell prompt + (red for root and purple for the user). + + ```shell + su + echo 'export PS1="[\[\033[01;31m\]\u@\h \W\[\033[32m\]\[\033[00m\]]# "' >> ~/.bashrc + source ~/.bashrc + apt update + apt install -y gcc g++ wget + exit + echo 'export PS1="[\[\033[01;35m\]\u@\h \W\[\033[32m\]\[\033[00m\]]$ "' >> ~/.bashrc + source ~/.bashrc + ``` + + 10. Now that the compiler is ready, we can start Maneage's + configuration. So let's go into the project source directory and run + these commands to build the software environment. ```shell cd source - ./project configure --build-dir=/home/maneager/build \ - --software-dir=/home/maneager/software \ - --input-dir=/home/maneager/data + ./project configure --input-dir=/home/maneager/data \ + --build-dir=/home/maneager/build \ + --software-dir=/home/maneager/tarballs-software ``` - 10. After the configuration finishes successfully, it will say so. It will - then ask you to run `./project make`. **But don't do that yet**. Keep - this Docker container open and don't exit the container or + 11. After the configuration finishes successfully, it will say so. It will + then ask you to run `./project make`. **But don't do that + yet**. Keep this Docker container open and don't exit the container or terminal. Open a new terminal, and follow the steps described in the sub-section above to preserve (or "commit") the built container as a Docker image. Let's assume you call it `MY-PROJECT-ENV`. After the new @@ -579,13 +695,13 @@ steps. docker image list # In the other terminal. ``` - 11. Now that you have safely "committed" your current Docker container + 12. Now that you have safely "committed" your current Docker container into a separate Docker image, you can **exit the container** safely with the `exit` command. Don't worry, you won't loose the built software environment: it is all now saved separately within the Docker image. - 12. Re-open your `docker-run` script and change `MANEAGEBASE` to + 13. Re-open your `docker-run` script and change `MANEAGEBASE` to `MY-PROJECT-ENV` (or any other name you set for the environment you committed above). @@ -593,7 +709,7 @@ steps. emacs docker-run ``` - 13. That is it! You can now always easily enter your container (only for + 14. That is it! You can now always easily enter your container (only for the software environemnt) with the command below. Within the container, any file you save/edit in the `source` directory of the docker container is the same file on your host OS and any file you @@ -607,7 +723,7 @@ steps. ./docker-run ``` - 14. In case you want to store the image as a single file as backup or to + 15. In case you want to store the image as a single file as backup or to move to another computer, you can run the commands below. They will produce a single `project-env.tar.gz` file. @@ -616,7 +732,7 @@ steps. gzip --best project-env.tar ``` - 15. To load the tarball above into a clean docker environment (for example + 16. To load the tarball above into a clean docker environment (for example on another system) copy the `my-project-env.tar.gz` file there and run the command below. You can then create the `docker-run` script for that system and run it to enter. Just don't forget that if your -- cgit v1.2.1