diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 358 |
1 files changed, 274 insertions, 84 deletions
@@ -1,7 +1,7 @@ Reproducible source for Akhlaghi et al. (2021, arXiv:2006.03018) ---------------------------------------------------------------- -Copyright (C) 2018-2021 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ +Copyright (C) 2018-2022 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ See the end of the file for license conditions. This is the reproducible project source for the paper titled "**Toward @@ -195,6 +195,55 @@ finally create the final paper). +### Building on ARM + +As of 2021-10-13, very little testing of Maneage has been done on arm64 +(tested in [aarch64](https://en.wikipedia.org/wiki/AArch64)). However, +_some_ testing has been done on [the +PinePhone](https://en.wikipedia.org/wiki/PinePhone), running +[Debian/Mobian](https://wiki.mobian-project.org/doku.php?id=pinephone). In +principle default Maneage branch (not all high-level software have been +tested) should run fully (configure + make) from the raw source to the +final verified pdf. Some issues that you might need to be aware of are +listed below. + +#### Older packages + +In old packages that may be still needed and that have an old +`config.guess` file (e.g. from 2002, such as fftw2-2.1.5-4.2, that are not +in the base Maneage branch) may crash during the build. A workaround is to +provide an updated (e.g. 2018) 'config.guess' file (automake --add-missing +--force-missing --copy) in 'reproduce/software/patches/' and copy it over +the old file during the build of the package. + +#### An un-killable running job + +Vampires may be a problem on the pinephone/aarch64. A "vampire" is defined +here as a job that is in the "R" (running) state, using nearly 95-100% of a +cpu, for an extremely long time (hours), without producing any output to +its log file, and is immune to being killed by the user or root with 'kill +-9'. A reboot and relaunch of the './project configure --existing-conf' +command is the only solution currently known (as of 2021-10-13) for +vampires. These are known to have occurred with linux-image-5.13-sunxi64. + + +#### RAM/swap space + +Adding atleast 3 Gb of swap space (man swapon, man mkswap, man dd) on the +eMMC may help to reduce the chance of having errors due to the lack of RAM. + + +#### Time scale + +On the PinePhone v1.2b, apart from the time wasted by vampires, expect +roughly 24 hours' wall time in total for the full 'configure' phase. The +default 'maneage' example calculations, diagrams and pdf production are +light and should be very fast. + + + + + ### Building in Docker containers Docker containers are a common way to build projects in an independent @@ -258,8 +307,9 @@ MB), not the full TeXLive collection! ```shell FROM debian:stable-slim - RUN apt-get update && apt-get install -y gcc g++ wget + RUN apt update && apt install -y gcc g++ wget RUN useradd -ms /bin/sh maneager + RUN printf '123\n123' | passwd root USER maneager WORKDIR /home/maneager RUN mkdir build @@ -292,22 +342,22 @@ MB), not the full TeXLive collection! ```shell # C and C++ compiler. - RUN apt-get update && apt-get install -y gcc g++ + RUN apt update && apt install -y gcc g++ # Uncomment this if you don't have 'software-XXXX.tar.gz' (below). - #RUN apt-get install -y wget + #RUN apt install -y wget ``` 3. **Define a user:** Some core software packages will complain if you try to install them as the default (root) user. Generally, it is also good - practice to avoid being the root user. After building the Docker image, - you can always run it as root with this command: `docker run -u 0 -it - XXXXXXX` (where `XXXXXXX` is the image identifier). Hence with the - commands below we define a `maneager` user and activate it for the next - steps. + practice to avoid being the root user. Hence with the commands below we + define a `maneager` user and activate it for the next steps. But just + in case root access is necessary temporarily, with the `passwd` + command, we are setting the root password to `123`. ```shell RUN useradd -ms /bin/sh maneager + RUN printf '123\n123' | passwd root USER maneager WORKDIR /home/maneager ``` @@ -466,6 +516,8 @@ docker cp CONTAINER:/file/path/within/container /host/path/target + + #### Only software environment in the Docker image You can set the docker image to only contain the software environment and @@ -475,96 +527,234 @@ image to a minimum (only containing the built software environment) to easily move it from one computer to another. Below we'll summarize the steps. -1. Get your user ID with this command: `id -u`. + 1. Get your user ID with this command: `id -u`. -2. Put the following lines into a `Dockerfile` of an otherwise empty -directory. Just replacing `UID` with your user ID (found in the step -above). This will build the basic directory structure. for the next steps. + 2. Make a new (empty) directory called `docker` temporarily (will be + deleted later). -```shell -FROM debian:stable-slim -RUN apt-get update && apt-get install -y gcc g++ wget -RUN useradd -ms /bin/sh --uid UID maneager -USER maneager -WORKDIR /home/maneager -RUN mkdir build -``` + ```shell + mkdir docker-tmp + cd docker-tmp + ``` -3. Create an image based on the `Dockerfile` above. Just replace `PROJECT` -with your desired name. + 3. Make a `Dockerfile` (within the new/empty directory) with the + following contents. Just replace `UID` with your user ID (found in + step 1 above). Note that we are manually setting the `maneager` (user) + password to `123` and the root password to '456' (both should be + repeated because they must be confirmed by `passwd`). -```shell -docker build -t PROJECT ./ -``` + ``` + FROM debian:stable-slim + RUN useradd -ms /bin/sh --uid UID maneager; \ + printf '123\n123' | passwd maneager; \ + printf '456\n456' | passwd root + USER maneager + WORKDIR /home/maneager + RUN mkdir build; mkdir build/analysis + ``` -4. Run the command below to create a container based on the image and mount -the desired directories on your host into the special directories of your -container. Just don't forget to replace `PROJECT` and set the `/PATH`s to -the respective paths in your host operating system. + 4. Create a Docker image based on the `Dockerfile` above. Just replace + `MANEAGEBASE` with your desired name (this won't be your final image, + so you can safely use a name like `maneage-base`). Note that you need + to have root/administrator previlages when running it, so -```shell -docker run -v /PATH/TO/PROJECT/SOURCE:/home/maneager/source \ - -v /PATH/TO/PROJECT/ANALYSIS/OUTPUTS:/home/maneager/build/analysis \ - -v /PATH/TO/SOFTWARE/SOURCE/CODE/DIR:/home/maneager/software \ - -v /PATH/TO/RAW/INPUT/DATA:/home/maneager/data \ - -it PROJECT -``` + ```shell + sudo docker build -t MANEAGEBASE ./ + ``` -5. After running the command above, you are within the container. Go into -the project source directory and run these commands to build the software -environment. + 5. You don't need the temporary directory any more (the docker image is + saved in Docker's own location, and accessible from anywhere). -```shell -cd /home/maneager/source -./project configure --build-dir=/home/maneager/build \ - --software-dir=/home/maneager/software \ - --input-dir=/home/maneager/data -``` + ```shell + cd .. + rm -rf docker-tmp + ``` -6. After the configuration finishes successfully, it will say so and ask -you to run `./project make`. But don't do that yet. Keep this Docker -container open and don't exit the container or terminal. Open a new -terminal, and follow the steps described in the sub-section above to -preserve the built container as a Docker image. Let's assume you call it -`PROJECT-ENV`. After the new image is made, you should be able to see the -new image in the list of images with this command (in the same terminal -that you created the image): + 6. Put the following contents into a newly created plain-text file called + `docker-run`, while setting the mandatory variables based on your + system. The name `docker-run` is already inside Maneage's `.gitignore` + file, so you don't have to worry about mistakenly commiting this file + (which contains private information: directories in this computer). -```shell -docker image list # In the other terminal. -``` + ``` + #!/bin/sh + # + # Create a Docker container from an existing image of the built + # software environment, but with the source, data and build (analysis) + # directories directly within the host file system. This script should + # be run in the top project source directory (that has 'README.md' and + # 'paper.tex'). If not, replace the '$(pwd)' part with the project + # source directory. + + # MANDATORY: Name of Docker container + docker_name=MANEAGEBASE + + # MANDATORY: Location of "build" directory on this system (to host the + # 'analysis' sub-directory for output data products and possibly others). + build_dir=/PATH/TO/THIS/PROJECT/S/BUILD/DIR + + # OPTIONAL: Location of project's input data in this system. If not + # present, a 'data' directory under the build directory will be created. + data_dir=/PATH/TO/THIS/PROJECT/S/DATA/DIR + + # OPTIONAL: Location of software tarballs to use in building Maneage's + # internal software environment. + software_dir=/PATH/TO/SOFTWARE/TARBALL/DIR + + + + + + # Internal proceessing + # -------------------- + # + # Sanity check: Make sure that the build directory actually exists. + if ! [ -d $build_dir ]; then + echo "ERROR: '$build_dir' doesn't exist"; exit 1; + fi + + # If the host operating system has '/dev/shm', then give Docker access + # to it also for improved speed in some scenarios (like configuration). + if [ -d /dev/shm ]; then shmopt="-v /dev/shm:/dev/shm"; + else shmopt=""; fi + + # If the 'analysis' and 'data' directories (that are mounted), don't exist, + # then create them (otherwise Docker will create them as 'root' before + # creating the container, and we won't have permission to write in them. + analysis_dir="$build_dir"/analysis + if ! [ -d $analysis_dir ]; then mkdir $analysis_dir; fi + + # If the data or software directories don't exist, put them in the build + # directory (they will remain empty, but this helps in simplifiying the + # mounting command!). + if ! [ -d $data_dir ]; then + data_dir="$build_dir"/data + if ! [ -d $data_dir ]; then mkdir $data_dir; fi + fi + if ! [ -d $software_dir ]; then + software_dir="$build_dir"/tarballs-software + if ! [ -d $software_dir ]; then mkdir $software_dir; fi + fi + + # Run the Docker image while setting up the directories. + sudo docker run -v "$software_dir":/home/maneager/tarballs-software \ + -v "$analysis_dir":/home/maneager/build/analysis \ + -v "$data_dir":/home/maneager/data \ + -v "$(pwd)":/home/maneager/source \ + $shmopt -it $docker_name + ``` -7. Now you can run `./project make` in the initial container. You will see -that all the built products (temporary or final datasets or PDFs), will be -written in the `/PATH/TO/PROJECT/ANALYSIS/OUTPUTS` directory of your -host. You can even change the source of your project on your host operating -system an re-run Make to see the effect on the outputs and add/commit the -changes to your Git history within your host. You can also exit the -container any time. You can later load the `PROJECT-ENV` environment image -into a new container with the same `docker run -v ...` command above, just -use `PROJECT-ENV` instead of `PROJECT`. + 7. Make the `docker-run` script executable. -8. In case you want to store the image as a single file as backup or to -move to another computer, you can run the commands below. They will produce -a single `project-env.tar.gz` file. + ```shell + chmod +x docker-run + ``` -```shell -docker save -o project-env.tar PROJECT-ENV -gzip --best project-env.tar -``` + 8. You can now start the Docker image by executing your newly added + script like below (it will ask for your root password). You will + notice that you are in the Docker container with the changed prompt. + + ```shell + ./docker-run + ``` + + 9. You are now within the container. First, we'll add the GNU C and C++ + compilers (which are necessary to build our own programs in Maneage) + and the GNU WGet downloader (which may be necessary if you don't have + a core software's tarball already). Maneage will build pre-defined + versions of both and will use them. But for the very first packages, + they are necessary. In the process, by setting the `PS1` environment + variable, we'll define a color-coding for the interactive shell prompt + (red for root and purple for the user). + + ```shell + su + echo 'export PS1="[\[\033[01;31m\]\u@\h \W\[\033[32m\]\[\033[00m\]]# "' >> ~/.bashrc + source ~/.bashrc + apt update + apt install -y gcc g++ wget + exit + echo 'export PS1="[\[\033[01;35m\]\u@\h \W\[\033[32m\]\[\033[00m\]]$ "' >> ~/.bashrc + source ~/.bashrc + ``` + + 10. Now that the compiler is ready, we can start Maneage's + configuration. So let's go into the project source directory and run + these commands to build the software environment. + + ```shell + cd source + ./project configure --input-dir=/home/maneager/data \ + --build-dir=/home/maneager/build \ + --software-dir=/home/maneager/tarballs-software + ``` + + 11. After the configuration finishes successfully, it will say so. It will + then ask you to run `./project make`. **But don't do that + yet**. Keep this Docker container open and don't exit the container or + terminal. Open a new terminal, and follow the steps described in the + sub-section above to preserve (or "commit") the built container as a + Docker image. Let's assume you call it `MY-PROJECT-ENV`. After the new + image is made, you should be able to see the new image in the list of + images with this command (in yet another terminal): + + ```shell + docker image list # In the other terminal. + ``` + + 12. Now that you have safely "committed" your current Docker container + into a separate Docker image, you can **exit the container** safely + with the `exit` command. Don't worry, you won't loose the built + software environment: it is all now saved separately within the Docker + image. + + 13. Re-open your `docker-run` script and change `MANEAGEBASE` to + `MY-PROJECT-ENV` (or any other name you set for the environment you + committed above). + + ```shell + emacs docker-run + ``` + + 14. That is it! You can now always easily enter your container (only for + the software environemnt) with the command below. Within the + container, any file you save/edit in the `source` directory of the + docker container is the same file on your host OS and any file you + build in your `build/analysis` directory (within the Maneage'd + project) will be on your host OS. You can even use your container's + Git to store the history of your project in your host OS. See the next + step in case you want to move your built software environment to + another computer. + + ```shell + ./docker-run + ``` + + 15. In case you want to store the image as a single file as backup or to + move to another computer, you can run the commands below. They will + produce a single `project-env.tar.gz` file. + + ```shell + docker save -o my-project-env.tar MY-PROJECT-ENV + gzip --best project-env.tar + ``` + + 16. To load the tarball above into a clean docker environment (for example + on another system) copy the `my-project-env.tar.gz` file there and run + the command below. You can then create the `docker-run` script for + that system and run it to enter. Just don't forget that if your + `analysis_dir` directory is empty on the new/clean system. So you + should first run the same `./project configure ...` command above in + the docker image so it connects the environment to your source. Don't + worry, it won't build any software and should finish in a second or + two. Afterwards, you can safely run `./project make` and continue + working like you did on the old system. + + ```shell + docker load --input my-project-env.tar.gz + ``` -9. To load the tarball above into a clean docker environment (either on the -same system or in another system), and create a new container from the -image like above (the `docker run -v ...` command). Just don't forget that -if your `/PATH/TO/PROJECT/ANALYSIS/OUTPUTS` directory is empty on the -new/clean system, you should first run `./project configure -e` in the -docker image so it builds the core file structure there. Don't worry, it -won't build any software and should finish in a second or two. Afterwards, -you can safely run `./project make`. -```shell -docker load --input project-env.tar.gz -``` |