diff options
| author | Mohammad Akhlaghi <mohammad@akhlaghi.org> | 2021-10-04 02:51:45 +0200 |
|---|---|---|
| committer | Mohammad Akhlaghi <mohammad@akhlaghi.org> | 2022-01-21 01:15:24 +0100 |
| commit | 8463df97c6f26ec4d22cd5828bb0574fd5e450d2 (patch) | |
| tree | dbaa2e7c2fc44856eb98555b79c6814f210a6c17 /README.md | |
| parent | 775fc036e0091f05ff56e41b855bc416b9ed36c8 (diff) | |
IMPORTANT: Updates to almost all software
This commit primarily affects the configuration step of Maneage'd projects,
and in particular, updated versions of the many of the software (see
P.S.). So it shouldn't affect your high-level analysis other than the
version bumps of the software you use (and the software's possibly
improve/changed behavior).
The following software (and thus their dependencies) couldn't be updated as
described below:
- Cryptography: isn't building because it depends on a new
setuptools-rust package that has problems
(https://savannah.nongnu.org/bugs/index.php?61731), so it has been
commented in 'versions.conf'.
- SecretStorage: because it depends on Cryptography.
- Keyring: because it depends on SecretStorage.
- Astroquery: because it depends on Keyring.
This is a "squashed" commit after rebasing a development branch of 60
commits corresponding to a roughly two-month time interval. The following
people contributed to this branch.
- Boudewijn Roukema added all the R software infrastructure and the R
packages, as well as greatly helping in fixing many bugs during the
update.
- Raul Infante-Sainz helped in testing and debugging the build.
- Pedram Ashofteh Ardakani found and fixed a bug.
- Zahra Sharbaf helped in testing and found several bugs.
Below a description of the most noteworthy points is given.
- Software tarballs: all updated software now have a unified format
tarball (ustar; if not possible, pax) and unified compression (Lzip) in
Maneage's software repository in Zenodo
(https://doi.org/10.5281/zenodo.3883409). For more on this See
https://savannah.nongnu.org/task/?15699 . This won't affect any extra
software you would like to add; you can use any format recognized by
GNU Tar, and all common compression algorithms. This new requirement is
only for software that get merged to the core Maneage branch.
- Metastore (and thus libbsd and libmd) moved to highlevel: Metastore
(and the packages it depends on) is a high-level product that is only
relevant during the project development (like Emacs!): when the user
wants the file meta data (like dates) to be unchanged after checking
out branches. So it should be considered a high-level software, not
basic. Metastore also usually causes many more headaches and error
messages, so personally, I have stopped using it! Instead I simply
merge my branches in a separate clone, then pull the merge commit: in
this way, the files of my project aren't re-written during the checkout
phase and therefore their dates are untouched (which can conflict with
Make's dates on configuration files).
- The un-official cloned version of Flex (2.6.4-91 until this commit) was
causing problems in the building of Netpbm, so with this commit, it has
been moved back to version 2.6.4.
- Netpbm's official page had version 10.73.38 as the latest stable
tarball that was just released in late 2021. But I couldn't find our
previously-used version 10.86.99 anywhere (to see when it was released
and why we used it! Its at last more than one year old!). So the
official stable version is being used now.
- Improved instructions in 'README.md' for building software environment
in a Docker container (while having project source and output data
products on the local system; including the usage of the host's
'/dev/shm' to speed up temporary operations).
- Until now, the convention in Maneage was to put eight SPACE characters
before the comment lines within recipes. This was done because by
default GNU Emacs (also many other editors) show a TAB as eight
characters. However, in other text editors, online browsers, or even
the Git diff, a TAB can correspond to a different number of
characters. In such cases, the Maneage recipes wouldn't look too
interesting (the comments and the recipe commands would show a
different indentation!).
With this commit, all the comment lines in the Makefiles within the
core Maneage branch have a hash ('#') as their first character and a
TAB as the second. This allows the comment lines in recipes to have the
same indentation as code; making the code much more easier to read in a
general scenario including a 'git diff' (editor agnostic!).
P.S. List of updated software with their old and new versions
- Software with no version update are not mentioned.
- The old version of newly added software are shown with '--'.
Name (Basic) Old version New version
------------ ----------- -----------
Bzip2 1.0.6 1.0.8
CURL 7.71.1 7.79.1
Dash 0.5.10.2 0.5.11.5
File 5.39 5.41
Flock 0.2.3 0.4.0
GNU Bash 5.0.18 5.1.8
GNU Binutils 2.35 2.37
GNU Coreutils 8.32 9.0
GNU GCC 10.2.0 11.2.0
GNU M4 1.4.18 1.4.19
GNU Readline 8.0 8.1.1
GNU Tar 1.32 1.34
GNU Texinfo 6.7 6.8
GNU diffutils 3.7 3.8
GNU findutils 4.7.0 4.8.0
GNU gmp 6.2.0 6.2.1
GNU grep 3.4 3.7
GNU gzip 1.10 1.11
GNU libunistring 0.9.10 1.0
GNU mpc 1.1.0 1.2.1
GNU mpfr 4.0.2 4.1.0
GNU nano 5.2 6.0
GNU ncurses 6.2 6.3
GNU wget 1.20.3 1.21.2
Git 2.28.0 2.34.0
Less 563 590
Libxml2 2.9.9 2.9.12
Lzip 1.22-rc2 1.22
OpenSLL 1.1.1a 3.0.0
Patchelf 0.10 0.13
Perl 5.32.0 5.34.0
Podlators -- 4.14
Name (Highlevel) Old version New version
---------------- ----------- -----------
Apachelog4cxx 0.10.0-603 0.12.1
Astrometry.net 0.80 0.85
Boost 1.73.0 1.77.0
CFITSIO 3.48 4.0.0
Cmake 3.18.1 3.21.4
Eigen 3.3.7 3.4.0
Expat 2.2.9 2.4.1
FFTW 3.3.8 3.3.10
Flex 2.6.4-91 2.6.4
Fontconfig 2.13.1 2.13.94
Freetype 2.10.2 2.11.0
GNU Astronomy Utilities 0.12 0.16.1-e0f1
GNU Autoconf 2.69.200-babc 2.71
GNU Automake 1.16.2 1.16.5
GNU Bison 3.7 3.8.2
GNU Emacs 27.1 27.2
GNU GDB 9.2 11.1
GNU GSL 2.6 2.7
GNU Help2man 1.47.11 1.48.5
Ghostscript 9.52 9.55.0
ICU -- 70.1
ImageMagick 7.0.8-67 7.1.0-13
Libbsd 0.10.0 0.11.3
Libffi 3.2.1 3.4.2
Libgit2 1.0.1 1.3.0
Libidn 1.36 1.38
Libjpeg 9b 9d
Libmd -- 1.0.4
Libtiff 4.0.10 4.3.0
Libx11 1.6.9 1.7.2
Libxt 1.2.0 1.2.1
Netpbm 10.86.99 10.73.38
OpenBLAS 0.3.10 0.3.18
OpenMPI 4.0.4 4.1.1
Pixman 0.38.0 0.40.0
Python 3.8.5 3.10.0
R 4.0.2 4.1.2
SWIG 3.0.12 4.0.2
Util-linux 2.35 2.37.2
Util-macros 1.19.2 1.19.3
Valgrind 3.15.0 3.18.1
WCSLIB 7.3 7.7
Xcb-proto 1.14 1.14.1
Xorgproto 2020.1 2021.5
Name (Python) Old version New version
------------- ----------- -----------
Astropy 4.0 5.0
Beautifulsoup4 4.7.1 4.10.0
Beniget -- 0.4.1
Cffi 1.12.2 1.15.0
Cryptography 2.6.1 36.0.1
Cycler 0.10.0 0.11.0+}
Cython 0.29.21 0.29.24
Esutil 0.6.4 0.6.9
Extension-helpers -- 0.1
Galsim 2.2.1 2.3.3
Gast -- 0.5.3
Jinja2 -- 3.0.3
MPI4py 3.0.3 3.1.3
Markupsafe -- 2.0.1
Numpy 1.19.1 1.21.3
Packaging -- 21.3
Pillow -- 8.4.0
Ply -- 3.11
Pyerfa -- 2.0.0.1
Pyparsing 2.3.1 3.0.4
Pythran -- 0.11.0
Scipy 1.5.2 1.7.3
Setuptools 41.6.0 58.3.0
Six 1.12.0 1.16.0
Uncertainties 3.1.2 3.1.6
Wheel -- 0.37.0
Name (R) Old version New version
-------- ----------- -----------
Cli -- 2.5.0
Colorspace -- 2.0-1
Cowplot -- 1.1.1
Crayon -- 1.4.1
Digest -- 0.6.27
Ellipsis -- 0.3.2
Fansi -- 0.5.0
Farver -- 2.1.0
Ggplot2 -- 3.3.4
Glue -- 1.4.2
GridExtra -- 2.3
Gtable -- 0.3.0
Isoband -- 0.2.4
Labeling -- 0.4.2
Lifecycle -- 1.0.0
Magrittr -- 2.0.1
MASS -- 7.3-54
Mgcv -- 1.8-36
Munsell -- 0.5.0
Pillar -- 1.6.1
R-Pkgconfig -- 2.0.3
R6 -- 2.5.0
RColorBrewer -- 1.1-2
Rlang -- 0.4.11
Scales -- 1.1.1
Tibble -- 3.1.2
Utf8 -- 1.2.1
Vctrs -- 0.3.8
ViridisLite -- 0.4.0
Withr -- 2.4.2
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 232 |
1 files changed, 174 insertions, 58 deletions
@@ -1,7 +1,7 @@ Reproducible source for XXXXXXXXXXXXXXXXX ------------------------------------------------------------------------- -Copyright (C) 2018-2021 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ +Copyright (C) 2018-2022 Mohammad Akhlaghi <mohammad@akhlaghi.org>\ See the end of the file for license conditions. This is the reproducible project source for the paper titled "**XXX XXXXX @@ -188,6 +188,55 @@ analysis and finally create the final paper). +### Building on ARM + +As of 2021-10-13, very little testing of Maneage has been done on arm64 +(tested in [aarch64](https://en.wikipedia.org/wiki/AArch64)). However, +_some_ testing has been done on [the +PinePhone](https://en.wikipedia.org/wiki/PinePhone), running +[Debian/Mobian](https://wiki.mobian-project.org/doku.php?id=pinephone). In +principle default Maneage branch (not all high-level software have been +tested) should run fully (configure + make) from the raw source to the +final verified pdf. Some issues that you might need to be aware of are +listed below. + +#### Older packages + +In old packages that may be still needed and that have an old +`config.guess` file (e.g. from 2002, such as fftw2-2.1.5-4.2, that are not +in the base Maneage branch) may crash during the build. A workaround is to +provide an updated (e.g. 2018) 'config.guess' file (automake --add-missing +--force-missing --copy) in 'reproduce/software/patches/' and copy it over +the old file during the build of the package. + +#### An un-killable running job + +Vampires may be a problem on the pinephone/aarch64. A "vampire" is defined +here as a job that is in the "R" (running) state, using nearly 95-100% of a +cpu, for an extremely long time (hours), without producing any output to +its log file, and is immune to being killed by the user or root with 'kill +-9'. A reboot and relaunch of the './project configure --existing-conf' +command is the only solution currently known (as of 2021-10-13) for +vampires. These are known to have occurred with linux-image-5.13-sunxi64. + + +#### RAM/swap space + +Adding atleast 3 Gb of swap space (man swapon, man mkswap, man dd) on the +eMMC may help to reduce the chance of having errors due to the lack of RAM. + + +#### Time scale + +On the PinePhone v1.2b, apart from the time wasted by vampires, expect +roughly 24 hours' wall time in total for the full 'configure' phase. The +default 'maneage' example calculations, diagrams and pdf production are +light and should be very fast. + + + + + ### Building in Docker containers Docker containers are a common way to build projects in an independent @@ -251,8 +300,9 @@ MB), not the full TeXLive collection! ```shell FROM debian:stable-slim - RUN apt-get update && apt-get install -y gcc g++ wget + RUN apt update && apt install -y gcc g++ wget RUN useradd -ms /bin/sh maneager + RUN printf '123\n123' | passwd root USER maneager WORKDIR /home/maneager RUN mkdir build @@ -285,22 +335,22 @@ MB), not the full TeXLive collection! ```shell # C and C++ compiler. - RUN apt-get update && apt-get install -y gcc g++ + RUN apt update && apt install -y gcc g++ # Uncomment this if you don't have 'software-XXXX.tar.gz' (below). - #RUN apt-get install -y wget + #RUN apt install -y wget ``` 3. **Define a user:** Some core software packages will complain if you try to install them as the default (root) user. Generally, it is also good - practice to avoid being the root user. After building the Docker image, - you can always run it as root with this command: `docker run -u 0 -it - XXXXXXX` (where `XXXXXXX` is the image identifier). Hence with the - commands below we define a `maneager` user and activate it for the next - steps. + practice to avoid being the root user. Hence with the commands below we + define a `maneager` user and activate it for the next steps. But just + in case root access is necessary temporarily, with the `passwd` + command, we are setting the root password to `123`. ```shell RUN useradd -ms /bin/sh maneager + RUN printf '123\n123' | passwd root USER maneager WORKDIR /home/maneager ``` @@ -476,21 +526,24 @@ steps. deleted later). ```shell - mkdir docker - cd docker + mkdir docker-tmp + cd docker-tmp ``` 3. Make a `Dockerfile` (within the new/empty directory) with the - following contents. Just replacing `UID` with your user ID (found in - step 1 above). + following contents. Just replace `UID` with your user ID (found in + step 1 above). Note that we are manually setting the `maneager` (user) + password to `123` and the root password to '456' (both should be + repeated because they must be confirmed by `passwd`). ``` FROM debian:stable-slim - RUN apt-get update && apt-get install -y gcc g++ wget - RUN useradd -ms /bin/sh --uid UID maneager + RUN useradd -ms /bin/sh --uid UID maneager; \ + printf '123\n123' | passwd maneager; \ + printf '456\n456' | passwd root USER maneager WORKDIR /home/maneager - RUN mkdir build + RUN mkdir build; mkdir build/analysis ``` 4. Create a Docker image based on the `Dockerfile` above. Just replace @@ -499,9 +552,7 @@ steps. to have root/administrator previlages when running it, so ```shell - sudo su - docker build -t MANEAGEBASE ./ - exit + sudo docker build -t MANEAGEBASE ./ ``` 5. You don't need the temporary directory any more (the docker image is @@ -509,43 +560,87 @@ steps. ```shell cd .. - rm -rf docker + rm -rf docker-tmp ``` 6. Put the following contents into a newly created plain-text file called - `docker-run`, while setting the initial variables based on your system - (the `software_dir` and `data_dir` can point to empty directories: if - you don't already have the necessary software or data, they - will/should be downloaded automatically). + `docker-run`, while setting the mandatory variables based on your + system. The name `docker-run` is already inside Maneage's `.gitignore` + file, so you don't have to worry about mistakenly commiting this file + (which contains private information: directories in this computer). ``` #!/bin/sh - # Create Docker container from existing image. This script be run in the - # top project source directory (that has 'README.md' and 'paper.tex'). If - # not, replace the '$(pwd)' with the project source directory. + # + # Create a Docker container from an existing image of the built + # software environment, but with the source, data and build (analysis) + # directories directly within the host file system. This script should + # be run in the top project source directory (that has 'README.md' and + # 'paper.tex'). If not, replace the '$(pwd)' part with the project + # source directory. + + # MANDATORY: Name of Docker container docker_name=MANEAGEBASE - data_dir=/PATH/TO/DATA/DIRECTORY - analysis_dir=/PATH/TO/ANALYSIS/DIRECTORY - software_dir=/PATH/TO/SOFTWARE/DIRECTORY - sudo docker run -v $(pwd):/home/maneager/source \ - -v $analysis_dir:/home/maneager/build/analysis \ - -v $software_dir:/home/maneager/software \ - -v $data_dir:/home/maneager/data \ - -it $docker_name + + # MANDATORY: Location of "build" directory on this system (to host the + # 'analysis' sub-directory for output data products and possibly others). + build_dir=/PATH/TO/THIS/PROJECT/S/BUILD/DIR + + # OPTIONAL: Location of project's input data in this system. If not + # present, a 'data' directory under the build directory will be created. + data_dir=/PATH/TO/THIS/PROJECT/S/DATA/DIR + + # OPTIONAL: Location of software tarballs to use in building Maneage's + # internal software environment. + software_dir=/PATH/TO/SOFTWARE/TARBALL/DIR + + + + + + # Internal proceessing + # -------------------- + # + # Sanity check: Make sure that the build directory actually exists. + if ! [ -d $build_dir ]; then + echo "ERROR: '$build_dir' doesn't exist"; exit 1; + fi + + # If the host operating system has '/dev/shm', then give Docker access + # to it also for improved speed in some scenarios (like configuration). + if [ -d /dev/shm ]; then shmopt="-v /dev/shm:/dev/shm"; + else shmopt=""; fi + + # If the 'analysis' and 'data' directories (that are mounted), don't exist, + # then create them (otherwise Docker will create them as 'root' before + # creating the container, and we won't have permission to write in them. + analysis_dir="$build_dir"/analysis + if ! [ -d $analysis_dir ]; then mkdir $analysis_dir; fi + + # If the data or software directories don't exist, put them in the build + # directory (they will remain empty, but this helps in simplifiying the + # mounting command!). + if ! [ -d $data_dir ]; then + data_dir="$build_dir"/data + if ! [ -d $data_dir ]; then mkdir $data_dir; fi + fi + if ! [ -d $software_dir ]; then + software_dir="$build_dir"/tarballs-software + if ! [ -d $software_dir ]; then mkdir $software_dir; fi + fi + + # Run the Docker image while setting up the directories. + sudo docker run -v "$software_dir":/home/maneager/tarballs-software \ + -v "$analysis_dir":/home/maneager/build/analysis \ + -v "$data_dir":/home/maneager/data \ + -v "$(pwd)":/home/maneager/source \ + $shmopt -it $docker_name ``` - 7. Make the `docker-run` script executable and **put its name in your - `.gitignore`**. It is important that this file doesn't go into your - project's history because it contains directory names only for this - particular system (you can always recreate it and update the directory - values for another system, by looking at this `README.md` and - copy-pasting). If you use the standard `docker-run` name for this tiny - script, it is already included in Maneage's `.gitignore`, so you don't - need to re-insert it there. + 7. Make the `docker-run` script executable. ```shell - chmod docker-run - emacs .gitignore + chmod +x docker-run ``` 8. You can now start the Docker image by executing your newly added @@ -556,19 +651,40 @@ steps. ./docker-run ``` - 9. You are now within the container. Go into the project source directory - and run these commands to build the software environment. + 9. You are now within the container. First, we'll add the GNU C and C++ + compilers (which are necessary to build our own programs in Maneage) + and the GNU WGet downloader (which may be necessary if you don't have + a core software's tarball already). Maneage will build pre-defined + versions of both and will use them. But for the very first packages, + they are necessary. In the process, by setting the `PS1` environment + variable, we'll define a color-coding for the interactive shell prompt + (red for root and purple for the user). + + ```shell + su + echo 'export PS1="[\[\033[01;31m\]\u@\h \W\[\033[32m\]\[\033[00m\]]# "' >> ~/.bashrc + source ~/.bashrc + apt update + apt install -y gcc g++ wget + exit + echo 'export PS1="[\[\033[01;35m\]\u@\h \W\[\033[32m\]\[\033[00m\]]$ "' >> ~/.bashrc + source ~/.bashrc + ``` + + 10. Now that the compiler is ready, we can start Maneage's + configuration. So let's go into the project source directory and run + these commands to build the software environment. ```shell cd source - ./project configure --build-dir=/home/maneager/build \ - --software-dir=/home/maneager/software \ - --input-dir=/home/maneager/data + ./project configure --input-dir=/home/maneager/data \ + --build-dir=/home/maneager/build \ + --software-dir=/home/maneager/tarballs-software ``` - 10. After the configuration finishes successfully, it will say so. It will - then ask you to run `./project make`. **But don't do that yet**. Keep - this Docker container open and don't exit the container or + 11. After the configuration finishes successfully, it will say so. It will + then ask you to run `./project make`. **But don't do that + yet**. Keep this Docker container open and don't exit the container or terminal. Open a new terminal, and follow the steps described in the sub-section above to preserve (or "commit") the built container as a Docker image. Let's assume you call it `MY-PROJECT-ENV`. After the new @@ -579,13 +695,13 @@ steps. docker image list # In the other terminal. ``` - 11. Now that you have safely "committed" your current Docker container + 12. Now that you have safely "committed" your current Docker container into a separate Docker image, you can **exit the container** safely with the `exit` command. Don't worry, you won't loose the built software environment: it is all now saved separately within the Docker image. - 12. Re-open your `docker-run` script and change `MANEAGEBASE` to + 13. Re-open your `docker-run` script and change `MANEAGEBASE` to `MY-PROJECT-ENV` (or any other name you set for the environment you committed above). @@ -593,7 +709,7 @@ steps. emacs docker-run ``` - 13. That is it! You can now always easily enter your container (only for + 14. That is it! You can now always easily enter your container (only for the software environemnt) with the command below. Within the container, any file you save/edit in the `source` directory of the docker container is the same file on your host OS and any file you @@ -607,7 +723,7 @@ steps. ./docker-run ``` - 14. In case you want to store the image as a single file as backup or to + 15. In case you want to store the image as a single file as backup or to move to another computer, you can run the commands below. They will produce a single `project-env.tar.gz` file. @@ -616,7 +732,7 @@ steps. gzip --best project-env.tar ``` - 15. To load the tarball above into a clean docker environment (for example + 16. To load the tarball above into a clean docker environment (for example on another system) copy the `my-project-env.tar.gz` file there and run the command below. You can then create the `docker-run` script for that system and run it to enter. Just don't forget that if your |