aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/config/versions.conf
AgeCommit message (Collapse)AuthorLines
2023-05-23Configuration: portability fixes in Less and WCSLIBHEADmaneageMohammad Akhlaghi-1/+1
SUMMARY: no change is necessary in your project, unless you use the Fortran features of WCSLIB in your project. Until now, there were two compilation failures on recent macOS computers with an M1 CPU: Less would crash because it couldn't find the relevant PCRE (perl-compatible regular expression) libraries and WCSLIB would crash because the LLVM compiler's Fortran features could not be built. With this commit, both issues have been fixed by disabling the relevant feature. Extensive comments have been placed in both places in case your project needs these features, so please see the comments in the relevant part of 'reproduce/software/make/basic.mk' for Less and 'reproduce/software/make/high-level.mk' for WCSLIB. In fact the previous solution (where we would not have Fortran features in WCSLIB on macOS systems was problematic and non-reproducibile (the features of WCSLIB depended on the operating system!). Another minor change was that for macOS, we now directly use the version-string of WCSLIB to fix the internal linking issue there. As a result, WCSLIB is no longer a "Version-dependent build" software (in 'reproduce/software/config/versions.conf'). Recall that these are software that when changing the version, it is also necessary to inspect their build recipe. These two issues and their fix were discovered and fixed with the help of James Robinson.
2023-05-07Copyright years: updated to 2023, accompanied by some minor fixesMohammad Akhlaghi-3/+3
SUMMARY: just house-cleaning, no need to do anything major in your branch. Just update the copyright years in files that you have added. Until now, the latest copyright years of the whole Maneage source code was 2022! As of this commit, we have already moved to 2023 for 5 months! Furthermore, there were a few other minor issues that needed correction: - The URL to download input datasets wasn't quoted in 'initialize.mk' or the download script! As a result, when the input URL had characters that are meaningful to the shell (like '&'), the download command would not work. - The only program that had 'make check' in the 'basic.mk' programs was MPFR. At that stage, we still haven't built our own compiler at this stage, this is not accurate. - The 'pyerfa' and 'extension-helpers' packages in Python need 'setuptools_scm' on some systems. But until now, it was not in the list of their prerequisites. With this commit, all the issues above have been corrected.
2022-08-13IMPORTANT: Software updateMohammad Akhlaghi-57/+70
SUMMARY: This is a software update to make Maneage more portable and up to date. It does not involve any Maneage infrastructure changes. You should just re-build your project to make sure the updated software haven't removed/changed any of their features that you were using. In particular, for Astrometry.net users, please see the respective note in P.S.2 below. Until now, there have been many updates in the software that are built within Maneage. The last software upadte was almost one year ago. With this commit, the software in the P.S.1 have been updated. A description of notable changes in the software environment is given in P.S.2. This software environment has been tested on an Arch GNU/Linux, Debian, CentOS-7 and macOS. This commit is the merging of 24 individual commits by Raul Infante-Sainz (who put a lot of energy on porting the software below for macOS, and updating citations), Boudewijn Roukema (who helped with memory checking for GCC, and testing on Debian and CentOS), Sepideh Eskandarlou (who tested the environment) and myself. Besides the updates in the core software, the followimg improvements have also been implemented in this commit: - When you run './project shell': - A welcome message is printed that will remind the caller that they have entered a new environment, it will print the location of 'HOME' and the location of the shell startup file. - The 'reproduce/software/shell/bashrc.sh' is loaded as a startup file. This allows you to customize your interactive Maneage shell. A default step has already been placed there that will put the git branch name (in green) within the shell prompt (which was purple). This greatly helps when dealing with directories under Git version control. These settings won't bother with Maneage's default operations: through environment variables we make sure that these './project shell' features will not slow-down the calls to the shell within the non-interactive Make calls. - The host's 'COLORTERM' is passed to the Maneage environment. It is used by some programs that can have color outputs on the terminal. - Updates to citations: - Numpy and Scipy (as requested on their pages): https://numpy.org/citing-numpy and https://scipy.org/citing-scipy - Gnuastro: Added https://arxiv.org/abs/1909.11230 which describes major updates to Gnuastro after 10 releases. - When a software's paper is indexed in the SAO/NASA Astrophysics Data System (ADS), Maneage now use the BibTeX entries provided by ADS. This helps to give a unified format to most software, and more information (like ADS+arXiv hyperlinks in the BibLaTeX compilation of the default bibliography). - We were able to build this version of Maneage on a Debian system from 2010 (+12 years ago!). Only three downgrades were necessary in the "basic" software (not affecting the high-level science software!). A description of the necessary downgrades for such old systems has been added in 'README.md'. P.S.1 List of updated software: Basic software: cURL 7.79.1 --> 7.84.0 Dash 0.5.11.5 --> 0.5.11-057cd65 File 5.41 --> 5.42 GNU AWK 5.1.0 --> 5.1.1 GNU Bash 5.1.8 --> 5.2-rc2 GNU Binutils 2.37 --> 2.39 GNU Compiler Collection (GCC) 11.2.1 --> 12.1.0 GNU Findutils 4.8.0 --> 4.9.0 GNU Gzip 1.11 --> 1.12 GNU Help2man 1.48.5 --> 1.49.2 GNU Integer Set Library (ISL) 0.18 --> 0.24 GNU Libtool 2.4.6 --> 2.4.7 GNU Nano 6.0 --> 6.4 GNU Readline 8.1.1 --> 8.2-rc2 GNU libiconv 0.16 --> 0.17 Git 2.36.0 --> 2.37.1 OpenSSL 3.0.0 --> 3.0.5 PatchELF 0.13 --> 0.15.0 Perl 5.34.0 --> 5.36.0 High-level software: Astrometry.net 0.89 --> 0.91 CFITSIO 4.0.0 --> 4.1.0 CMake 3.21.4 --> 3.24.0 GNU Astronomy Utilities (Gnuastro) 0.16.1 --> 0.18 GPL Ghostscript 9.55.0 --> 9.56.1 HDF5 1.10.5 --> 1.13.1 Libjpeg 9d --> 9e Libtiff 4.3.0 --> 4.4.0 OpenBLAS 0.3.18 --> 0.3.21 PLplot n/a --> 5.15.0 Python 3.10.0 --> 3.10.6 SCAMP 2.6.7 --> 2.10.0 SWarp 2.38.0 --> 2.41.5 Util-Linux 2.37.2 --> 2.38.1 Vim 8.2 --> 9.0 WCSLIB 7.7 --> 7.11 X.org packages (used by graphical software like Ghostscript and LaTeX): Fontconfig 2.13.94 --> 2.14.0 LibX11 1.7.2 --> 1.8 LibXCB 1.14 --> 1.15 XCB-proto 1.14.1 --> 1.15 Xorg-proto 2021.5 --> 2022.1 Python modules: Astropy 5.0 --> 5.1 GalSim 2.3.3 --> 2.3.5 P.S.2: Notable points regarding the software environment: - Two new links from the host's low-level tools are now included in Maneage's build environment: - On GNU/Linux systems, the host's 'ldd' is linked inside the custom environment. This belongs to the GNU C Library (which is not yet installed in Maneage). But helps in checking the linking status of the binaries on GNU/Linux systems. - On macOS: the 'codesign' binary is included, which is used by GNU Emacs on macOS to sign the built executable. - GNU Bison has been moved in basic software (necessary for GNU Binutils). - The Zip and Unzip programs have been moved as high-level software that have to be manually requested when necessary. This is because they are not used by any of the basic software anymore. They were just installed as dependencies of GNU Tar to be close the other compression programs. Also, in the past we would use the original tarballs, and some (for example Numpy) were distributed in Zip format. However, by default, we now use a custom Lzip tarball and don't need Zip or Unzip. This was suggested by Zahra Sharbaf and Raul Infante-Sainz. - Some minor edits in 'reproduce/software/shell/tarball-prepare.sh'. In particular the 'awk' command was effectively just replacing a '_' with '-', so it just uses a simple SED expression instead. - Fixed bug 62700 (https://savannah.nongnu.org/bugs/index.php?62700) by compiling 'xz' with a patched version of the xz source file 'src/liblzma/liblzma.map'. - Astrometry.net doesn't depend on NetPBM any more. NetPBM (and its dependencies) were causing many crashes on macOS and it also a very strange build system that is hard to maintain. Astrometry.net uses it to take images as input. However, it isn't necessary when you provide Astrometry.net with a catalog. Therefore, Raul added some instructions on how to run astrometry from your own custom X-Y catalog. These instruction can be seen on top of the build rule of Astrometry.net in 'reproduce/software/make/high-level.mk'. - h5py has been removed as a dependency of Astropy. It is an optional dependency to write tables into HDF5 format. But since we couldn't get it to build on macOS it has been removed. None of the current Maneage users/developers also use this feature of Astropy! - PLplot is added a new software, but not a default pre-requisite of SCAMP (which can use it to generate figures), because there were many build problems on macOS. Instructions have been added on top of SCAMP on how to add PLplot as a dependency. - With the aim of being able to install Plplot on macOS, we have wrote several lines to fix header problems. However, we didn't succeed. In any case we are leaving these lines in case they are useful in the future. - The '-Wno-nullability-completeness' compiler flag (which is primarily necessary for macOS) is now only added for macOS systems. It was causing many warnings of un-recognized option in GNU/Linux systems. - The 'mkswap' program of Util-Linux has been disabled because it caused crashes on older kernels. Generally, its not necessary for a Maneage project because it needs root permissions to run! - LibXT (of the x.org software) has been added as a dependency of Cairo. - ImageMagick and Lzip were using the host's C++ standard library! But on GNU/Linux we build our own C++ Standard Library with GCC, so with this commit, they properly link with Maneage's C++ standard library. - ImageMagick on macOS couldn't properly link with Maneage's Ghostscript library! This has been fixed using macOS's install_name_tool. - Necessary RAM to build GCC on GNU/Linux systems changed to ~8GB, see https://savannah.nongnu.org/task/?16244#comment12 - Pythran is no longer as prerequisite of Scipy. Until now, Pythran was a prerequisite of Scipy. But we noticed that it is optional and was causing problems on macOS. - The URLs of some of the software have been updated in 'reproduce/software/config/urls.conf'. By default, these are all commented, but they can be useful when searching for new versions or when a project needs custom software that is not (yet) in Maneage.
2022-06-10Housekeeping: some portability issues fixed; four software updatesMohammad Akhlaghi-4/+4
Until now, there were several portability issues in Maneage: 1. Maneage would crash on older operating systems (checked on Debian 6), where Wget didn't have the '--no-use-server-timestamps'. 2. On a Linux kernel 2.6.32 (of the same Debian 6 above) some features in 'util-linux' (like 'swapon' or 'libmount') wouldn't build and wouldn't let 'util-linux' complete. These features need root permissions to be useful, so the wouldn't be used in Maneage any way! But they wouldn't let Maneage get built 3. The './project shell' command would still read the host's '~/.bashrc', letting the host environment leak-in to Maneage's interactive shell. 4. The building of Flex 2.64 wouldn't complete due to a segmentation fault an Ubuntu, but NetPBM (which depends on Flex) would crash with a wrong usage of 'yyunput'. This had actually caused a non-update to Flex in a previous Maneage software update. 5. The update Astrometry.net would assume SExtractor's executable name is 'source-extractor'; causing a crash in usage. This forced the users to manually create a 'source-extractor' symbolic link in the '.local/bin' directory. 6. The 'reproduce/software/shell/tarball-prepare.sh' script (that is used for making Maneage-standard tarballs) wouldn't accept option values with an '=' between the option name and value! It also didnt' print sufficiently informative messages and errors (for example it would say "skipping ..." (making the user think there is a problem!), but it was actually that the file already existed! 7. The 'reproduce/analysis/make/prepare.mk' and 'reproduce/analysis/make/verify.mk' Makefiles that needed to reject some of the 'makesrc' sub-Makefiles would simply substitute their names with nothing. But this would cause problems when the name is part of the name of another sub-Makefile. 8. On the Debian 6 system mentioned above the raw 'df' command's output wasn't in the expected format; so Maneage would fail to properly detect the free space in the disk. With these commit, all the issues above have been solved: for 1, A check has been added to avoid using that option. For 2, those 'util-linux' features have been disabled. For 3, the '--norc' and '--noprofile' options have beed added to the call to Bash. For 4, see below. For 5, the symbolic link is now automatically made with SExtractor. For 6, the option reading components of that script have been fully re-written and more robust sanity checks are also added, with more informative warnings. For 7, the 'subst' function of Make was replaced with 'filter-out' and this fixed the problem. For 8, 'df' is called with the '-P' option so it has a unified format in all versions. For 4, the versions of 'flex' and 'netpbm' have been updated. Since they were the dependency of 'astrometrynet', that has also been updated. In the process, we discovered that 'lzip' has a new version which claims to be faster, so that is also updated. lzip 1.22 --> 1.23 astrometrynet 0.85 --> 0.89 flex 2.6.4 --> 2.6.4-410-74a89fd netpbm 10.73.39 --> 10.73.39 NetPBM needed some manual manipulation in its source (to remove the extra line), so the necessary steps have been added to its build recipe in 'reproduce/software/make/high-level.mk'.
2022-05-10initialize.mk: Git call in variable works with LD_LIBRARY_PATHMohammad Akhlaghi-4/+1
Until now, the '$(project-commit-hash)' Make variable of 'initialize.mk' simply called 'git' to find the commit hash. However, due to one of the recent software updates, we noticed that this command is no longer working (and the project commit hash wasn't getting printed in the PDF)! The problem was that Maneage's Git, couldn't find the 'libiconv' library that it was built with. With this commit, the '$(shell' command that calls Git, first exports 'LD_LIBRARY_PATH' to Maneage's software build directory. As a result, the Git command can work and will report the commit as a LaTeX macro to be used in the paper. To avoid relying on PATH outside of Make recipes, we now also directly call the Git executable with Maneage. Some other minor issues have been found and fixed in this commit: - README-hacking.md: some minor edits and typo corrections. - initialize.mk: the '$(curdir)' variable is now used in several places that we were calling 'pwd'. - versions.conf: 'xlsxio-version' now included with other programs. Until now it was commented because GCC 11.1.0 had issues with it. However, GCC 11.2.0 doesn't have a problem any more, so it has been returned to the list of all high-level programs. - xorg.mk: used same format to comment recipe lines as the other Makefiles (a '#' followed by a TAB). - preamble-pgfplots.tex: lines to comment for building an EPS figure with PGFPlots have been re-formatted to be more human-readable.
2022-04-20Updated Git, Coreutils and Emacs, new script to prepare tarballsPedram Ashofteh Ardakani-3/+4
Until now, one had to follow the instructions from [1] to prepare a standard software tarball before merging with the low-level tarballs-software repository [2]. The script only worked for '.tar.gz' suffix and was only available as a comment on Savannah (in [1]). With this commit, the script has been imported into Maneage as 'reproduce/software/shell/tarball-prepare.sh' to simplify future software updates. It work with all supported '.tar.*' suffixes (of the upstream tarball repository) and will convert the tarballs to Maneage's standard format. Also, this script has a minimal argument parser and can skip the tarballs that are already unpacked, allowing faster tests. This script was used to update the versions of: Coreutiles 9.0 --> 9.1 Git 2.34 --> 2.36 Emacs 27.2 --> 28.1 The main motive behind this update was Git which announced a vulnerability issue [3] and suggested an update to the latest version as soon as possible. More detail is described in this github blog [4], but in summary, it was a security issue on multi-user systems that has been found and fixed by Git developers. Since Maneage is often installed on such shared systems, it was important to make this update. GNU Coreutils and GNU Emacs were also updated because they are also commonly used. The following improvements have also done with this commit: - .gitignore: ignore emacs auto-save files (that end with a '#') - README-hacking.md: In the checklist for updating the Maneage branch, the no-longer-necessary '--decorate' option of Git was removed from the command to check the general branch history. [1] https://savannah.nongnu.org/task/?15699 [2] https://git.maneage.org/tarballs-software.git/ [3] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/ [4] https://github.blog/2022-04-12-git-security-vulnerability-announced/
2022-01-21IMPORTANT: Updates to almost all softwareMohammad Akhlaghi-110/+201
This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2
2021-01-02Copyright year updated in all source filesMohammad Akhlaghi-2/+2
Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.
2020-12-02Less is now built as a basic softwareMohammad Akhlaghi-0/+1
Less is rarely used in non-interactive mode and is primarily intended for interactively viewing large files. So its need within Maneage (for batch processing) wasn't often felt until now. However, when running './project shell' (which completely closes-off the outside environment), or building a Maneage'd project within a minimal container that doesn't have less, it becomes hard to use Git (and in particular its 'diff' output which depends on 'less'). With this commit, Less has been added as a dependency of Git in 'basic.mk'. In total its built product is roughly 800KB and builds within a second or two. So it isn't a burden on any project. But it can be very useful when the projects are being developed within the Maneage environment itself.
2020-08-28Plain text editors: nano in basic, emacs and vim in high-levelMohammad Akhlaghi-0/+3
While a project is under development, the raw analysis software are not the only necessary software in a project. We also need tools to all the edit plain-text files within the Maneaged project. Usually people use their operating system's plain-text editor. However, when working on the project on a new computer, or in a container, the plain-text editors will have different versions, or may not be present at all! This can be very annoying and frustrating! With this commit, Maneage now installs GNU Nano as part of the basic tools. GNU Nano is a very simple and small plain text editor (the installed size is only ~3.5MB, and it is friendly to new users). Therefore, any Maneaged project can assume atleast Nano will be present (in particular when no editor is available on the running system!). GNU Emacs and VIM (both without extra dependencies, in particular without GUI support) are also optionally available in 'high-level.mk' (by adding them to 'TARGETS.conf'). The basic idea for the more advanced editors (Emacs and VIM) is that project authors can add their favorite editor while they are working on the project, but upon publication they can remove them from 'TARGETS.conf'. A few other minor things came up during this work and are now also fixed: - The 'file' program and its libraries like 'libmagic' were linking to system's 'libseccomp'! This dependency then leaked into Nano (which depends on 'libmagic'). But this is just an extra feature of 'file', only for the Linux kernel. Also, we have no dependency on it so far. So 'file' is not configured to not build with 'libseccomp'. - A typo was fixed in the line where the physical core information is being read on macOS. - The top-level directories when running './project shell' are now quoted (in case they have special characters).
2020-08-08IMPORTANT: New software versions (17 basic, 16 high-level and 7 Python)Mohammad Akhlaghi-41/+44
It was a long time that the Maneage software versions hadn't been updated. With this commit, the versions of all basic software were checked and 17 of that had newer versions were updated. Also, 16 high-level programs and libraries were updated as well as 7 Python modules. The full list is available below. Basic Software (affecting all projects) --------------------------------------- bash 5.0.11 -> 5.0.18 binutils 2.32 -> 2.35 coreutils 8.31 -> 8.32 curl 7.65.3 -> 7.71.1 file 5.36 -> 5.39 gawk 5.0.1 -> 5.1.0 gcc 9.2.0 -> 10.2.0 gettext 0.20.2 -> 0.21 git 2.26.2 -> 2.28.0 gmp 6.1.2 -> 6.2.0 grep 3.3 -> 3.4 libbsd 0.9.1 -> 0.10.0 ncurses 6.1 -> 6.2 perl 5.30.0 -> 5.32.0 sed 4.7 -> 4.8 texinfo 6.6 -> 6.7 xz 5.2.4 -> 5.2.5 Custom programs/libraries ------------------------- astrometrynet 0.77 -> 0.80 automake 0.16.1 -> 0.16.2 bison 3.6 -> 3.7 cfitsio 3.47 -> 3.48 cmake 3.17.0 -> 3.18.1 freetype 2.9 -> 2.10.2 gdb 8.3 -> 9.2 ghostscript 9.50 -> 9.52 gnuastro 0.11 -> 0.12 libgit2 0.28.2 -> 1.0.1 libidn 1.35 -> 1.36 openmpi 4.0.1 -> 4.0.4 R 3.6.2 -> 4.0.2 python 3.7.4 -> 3.8.5 wcslib 6.4 -> 7.3 yaml 0.2.2 -> 0.2.5 Python modules -------------- cython 0.29.6 -> 0.29.21 h5py 2.9.0 -> 2.10.0 matplotlib 3.1.1 -> 3.3.0 mpi4py 3.0.2 -> 3.0.3 numpy 1.17.2 -> 1.19.1 pybind11 2.4.3 -> 2.5.0 scipy 1.3.1 -> 1.5.2
2020-06-30Core Xorg libraries necessary for Ghostscript now includedMohammad Akhlaghi-0/+21
Until now, in order to build Ghostscript, the project used the host's Xorg libraries. This was because we hadn't yet added the necessary build rules for them. With this commit, the instructions to build the necessary Xorg libraries for Ghostscript have also been added. Also, the shared Ghostscript library has been built with this commit and two sets of standard fonts are also included, setting us on the path to build TeXLive from source later. This task was done with the help and support of Raul Infante-Sainz.
2020-06-28Bison installation on macOS fixed by updating to version 3.6Raul Infante-Sainz-1/+1
Until this commit, there was a problem when building Bison in parallel in macOS systems. With this commit, this problem has been fixed by updating Bison to its most recent version (3.6).
2020-06-27IMPORTANT: many improvements to low-level software building phaseMohammad Akhlaghi-44/+49
POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause conflicts to your project if you have changed the software building Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and 'python.mk'). If your project has only added analysis, it shouldn't be affected. This is a large commit, involving a long series of corrections in a differnt branch which is now finally being merged into the core Maneage branch. All changes were related and came up naturally as the low-level infrastructure was improved. So separating them in the end for the final merge would have been very time consuming and we are merging them as one commit. In general, the software building Makefiles are now much more easier to read, modify and use, along with several new features that have been added. See below for the full list. - Until now, Maneage needed the host to have a 'make' implementation because Make was necessary to build Lzip. Lzip is then used to uncompress the source of our own GNU Make. However, in the minimalist/slim versions of operating systems (for example used to build Docker images) Make isn't included by default. Since Lzip was the only program before our own GNU Make was installed, we consulting Antonio Diaz Diaz (creator of Lzip) and he kindly added the necessary functionality to a new version of Lzip, which we are using now. Hence we don't need to assume a Make implementation on the host any more. With this commit, Lzip and GNU Make are built without Make, allowing everything else to be safely built with our own custom version of GNU Make and not using the host's 'make' at all. - Until recently (Commit 3d8aa5953c4) GNU Make was built in 'basic.mk'. Therefore 'basic.mk' was written in a way that it can be used with other 'make' implementations also (i.e., important shell commands starting with '&&' and ending in '\' without any comments between them!). Furthermore, to help in style uniformity, the rules in 'high-level.mk' and 'python.mk' also followed a similar structure. But due to the point above, we can now guarantee that GNU Make is used from the very first Makefile, so this hard-to-read structure has been removed in the software build recipes and they are much more readable and edit-friendly now. - Until now, the default backup servers where at some fixed URLs, on our own pages or on Gitlab. But recently we uploaded all the necessary software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is more suitable for this task (it promises longevity, has a fixed DOI, while allowing us to add new content, or new software tarball versions). With this commit, a small script has been written to extract the most recent Zenodo upload link from the Zenodo DOI and use it for downloading the software source codes. - Until now, we primarily used the webpage of each software for downloading its tarball. But this caused many problems: 1) Some of them needed Javascript before the download, 2) Some URLs had a complex dependency on the version number, 3) some servers would be randomly down for maintenance and etc. So thanks to the point above, we now use the Zenodo server as the primary download location. However, if a user wants to use a custom software that is not (yet!) in Zenodo, the download script gives priority to a custom URL that the users can give as Make variables. If that variable is defined, then the script will use that URL before going onto Zenodo. We now have a special place for such URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a good documentation themselves) are preserved here, but are commented by default. - The software source code downloading and checksum verification step has been moved into a Make function called 'import-source' (defined in the 'build-rules.mk' and loaded in all software Makefiles). Having taken all the low-level steps there, I noticed that there is no more need for having the tarball as a separate target! So with this commit, a single rule is the only place that needs to be edited/added (greatly simplifying the software building Makefiles). - Following task #15272, A new option has been added to the './project' script called '--all-highlevel'. When this option is given, the contents of 'TARGETS.conf' are ignored and all the software in Maneage are built (selected by parsing the 'versions.conf' file). This new option was added to confirm the extensive changes made in all the software building recipes and is great for development/testing purposes. - Many of the software hadn't been tested for a long time! So after using the newly added '--all-highlevel', we noticed that some need to be updated. In general, with this commit, 'libpaper' and 'pcre' were added as new software, and the versions of the following software was updated: 'boost', 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in' shell script was added in 'reproduce/software/shell/' which is installed with 'libpaper'. - Even though we intentionally add the necessary flags to add RPATH inside the built executable at compilation time, some software don't do it (different software on different operating systems!). Until now, for historical reasons this check was done in different ways for different software on GNU/Linux sytems. But now it is unified: if 'patchelf' is present we apply it. Because of this, 'patchelf' has been put as a top-level prerequisite, right after Tar and is installed before anything else. - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in 'basic.mk', it was 'glibtool'! This caused many confusions and is corrected with this commit (in 'basic.mk', it is also 'libtool'). - A new argument is added to the './project' script to allow easy loading of the project's shell and environment for fast/temporary testing of things in the same environment as the project. Before activating the project's shell, we completely remove all host environment variables to simulate the project's environment. It can be called with this command: './project shell'. A simple prompt has also been added to highlight that the user is using the Maneage shell!
2020-06-02Core software build before using Make to build other softwareMohammad Akhlaghi-1/+2
Until now, Maneage would only build Flock before building everything else using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid parallel downloads during the building of software (which could cause network problems). But after recently trying Maneage on FreeBSD (which is not yet complete, see bug #58465), we noticed that the BSD implemenation of Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq' parts) and its shell also had some peculiarities. It was thus decided to also install our own minimalist shell, Make and compressor program before calling 'basic.mk'. In this way, 'basic.mk' can now assume the same GNU Make features that high-level.mk and python.mk assume. The pre-make building of software is now organized in 'reproduce/software/shell/pre-make-build.sh'. Another nice feature of this commit is for macOS users: until now the default macOS Make had problems for parallel building of software, so 'basic.mk' was built in one thread. But now that we can build the core tools with GNU Make on macOS too, it uses all threads. Furthermore, since we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have to finish every line of a long rule with a backslash to keep variables and such. Generally, the pre-make software are now organized like this: first we build Lzip before anything else: it is downloaded as a simple '.tar' file that is not compressed (only ~400kb). Once Lzip is built, the pre-make phase continues with building GNU Make, Dash (a minimalist shell) and Flock. All of their tarballs are in '.tar.lz'. Maneage then enters 'basic.mk' and the first program it builds is GNU Gzip (itself packaged as '.tar.lz'). Once Gzip is built, we build all the other compression software (all downloaded as '.tar.gz'). Afterwards, any compression standard for other software is fine because we have it. In the process, a bug related to using backup servers was found in 'reproduce/analysis/bash/download-multi-try' for calling outside of 'basic.mk' and removed Bash-specific features. As a result of that bug-fix, because we now have multiple servers for software tarballs, the backup servers now have their own configuration file in 'reproduce/software/config/servers-backup.conf'. This makes it much easier to maintain the backup server list across the multiple places that we need it. Some other minor fixes: - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'. - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD (which doesn't have GCC). - We are now using 'uname -s' to specify if we are on a Linux kernel or not, if not, we are still using the old 'on_mac_os' variable. - While I was trying to build on FreeBSD, I noticed some further corrections that could help. For example the 'makelink' Make-function now takes a third argument which can be a different name compared to the actual program (used for examle to make a link to '/usr/bin/cc' from 'gcc'. - Until now we didn't know if the host's Make implementation supports placing a '@' at the start of the recipe (to avoid printing the actual commands to standard output). Especially in the tarball download phase, there are many lines that are printed for each download which was really annoying. We already used '@' in 'high-level.mk' and 'python.mk' before, but now that we also know that 'basic.mk' is called with our custom GNU Make, we can use it at the start for a cleaner stdout. - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a system where we can't install GCC (or has activated the '--host-cc' option), it may not be present and the project shouldn't break because of this. So with this commit, when a Fortran compiler isn't present, WCSLIB will be built with the '--disable-fortran' configuration option. This commit (task #15667) was completed with help/checks by Raul Infante-Sainz and Boud Roukema.
2020-05-23New software: Valgrind and PatchBoud Roukema-1/+6
With this commit, Maneage now includes instructions to build the memory tracing tool Valgrind and the program 'patch' (to apply corrections/patches in text files and in particular the sources of programs). For this version of Valgrind, some patches were necessary for an interface with OpenMPI 2.x (which is the case now). Also note that this version of Valgrind's checks can fail with GCC 10.1.x (when using '--host-cc'), and the failures aren't due to internal problems but due to how the tests are designed (https://bugs.gentoo.org/707598). So currently if any of Valgrind's checks fail, Maneage still assumes that Valgrind was built and installed successfully. While testing on macOS, we noticed that it needs the macOS-specific 'mig' program which we can't build in Maneage. DESCRIPTION: The mig command invokes the Mach Interface Generator to generate Remote Procedure Call (RPC) code for client-server style Mach IPC from specification files. So a symbolic link to the system's 'mig' is now added to the project's programs on macOS systems. This commit's build of Patch and Valgrind has been tested on two GNU/Linux distributions (Debian and ArchLinux) as well as macOS. Work on this commit started by Boud Roukema, but also involved tests and corrections by Mohammad Akhlaghi and Raul Infante-Sainz.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-9/+4
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-05-08GNU Gettext built as a dependency of BashRaul Infante-Sainz-2/+4
Until now Maneage used the host's GNU Gettext if it was present. Gettext is a relatively low-level software that enables programs to print messages in different languages based on the host environment. Even though it has not direct effect on the running of the software for Maneage and the lanugage environment in Maneage is pre-determined, it is necessary to have it because if the basic programs see it in the host they will link with it and will have problems if/when the host's Gettext is updated. With this commit (which is actually a squashed rebase of 9 commits by Raul and Mohammad), Gettext and its two extra dependencies (libxml2 and libunistring) are now installed within Maneage as a basic software and built before GNU Bash. As a result, all programs built afterwards will successfully link with our own internal version of Gettext and libraries. To get this working, some of the basic software dependencies had to updated and re-ordered and it has been tested in both GNU/Linux and macoS. Some other minor issues that are fixed with this commit - Until this commit, when TeX was not installed, the warning message saying how to run the configure step in order to re-configure the project was not showing the option `-e'. However, the use of this option is more convenient than entering the top-build directory and etc every time. So with this commit, the warning message has been changed in order use the option `-e' in the re-configure of the project. - Until now, on macOS systems, Bash was not linking with our internally built `libncurses'. With this commit, this has been fixed by setting `--withcurses=yes' for Bash's configure script.
2020-05-01Fixed OpenSSL deprecation bug on some OSs, causing problems in libgit2Boud Roukema-1/+5
Until this commit, the configure step would fail with an error when compiling libgit2 on a test system. The origin of this bug, on the OS that was tested, appears to be that in OpenSSL Version 1.1.1a, openssl/ec.h fails to include openssl/openconf.h. The bug is described in more detail at https://savannah.nongnu.org/bugs/index.php?58263 With this commit, this is fixed by manually inserting a necessary components. In particular, `sed` is used to insert a preprocessor instruction into `openssl/openconf.h`, defining `DEPRECATED_1_2_0(f)`, for an arbitrary section of code `f`, to include that code rather than exclude it or warn about it. This commit is valid provided that openssl remains at a version earlier than 1.2.0. Starting at version 1.2.0, deprecation warnings should be run normally. We have thus moved the version of OpenSSL in `versions.conf' to the section for programs that need to be manually checked for version updates with a note to remind the user when reaching that version. Other packages that use OpenSSL may benefit from this commit, not just libgit2.
2020-04-27Configuration: improved version separation from tarball nameBoud Roukema-2/+2
Until now, the sed script for determining URL download rules in the three software building Makefiles (`basic.mk', `high-level.mk' and `python.mk') considered package names such as `fftw-3...` and `fftw2-2.1...` to be identical. As the example above shows, this would make it hard to include some software that may hav conflicting non-number names. With this commit, the SED script that is used to separate the version from the tarball name only matches numbers that are after a dash (`-'). Therefore considers `fftw-3...` and `fftw-2...` to be identical, but `fftw-3-...` and `fftw2-2.1...` to be different. As a result of this change, the `elif' check for some of the other programs like `m4', or `help2man' was also corrected in all three Makefiles. While doing this check on all the software, we noticed that `zlib-version' is being repeated two times in `version.conf' so it was removed. It caused no complications, because both were the same number, but could lead to bugs later.
2020-04-20Maneage instead of Template in README-hacking.md and copyright noticesMohammad Akhlaghi-11/+8
Until now, throughout Maneage we were using the old name of "Reproducible Paper Template". But we have finally decided to use Maneage, so to avoid confusion, the name has been corrected in `README-hacking.md' and also in the copyright notices. Note also that in `README-hacking.md', the main Maneage branch is now called `maneage', and the main Git remote has been changed to `https://gitlab.com/maneage/project' (this is a new GitLab Group that I have setup for all Maneage-related projects). In this repository there is only one `maneage' branch to avoid complications with the `master' branch of the projects using Maneage later.
2020-04-17IMPORTANT: software config directly under reproduce/software/configMohammad Akhlaghi-0/+221
Until now the software configuration parameters were defined under the `reproduce/software/config/installation/' directory. This was because the configuration parameters of analysis software (for example Gnuastro's configurations) were placed under there too. But this was terribly confusing, because the run-time options of programs falls under the "analysis" phase of the project. With this commit, the Gnuastro configuration files have been moved under the new `reproduce/analysis/config/gnuastro' directory and the software configuration files are directly under `reproduce/software/config'. A clean build was done with this change and it didn't crash, but it may cause crashes in derived projects, so after merging with Maneage, please re-configure your project to see if anything has been missed. Please let us know if there is a problem.