From 79ee52ed1e136fbd2592a5c2a058f73024d8b5b0 Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Fri, 5 Aug 2022 20:27:56 +0200 Subject: IMPORTANT: Software update SUMMARY: This is a software update to make Maneage more portable and up to date. It does not involve any Maneage infrastructure changes. You should just re-build your project to make sure the updated software haven't removed/changed any of their features that you were using. In particular, for Astrometry.net users, please see the respective note in P.S.2 below. Until now, there have been many updates in the software that are built within Maneage. The last software upadte was almost one year ago. With this commit, the software in the P.S.1 have been updated. A description of notable changes in the software environment is given in P.S.2. This software environment has been tested on an Arch GNU/Linux, Debian, CentOS-7 and macOS. This commit is the merging of 24 individual commits by Raul Infante-Sainz (who put a lot of energy on porting the software below for macOS, and updating citations), Boudewijn Roukema (who helped with memory checking for GCC, and testing on Debian and CentOS), Sepideh Eskandarlou (who tested the environment) and myself. Besides the updates in the core software, the followimg improvements have also been implemented in this commit: - When you run './project shell': - A welcome message is printed that will remind the caller that they have entered a new environment, it will print the location of 'HOME' and the location of the shell startup file. - The 'reproduce/software/shell/bashrc.sh' is loaded as a startup file. This allows you to customize your interactive Maneage shell. A default step has already been placed there that will put the git branch name (in green) within the shell prompt (which was purple). This greatly helps when dealing with directories under Git version control. These settings won't bother with Maneage's default operations: through environment variables we make sure that these './project shell' features will not slow-down the calls to the shell within the non-interactive Make calls. - The host's 'COLORTERM' is passed to the Maneage environment. It is used by some programs that can have color outputs on the terminal. - Updates to citations: - Numpy and Scipy (as requested on their pages): https://numpy.org/citing-numpy and https://scipy.org/citing-scipy - Gnuastro: Added https://arxiv.org/abs/1909.11230 which describes major updates to Gnuastro after 10 releases. - When a software's paper is indexed in the SAO/NASA Astrophysics Data System (ADS), Maneage now use the BibTeX entries provided by ADS. This helps to give a unified format to most software, and more information (like ADS+arXiv hyperlinks in the BibLaTeX compilation of the default bibliography). - We were able to build this version of Maneage on a Debian system from 2010 (+12 years ago!). Only three downgrades were necessary in the "basic" software (not affecting the high-level science software!). A description of the necessary downgrades for such old systems has been added in 'README.md'. P.S.1 List of updated software: Basic software: cURL 7.79.1 --> 7.84.0 Dash 0.5.11.5 --> 0.5.11-057cd65 File 5.41 --> 5.42 GNU AWK 5.1.0 --> 5.1.1 GNU Bash 5.1.8 --> 5.2-rc2 GNU Binutils 2.37 --> 2.39 GNU Compiler Collection (GCC) 11.2.1 --> 12.1.0 GNU Findutils 4.8.0 --> 4.9.0 GNU Gzip 1.11 --> 1.12 GNU Help2man 1.48.5 --> 1.49.2 GNU Integer Set Library (ISL) 0.18 --> 0.24 GNU Libtool 2.4.6 --> 2.4.7 GNU Nano 6.0 --> 6.4 GNU Readline 8.1.1 --> 8.2-rc2 GNU libiconv 0.16 --> 0.17 Git 2.36.0 --> 2.37.1 OpenSSL 3.0.0 --> 3.0.5 PatchELF 0.13 --> 0.15.0 Perl 5.34.0 --> 5.36.0 High-level software: Astrometry.net 0.89 --> 0.91 CFITSIO 4.0.0 --> 4.1.0 CMake 3.21.4 --> 3.24.0 GNU Astronomy Utilities (Gnuastro) 0.16.1 --> 0.18 GPL Ghostscript 9.55.0 --> 9.56.1 HDF5 1.10.5 --> 1.13.1 Libjpeg 9d --> 9e Libtiff 4.3.0 --> 4.4.0 OpenBLAS 0.3.18 --> 0.3.21 PLplot n/a --> 5.15.0 Python 3.10.0 --> 3.10.6 SCAMP 2.6.7 --> 2.10.0 SWarp 2.38.0 --> 2.41.5 Util-Linux 2.37.2 --> 2.38.1 Vim 8.2 --> 9.0 WCSLIB 7.7 --> 7.11 X.org packages (used by graphical software like Ghostscript and LaTeX): Fontconfig 2.13.94 --> 2.14.0 LibX11 1.7.2 --> 1.8 LibXCB 1.14 --> 1.15 XCB-proto 1.14.1 --> 1.15 Xorg-proto 2021.5 --> 2022.1 Python modules: Astropy 5.0 --> 5.1 GalSim 2.3.3 --> 2.3.5 P.S.2: Notable points regarding the software environment: - Two new links from the host's low-level tools are now included in Maneage's build environment: - On GNU/Linux systems, the host's 'ldd' is linked inside the custom environment. This belongs to the GNU C Library (which is not yet installed in Maneage). But helps in checking the linking status of the binaries on GNU/Linux systems. - On macOS: the 'codesign' binary is included, which is used by GNU Emacs on macOS to sign the built executable. - GNU Bison has been moved in basic software (necessary for GNU Binutils). - The Zip and Unzip programs have been moved as high-level software that have to be manually requested when necessary. This is because they are not used by any of the basic software anymore. They were just installed as dependencies of GNU Tar to be close the other compression programs. Also, in the past we would use the original tarballs, and some (for example Numpy) were distributed in Zip format. However, by default, we now use a custom Lzip tarball and don't need Zip or Unzip. This was suggested by Zahra Sharbaf and Raul Infante-Sainz. - Some minor edits in 'reproduce/software/shell/tarball-prepare.sh'. In particular the 'awk' command was effectively just replacing a '_' with '-', so it just uses a simple SED expression instead. - Fixed bug 62700 (https://savannah.nongnu.org/bugs/index.php?62700) by compiling 'xz' with a patched version of the xz source file 'src/liblzma/liblzma.map'. - Astrometry.net doesn't depend on NetPBM any more. NetPBM (and its dependencies) were causing many crashes on macOS and it also a very strange build system that is hard to maintain. Astrometry.net uses it to take images as input. However, it isn't necessary when you provide Astrometry.net with a catalog. Therefore, Raul added some instructions on how to run astrometry from your own custom X-Y catalog. These instruction can be seen on top of the build rule of Astrometry.net in 'reproduce/software/make/high-level.mk'. - h5py has been removed as a dependency of Astropy. It is an optional dependency to write tables into HDF5 format. But since we couldn't get it to build on macOS it has been removed. None of the current Maneage users/developers also use this feature of Astropy! - PLplot is added a new software, but not a default pre-requisite of SCAMP (which can use it to generate figures), because there were many build problems on macOS. Instructions have been added on top of SCAMP on how to add PLplot as a dependency. - With the aim of being able to install Plplot on macOS, we have wrote several lines to fix header problems. However, we didn't succeed. In any case we are leaving these lines in case they are useful in the future. - The '-Wno-nullability-completeness' compiler flag (which is primarily necessary for macOS) is now only added for macOS systems. It was causing many warnings of un-recognized option in GNU/Linux systems. - The 'mkswap' program of Util-Linux has been disabled because it caused crashes on older kernels. Generally, its not necessary for a Maneage project because it needs root permissions to run! - LibXT (of the x.org software) has been added as a dependency of Cairo. - ImageMagick and Lzip were using the host's C++ standard library! But on GNU/Linux we build our own C++ Standard Library with GCC, so with this commit, they properly link with Maneage's C++ standard library. - ImageMagick on macOS couldn't properly link with Maneage's Ghostscript library! This has been fixed using macOS's install_name_tool. - Necessary RAM to build GCC on GNU/Linux systems changed to ~8GB, see https://savannah.nongnu.org/task/?16244#comment12 - Pythran is no longer as prerequisite of Scipy. Until now, Pythran was a prerequisite of Scipy. But we noticed that it is optional and was causing problems on macOS. - The URLs of some of the software have been updated in 'reproduce/software/config/urls.conf'. By default, these are all commented, but they can be useful when searching for new versions or when a project needs custom software that is not (yet) in Maneage. --- .../README.xz-5.2.5_src_liblzma_liblzma.map | 8 ++ .../software/patches/util-linux-2.38.1-macos.patch | 114 ++++++++++++++++++++ .../patches/xz-5.2.5_src_liblzma_liblzma.map | 115 +++++++++++++++++++++ 3 files changed, 237 insertions(+) create mode 100644 reproduce/software/patches/README.xz-5.2.5_src_liblzma_liblzma.map create mode 100644 reproduce/software/patches/util-linux-2.38.1-macos.patch create mode 100644 reproduce/software/patches/xz-5.2.5_src_liblzma_liblzma.map (limited to 'reproduce/software/patches') diff --git a/reproduce/software/patches/README.xz-5.2.5_src_liblzma_liblzma.map b/reproduce/software/patches/README.xz-5.2.5_src_liblzma_liblzma.map new file mode 100644 index 0000000..15e2787 --- /dev/null +++ b/reproduce/software/patches/README.xz-5.2.5_src_liblzma_liblzma.map @@ -0,0 +1,8 @@ +2022-07-14 B Roukema +xz-5.2.5_src_liblzma_liblzma.map is a patched version of xz-5.2.5/src/liblzma/liblzma.map +based on discussion at https://savannah.nongnu.org/bugs/index.php?62700 +and https://github.com/easybuilders/easybuild-easyconfigs/issues/14991 and +https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/bcebb3320ffb63f9804ca8d4d64d1822ec7c9792/easybuild/easyconfigs/x/XZ/XZ-5.2.5_compat-libs.patch + +Since we don't yet have 'patch' in 'basic.mk', this file has to be copied +into place rather than patched. diff --git a/reproduce/software/patches/util-linux-2.38.1-macos.patch b/reproduce/software/patches/util-linux-2.38.1-macos.patch new file mode 100644 index 0000000..d89422d --- /dev/null +++ b/reproduce/software/patches/util-linux-2.38.1-macos.patch @@ -0,0 +1,114 @@ +From 3671d4a878fb58aa953810ecf9af41809317294f Mon Sep 17 00:00:00 2001 +From: Karel Zak +Date: Mon, 4 Apr 2022 13:17:44 +0200 +Subject: [PATCH] build-sys: improve dependences for lib/procfs.c + +* add #ifdefs when use statfs() and include statfs.h or vfs.h + +Addresses: https://github.com/util-linux/util-linux/issues/1634 +Signed-off-by: Karel Zak +--- + configure.ac | 2 ++ + include/statfs_magic.h | 4 +++- + lib/procfs.c | 15 +++++++++++++-- + misc-utils/hardlink.c | 2 +- + 4 files changed, 19 insertions(+), 4 deletions(-) + +diff --git a/configure.ac b/configure.ac +index 3fba95336a5b60b277d90615089093901fec723e..11c86b9a70809cbade98539b17fb2e1186984ebb 100644 +--- a/configure.ac ++++ b/configure.ac +@@ -346,6 +346,7 @@ AC_CHECK_HEADERS([ \ + sys/socket.h \ + sys/sockio.h \ + sys/stat.h \ ++ sys/statfs.h \ + sys/swap.h \ + sys/syscall.h \ + sys/sysmacros.h \ +@@ -355,6 +356,7 @@ AC_CHECK_HEADERS([ \ + sys/types.h \ + sys/ucred.h \ + sys/un.h \ ++ sys/vfs.h \ + sys/xattr.h \ + unistd.h \ + utmp.h \ +diff --git a/include/statfs_magic.h b/include/statfs_magic.h +index b6b0225e865aeb3c04610ecab177f3e95f1cc61a..67ad0af2595862795299f0bcdb7baa5778742185 100644 +--- a/include/statfs_magic.h ++++ b/include/statfs_magic.h +@@ -1,7 +1,9 @@ + #ifndef UTIL_LINUX_STATFS_MAGIC_H + #define UTIL_LINUX_STATFS_MAGIC_H + +-#include ++#ifdef HAVE_SYS_STATFS_H ++# include ++#endif + + /* + * If possible then don't depend on internal libc __SWORD_TYPE type. +diff --git a/lib/procfs.c b/lib/procfs.c +index 4d6d25b6d78eba56aada4cd9acacee4d2e69f656..0d58857c83b378cb1cb3baf00f88f992ff8dce1e 100644 +--- a/lib/procfs.c ++++ b/lib/procfs.c +@@ -6,9 +6,13 @@ + */ + #include + #include +-#include + #include + ++#ifdef HAVE_SYS_VFS_H ++# include ++# include "statfs_magic.h" ++#endif ++ + #include "c.h" + #include "pathnames.h" + #include "procfs.h" +@@ -16,7 +20,6 @@ + #include "all-io.h" + #include "debug.h" + #include "strutils.h" +-#include "statfs_magic.h" + + static void procfs_process_deinit_path(struct path_cxt *pc); + +@@ -356,6 +359,7 @@ int procfs_dirent_match_name(DIR *procfs, struct dirent *d, const char *name) + return 0; + } + ++#ifdef HAVE_SYS_VFS_H + /* checks if fd is file in a procfs; + * returns 1 if true, 0 if false or couldn't determine */ + int fd_is_procfs(int fd) +@@ -375,7 +379,14 @@ int fd_is_procfs(int fd) + } while (ret != 0); + + return st.f_type == STATFS_PROC_MAGIC; ++ return 0; + } ++#else ++int fd_is_procfs(int fd __attribute__((__unused__))) ++{ ++ return 0; ++} ++#endif + + static char *strdup_procfs_file(pid_t pid, const char *name) + { +diff --git a/misc-utils/hardlink.c b/misc-utils/hardlink.c +index dd55af12aab7903e6025d0a39ea020c2400300e8..08af2882c1359e41fba37377a08c9a18647b0f9f 100644 +--- a/misc-utils/hardlink.c ++++ b/misc-utils/hardlink.c +@@ -38,7 +38,7 @@ + #include /* tolower() */ + #include + +-#if defined(HAVE_LINUX_FIEMAP_H) ++#if defined(HAVE_LINUX_FIEMAP_H) && defined(HAVE_SYS_VFS_H) + # include + # include + # ifdef FICLONE diff --git a/reproduce/software/patches/xz-5.2.5_src_liblzma_liblzma.map b/reproduce/software/patches/xz-5.2.5_src_liblzma_liblzma.map new file mode 100644 index 0000000..8df17a9 --- /dev/null +++ b/reproduce/software/patches/xz-5.2.5_src_liblzma_liblzma.map @@ -0,0 +1,115 @@ +XZ_5.0 { +global: + lzma_alone_decoder; + lzma_alone_encoder; + lzma_auto_decoder; + lzma_block_buffer_bound; + lzma_block_buffer_decode; + lzma_block_buffer_encode; + lzma_block_compressed_size; + lzma_block_decoder; + lzma_block_encoder; + lzma_block_header_decode; + lzma_block_header_encode; + lzma_block_header_size; + lzma_block_total_size; + lzma_block_unpadded_size; + lzma_check_is_supported; + lzma_check_size; + lzma_code; + lzma_crc32; + lzma_crc64; + lzma_easy_buffer_encode; + lzma_easy_decoder_memusage; + lzma_easy_encoder; + lzma_easy_encoder_memusage; + lzma_end; + lzma_filter_decoder_is_supported; + lzma_filter_encoder_is_supported; + lzma_filter_flags_decode; + lzma_filter_flags_encode; + lzma_filter_flags_size; + lzma_filters_copy; + lzma_filters_update; + lzma_get_check; + lzma_index_append; + lzma_index_block_count; + lzma_index_buffer_decode; + lzma_index_buffer_encode; + lzma_index_cat; + lzma_index_checks; + lzma_index_decoder; + lzma_index_dup; + lzma_index_encoder; + lzma_index_end; + lzma_index_file_size; + lzma_index_hash_append; + lzma_index_hash_decode; + lzma_index_hash_end; + lzma_index_hash_init; + lzma_index_hash_size; + lzma_index_init; + lzma_index_iter_init; + lzma_index_iter_locate; + lzma_index_iter_next; + lzma_index_iter_rewind; + lzma_index_memusage; + lzma_index_memused; + lzma_index_size; + lzma_index_stream_count; + lzma_index_stream_flags; + lzma_index_stream_padding; + lzma_index_stream_size; + lzma_index_total_size; + lzma_index_uncompressed_size; + lzma_lzma_preset; + lzma_memlimit_get; + lzma_memlimit_set; + lzma_memusage; + lzma_mf_is_supported; + lzma_mode_is_supported; + lzma_physmem; + lzma_properties_decode; + lzma_properties_encode; + lzma_properties_size; + lzma_raw_buffer_decode; + lzma_raw_buffer_encode; + lzma_raw_decoder; + lzma_raw_decoder_memusage; + lzma_raw_encoder; + lzma_raw_encoder_memusage; + lzma_stream_buffer_bound; + lzma_stream_buffer_decode; + lzma_stream_buffer_encode; + lzma_stream_decoder; + lzma_stream_encoder; + lzma_stream_flags_compare; + lzma_stream_footer_decode; + lzma_stream_footer_encode; + lzma_stream_header_decode; + lzma_stream_header_encode; + lzma_version_number; + lzma_version_string; + lzma_vli_decode; + lzma_vli_encode; + lzma_vli_size; +}; + +XZ_5.1.2alpha { +global: + lzma_stream_encoder_mt; + lzma_stream_encoder_mt_memusage; +} XZ_5.0; + +XZ_5.2.2 { +global: + lzma_block_uncomp_encode; + lzma_cputhreads; + lzma_get_progress; + +local: + *; +} XZ_5.1.2alpha; + +XZ_5.2 { +} XZ_5.2.2; -- cgit v1.2.1