aboutsummaryrefslogtreecommitdiff
path: root/reproduce/software/make
AgeCommit message (Collapse)AuthorLines
2023-05-23Configuration: portability fixes in Less and WCSLIBHEADmaneageMohammad Akhlaghi-11/+30
SUMMARY: no change is necessary in your project, unless you use the Fortran features of WCSLIB in your project. Until now, there were two compilation failures on recent macOS computers with an M1 CPU: Less would crash because it couldn't find the relevant PCRE (perl-compatible regular expression) libraries and WCSLIB would crash because the LLVM compiler's Fortran features could not be built. With this commit, both issues have been fixed by disabling the relevant feature. Extensive comments have been placed in both places in case your project needs these features, so please see the comments in the relevant part of 'reproduce/software/make/basic.mk' for Less and 'reproduce/software/make/high-level.mk' for WCSLIB. In fact the previous solution (where we would not have Fortran features in WCSLIB on macOS systems was problematic and non-reproducibile (the features of WCSLIB depended on the operating system!). Another minor change was that for macOS, we now directly use the version-string of WCSLIB to fix the internal linking issue there. As a result, WCSLIB is no longer a "Version-dependent build" software (in 'reproduce/software/config/versions.conf'). Recall that these are software that when changing the version, it is also necessary to inspect their build recipe. These two issues and their fix were discovered and fixed with the help of James Robinson.
2023-05-07Copyright years: updated to 2023, accompanied by some minor fixesMohammad Akhlaghi-18/+20
SUMMARY: just house-cleaning, no need to do anything major in your branch. Just update the copyright years in files that you have added. Until now, the latest copyright years of the whole Maneage source code was 2022! As of this commit, we have already moved to 2023 for 5 months! Furthermore, there were a few other minor issues that needed correction: - The URL to download input datasets wasn't quoted in 'initialize.mk' or the download script! As a result, when the input URL had characters that are meaningful to the shell (like '&'), the download command would not work. - The only program that had 'make check' in the 'basic.mk' programs was MPFR. At that stage, we still haven't built our own compiler at this stage, this is not accurate. - The 'pyerfa' and 'extension-helpers' packages in Python need 'setuptools_scm' on some systems. But until now, it was not in the list of their prerequisites. With this commit, all the issues above have been corrected.
2022-08-13IMPORTANT: Software updateMohammad Akhlaghi-98/+362
SUMMARY: This is a software update to make Maneage more portable and up to date. It does not involve any Maneage infrastructure changes. You should just re-build your project to make sure the updated software haven't removed/changed any of their features that you were using. In particular, for Astrometry.net users, please see the respective note in P.S.2 below. Until now, there have been many updates in the software that are built within Maneage. The last software upadte was almost one year ago. With this commit, the software in the P.S.1 have been updated. A description of notable changes in the software environment is given in P.S.2. This software environment has been tested on an Arch GNU/Linux, Debian, CentOS-7 and macOS. This commit is the merging of 24 individual commits by Raul Infante-Sainz (who put a lot of energy on porting the software below for macOS, and updating citations), Boudewijn Roukema (who helped with memory checking for GCC, and testing on Debian and CentOS), Sepideh Eskandarlou (who tested the environment) and myself. Besides the updates in the core software, the followimg improvements have also been implemented in this commit: - When you run './project shell': - A welcome message is printed that will remind the caller that they have entered a new environment, it will print the location of 'HOME' and the location of the shell startup file. - The 'reproduce/software/shell/bashrc.sh' is loaded as a startup file. This allows you to customize your interactive Maneage shell. A default step has already been placed there that will put the git branch name (in green) within the shell prompt (which was purple). This greatly helps when dealing with directories under Git version control. These settings won't bother with Maneage's default operations: through environment variables we make sure that these './project shell' features will not slow-down the calls to the shell within the non-interactive Make calls. - The host's 'COLORTERM' is passed to the Maneage environment. It is used by some programs that can have color outputs on the terminal. - Updates to citations: - Numpy and Scipy (as requested on their pages): https://numpy.org/citing-numpy and https://scipy.org/citing-scipy - Gnuastro: Added https://arxiv.org/abs/1909.11230 which describes major updates to Gnuastro after 10 releases. - When a software's paper is indexed in the SAO/NASA Astrophysics Data System (ADS), Maneage now use the BibTeX entries provided by ADS. This helps to give a unified format to most software, and more information (like ADS+arXiv hyperlinks in the BibLaTeX compilation of the default bibliography). - We were able to build this version of Maneage on a Debian system from 2010 (+12 years ago!). Only three downgrades were necessary in the "basic" software (not affecting the high-level science software!). A description of the necessary downgrades for such old systems has been added in 'README.md'. P.S.1 List of updated software: Basic software: cURL 7.79.1 --> 7.84.0 Dash 0.5.11.5 --> 0.5.11-057cd65 File 5.41 --> 5.42 GNU AWK 5.1.0 --> 5.1.1 GNU Bash 5.1.8 --> 5.2-rc2 GNU Binutils 2.37 --> 2.39 GNU Compiler Collection (GCC) 11.2.1 --> 12.1.0 GNU Findutils 4.8.0 --> 4.9.0 GNU Gzip 1.11 --> 1.12 GNU Help2man 1.48.5 --> 1.49.2 GNU Integer Set Library (ISL) 0.18 --> 0.24 GNU Libtool 2.4.6 --> 2.4.7 GNU Nano 6.0 --> 6.4 GNU Readline 8.1.1 --> 8.2-rc2 GNU libiconv 0.16 --> 0.17 Git 2.36.0 --> 2.37.1 OpenSSL 3.0.0 --> 3.0.5 PatchELF 0.13 --> 0.15.0 Perl 5.34.0 --> 5.36.0 High-level software: Astrometry.net 0.89 --> 0.91 CFITSIO 4.0.0 --> 4.1.0 CMake 3.21.4 --> 3.24.0 GNU Astronomy Utilities (Gnuastro) 0.16.1 --> 0.18 GPL Ghostscript 9.55.0 --> 9.56.1 HDF5 1.10.5 --> 1.13.1 Libjpeg 9d --> 9e Libtiff 4.3.0 --> 4.4.0 OpenBLAS 0.3.18 --> 0.3.21 PLplot n/a --> 5.15.0 Python 3.10.0 --> 3.10.6 SCAMP 2.6.7 --> 2.10.0 SWarp 2.38.0 --> 2.41.5 Util-Linux 2.37.2 --> 2.38.1 Vim 8.2 --> 9.0 WCSLIB 7.7 --> 7.11 X.org packages (used by graphical software like Ghostscript and LaTeX): Fontconfig 2.13.94 --> 2.14.0 LibX11 1.7.2 --> 1.8 LibXCB 1.14 --> 1.15 XCB-proto 1.14.1 --> 1.15 Xorg-proto 2021.5 --> 2022.1 Python modules: Astropy 5.0 --> 5.1 GalSim 2.3.3 --> 2.3.5 P.S.2: Notable points regarding the software environment: - Two new links from the host's low-level tools are now included in Maneage's build environment: - On GNU/Linux systems, the host's 'ldd' is linked inside the custom environment. This belongs to the GNU C Library (which is not yet installed in Maneage). But helps in checking the linking status of the binaries on GNU/Linux systems. - On macOS: the 'codesign' binary is included, which is used by GNU Emacs on macOS to sign the built executable. - GNU Bison has been moved in basic software (necessary for GNU Binutils). - The Zip and Unzip programs have been moved as high-level software that have to be manually requested when necessary. This is because they are not used by any of the basic software anymore. They were just installed as dependencies of GNU Tar to be close the other compression programs. Also, in the past we would use the original tarballs, and some (for example Numpy) were distributed in Zip format. However, by default, we now use a custom Lzip tarball and don't need Zip or Unzip. This was suggested by Zahra Sharbaf and Raul Infante-Sainz. - Some minor edits in 'reproduce/software/shell/tarball-prepare.sh'. In particular the 'awk' command was effectively just replacing a '_' with '-', so it just uses a simple SED expression instead. - Fixed bug 62700 (https://savannah.nongnu.org/bugs/index.php?62700) by compiling 'xz' with a patched version of the xz source file 'src/liblzma/liblzma.map'. - Astrometry.net doesn't depend on NetPBM any more. NetPBM (and its dependencies) were causing many crashes on macOS and it also a very strange build system that is hard to maintain. Astrometry.net uses it to take images as input. However, it isn't necessary when you provide Astrometry.net with a catalog. Therefore, Raul added some instructions on how to run astrometry from your own custom X-Y catalog. These instruction can be seen on top of the build rule of Astrometry.net in 'reproduce/software/make/high-level.mk'. - h5py has been removed as a dependency of Astropy. It is an optional dependency to write tables into HDF5 format. But since we couldn't get it to build on macOS it has been removed. None of the current Maneage users/developers also use this feature of Astropy! - PLplot is added a new software, but not a default pre-requisite of SCAMP (which can use it to generate figures), because there were many build problems on macOS. Instructions have been added on top of SCAMP on how to add PLplot as a dependency. - With the aim of being able to install Plplot on macOS, we have wrote several lines to fix header problems. However, we didn't succeed. In any case we are leaving these lines in case they are useful in the future. - The '-Wno-nullability-completeness' compiler flag (which is primarily necessary for macOS) is now only added for macOS systems. It was causing many warnings of un-recognized option in GNU/Linux systems. - The 'mkswap' program of Util-Linux has been disabled because it caused crashes on older kernels. Generally, its not necessary for a Maneage project because it needs root permissions to run! - LibXT (of the x.org software) has been added as a dependency of Cairo. - ImageMagick and Lzip were using the host's C++ standard library! But on GNU/Linux we build our own C++ Standard Library with GCC, so with this commit, they properly link with Maneage's C++ standard library. - ImageMagick on macOS couldn't properly link with Maneage's Ghostscript library! This has been fixed using macOS's install_name_tool. - Necessary RAM to build GCC on GNU/Linux systems changed to ~8GB, see https://savannah.nongnu.org/task/?16244#comment12 - Pythran is no longer as prerequisite of Scipy. Until now, Pythran was a prerequisite of Scipy. But we noticed that it is optional and was causing problems on macOS. - The URLs of some of the software have been updated in 'reproduce/software/config/urls.conf'. By default, these are all commented, but they can be useful when searching for new versions or when a project needs custom software that is not (yet) in Maneage.
2022-07-13Configuration: astropy added as a dependency of astrometry.netMohammad Akhlaghi-0/+1
Until now, astrometry.net didn't explicitly depend on Astropy! However, astrometry.net does depend on it (or pyfits) and will crash later when running if astropy or pyfits aren't installed. We hadn't noticed this until now because of the data reduction projects we had built Astropy independently of astrometry.net! We noticed this bug in a project that didn't use astropy! With this commit, astropy is built as a dependency of astrometry.net and afterwards (during the analysis), astrometry.net was able to run without any crash. This bug was found with the help of Zohreh Ghaffari.
2022-06-11Configuration: replacing hard coded PATH in SConsRaul Infante-Sainz-2/+35
Until now, SCons (a high-level Python package builder) was using the OS PATH when building packages (like Imfit that use SCons), not Maneage's PATH. This happened even though 'reproduce/software/make/high-level.mk' completely removes the host's PATH to avoid any host OS dependency. After some investigation, we recognized that SCons hard-codes operating system directories into its source! This doesn't let the user (Maneage in this case; that builds packages that use SCons) customize the search directories. As a result, even though we have our own linker and compiler in Maneage, SCons would go and use the operating system's linker and compiler, causing a leak in the controlled environment we plan to achieve in Maneage. Not letting users customize such critical components of a software and hard-coding parameters is bad program design! This wasn't noticed until now because most operating systems we tested on were relatively recent and the versions of Maneage's linker and the OS linker weren't too different! However, after testing on a much older operating system (GNU/Linux 4.4.0-143-generic X86_64), the operating system's linker couldn't build Imfit (that uses SCons) and would crash. With this commit, after unpacking SCons's source (but before building or installing it), we have added a step to modify SCons's source and replace the hard-coded PATH directories with Maneage's PATH. This fixed the problem. This bug has been fixed with the help of Mohammad Akhlaghi.
2022-06-11IMPORTANT: download.mk removed, content moved to initialize.mkMohammad Akhlaghi-3/+3
SUMMARY: no special action should be necessary; but its an important update in low-level Maneage infra-structure (related with downloading and setting input checksums). Until now, we had a separate 'download.mk' as one of the default sub-Makefiles that should have been loaded in all the 'top-*.mk' files after 'initialize.mk'. This was due to historic reasons: until Commit 91799fe4b6d, we had to manually make some changes in 'download.mk' for every input file we defined in 'INPUTS.mk' (which was very inconvenient, and not easily possible for a large number of files!). But since Commit 91799fe4b6d, those manual changes are no longer necessary, and a normal user will hardly ever need to touch the contents of 'download.mk' (which also had one effective rule). Furthermore, based on shared projects with Zohre Ghaffari and Sepideh Eskandarlou (which involved a large number of large files), we recognized that it is very inconvenient to download a file once, update its checksum, and re-run Maneage (so the validation works). A robust solution was necesary to let project authors download the data and automatically update the checksum. With this commit, to help in high-level project management in Maneage, the single, and generic rule of 'download.mk' has been moved to 'initialize.mk', enabling us to fully remove this extra sub-Makefile from Maneage's source. Furthermore, with this commit, a usable solution to the automatic updating of the checksum has also been implemented (which has been described in the comments of 'INPUTS.conf'): the users can now set the checksum to '--auto-replace--'. In this case, the download rule (now in 'initialize.mk') will automatically update that line of 'INPUTS.conf' and add the checksum instead. After './project make' is complete, when the user runs 'git diff', they can see all the updated checksums in the source of their project and commit the updated 'INPUTS.conf' into the source so this will not be necessary later. Two other smaller issues have also been addressed in this commit: - There was an extra ',' in the call to 'filter-out' when we defined 'prepare-dep' in 'reproduce/analysis/make/prepare.mk'. This would cause a crash (with Make complaining that there is no rule for target 'initialize.mk,': notice the extra ','). With this commit, that extra ',' has been removed and the problem was solved. - The build recipe of Imfit (in 'reproduce/software/make/high-level.mk'), had two SPACE characters after '--no-openmp' which would make the reading hard. They have been updated to one SPACE.
2022-06-10Configuration: num Apple M1 cores, ImageMagick dep. on GhostScriptRaul Infante-Sainz-2/+4
Until now, we never had the opportunity of testing Maneage in a macOS laptop with an Apple M1 CPU (tested on macOS Monterey; version 12.3). The way of getting the number of cores for this type of CPU is different from other macOSs. It was therefore necessary to change the parameters of the 'sysctl' for properly accounting this CPU. Furthermore, until now, GhostScript and ImageMagick were built independently. However, they were not linked. As a consequence, when trying to obtain an image with the program 'convert' (that belongs to ImageMagick'), it complains about not having some fonts. This can be fixed by letting 'ImakeMagick' know that 'GhostScript' libraries are available. With this commit, GhostScript has been set as a dependency of ImageMagick, and ImageMagick is configured with the '--with-gslib' flag. Furthermore, to read the number of M1 CPU cores, we distinguish between the Apple M1 and all other CPU types. However, Maneage still does not successfully build all the software until the end of the configure step. There are other problems that need to be fixed for Apple's M1.
2022-06-10Housekeeping: some portability issues fixed; four software updatesMohammad Akhlaghi-0/+32
Until now, there were several portability issues in Maneage: 1. Maneage would crash on older operating systems (checked on Debian 6), where Wget didn't have the '--no-use-server-timestamps'. 2. On a Linux kernel 2.6.32 (of the same Debian 6 above) some features in 'util-linux' (like 'swapon' or 'libmount') wouldn't build and wouldn't let 'util-linux' complete. These features need root permissions to be useful, so the wouldn't be used in Maneage any way! But they wouldn't let Maneage get built 3. The './project shell' command would still read the host's '~/.bashrc', letting the host environment leak-in to Maneage's interactive shell. 4. The building of Flex 2.64 wouldn't complete due to a segmentation fault an Ubuntu, but NetPBM (which depends on Flex) would crash with a wrong usage of 'yyunput'. This had actually caused a non-update to Flex in a previous Maneage software update. 5. The update Astrometry.net would assume SExtractor's executable name is 'source-extractor'; causing a crash in usage. This forced the users to manually create a 'source-extractor' symbolic link in the '.local/bin' directory. 6. The 'reproduce/software/shell/tarball-prepare.sh' script (that is used for making Maneage-standard tarballs) wouldn't accept option values with an '=' between the option name and value! It also didnt' print sufficiently informative messages and errors (for example it would say "skipping ..." (making the user think there is a problem!), but it was actually that the file already existed! 7. The 'reproduce/analysis/make/prepare.mk' and 'reproduce/analysis/make/verify.mk' Makefiles that needed to reject some of the 'makesrc' sub-Makefiles would simply substitute their names with nothing. But this would cause problems when the name is part of the name of another sub-Makefile. 8. On the Debian 6 system mentioned above the raw 'df' command's output wasn't in the expected format; so Maneage would fail to properly detect the free space in the disk. With these commit, all the issues above have been solved: for 1, A check has been added to avoid using that option. For 2, those 'util-linux' features have been disabled. For 3, the '--norc' and '--noprofile' options have beed added to the call to Bash. For 4, see below. For 5, the symbolic link is now automatically made with SExtractor. For 6, the option reading components of that script have been fully re-written and more robust sanity checks are also added, with more informative warnings. For 7, the 'subst' function of Make was replaced with 'filter-out' and this fixed the problem. For 8, 'df' is called with the '-P' option so it has a unified format in all versions. For 4, the versions of 'flex' and 'netpbm' have been updated. Since they were the dependency of 'astrometrynet', that has also been updated. In the process, we discovered that 'lzip' has a new version which claims to be faster, so that is also updated. lzip 1.22 --> 1.23 astrometrynet 0.85 --> 0.89 flex 2.6.4 --> 2.6.4-410-74a89fd netpbm 10.73.39 --> 10.73.39 NetPBM needed some manual manipulation in its source (to remove the extra line), so the necessary steps have been added to its build recipe in 'reproduce/software/make/high-level.mk'.
2022-05-15Configuration: necessary compiler flag added for file in CentOS 7Manuel Solimano-0/+1
Until now, Maneage failed to build the 'file' program on at least one machine running CentOS version 7.9.2009 (with Linux kernel 3.10.0, and GCC 4.8.5) when running './project configure'. With this commit (as suggested by the error message issued by the compiler), the '-std=c99' is passed to the compiler in the 'file' recipe (within 'basic.mk'). This flag puts the compiler in C99 mode, which forces it to compile code according to the 1999 edition of the C standard. This was necessary for older versions of GCC (for example GCC 4.8.5 was released in June 2015); hence why others hadn't reported this issue until now. After this fix, File compiles succesfully on such systems; without causing any problem with newer GCC versions (tested in GCC 12.1.0). This issue was solved with the help of Pedram Ashofte Ardakani and Mohammad Akhlaghi.
2022-05-10initialize.mk: Git call in variable works with LD_LIBRARY_PATHMohammad Akhlaghi-5/+5
Until now, the '$(project-commit-hash)' Make variable of 'initialize.mk' simply called 'git' to find the commit hash. However, due to one of the recent software updates, we noticed that this command is no longer working (and the project commit hash wasn't getting printed in the PDF)! The problem was that Maneage's Git, couldn't find the 'libiconv' library that it was built with. With this commit, the '$(shell' command that calls Git, first exports 'LD_LIBRARY_PATH' to Maneage's software build directory. As a result, the Git command can work and will report the commit as a LaTeX macro to be used in the paper. To avoid relying on PATH outside of Make recipes, we now also directly call the Git executable with Maneage. Some other minor issues have been found and fixed in this commit: - README-hacking.md: some minor edits and typo corrections. - initialize.mk: the '$(curdir)' variable is now used in several places that we were calling 'pwd'. - versions.conf: 'xlsxio-version' now included with other programs. Until now it was commented because GCC 11.1.0 had issues with it. However, GCC 11.2.0 doesn't have a problem any more, so it has been returned to the list of all high-level programs. - xorg.mk: used same format to comment recipe lines as the other Makefiles (a '#' followed by a TAB). - preamble-pgfplots.tex: lines to comment for building an EPS figure with PGFPlots have been re-formatted to be more human-readable.
2022-04-20Updated Git, Coreutils and Emacs, new script to prepare tarballsPedram Ashofteh Ardakani-1/+1
Until now, one had to follow the instructions from [1] to prepare a standard software tarball before merging with the low-level tarballs-software repository [2]. The script only worked for '.tar.gz' suffix and was only available as a comment on Savannah (in [1]). With this commit, the script has been imported into Maneage as 'reproduce/software/shell/tarball-prepare.sh' to simplify future software updates. It work with all supported '.tar.*' suffixes (of the upstream tarball repository) and will convert the tarballs to Maneage's standard format. Also, this script has a minimal argument parser and can skip the tarballs that are already unpacked, allowing faster tests. This script was used to update the versions of: Coreutiles 9.0 --> 9.1 Git 2.34 --> 2.36 Emacs 27.2 --> 28.1 The main motive behind this update was Git which announced a vulnerability issue [3] and suggested an update to the latest version as soon as possible. More detail is described in this github blog [4], but in summary, it was a security issue on multi-user systems that has been found and fixed by Git developers. Since Maneage is often installed on such shared systems, it was important to make this update. GNU Coreutils and GNU Emacs were also updated because they are also commonly used. The following improvements have also done with this commit: - .gitignore: ignore emacs auto-save files (that end with a '#') - README-hacking.md: In the checklist for updating the Maneage branch, the no-longer-necessary '--decorate' option of Git was removed from the command to check the general branch history. [1] https://savannah.nongnu.org/task/?15699 [2] https://git.maneage.org/tarballs-software.git/ [3] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/ [4] https://github.blog/2022-04-12-git-security-vulnerability-announced/
2022-01-21IMPORTANT: Updates to almost all softwareMohammad Akhlaghi-979/+1598
This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2
2021-10-01Configuration: GCC not linking to system libunwind (crashed GCC's build)Boud Roukema-1/+10
This commit provides a hack/correction to the unwrapped GCC source files that sym-links the generic file 'libgcc/unwind-generic.h' to the two directories in which a file includes "unwind.h" or <unwind.h>. The aim is that the gcc compilation system uses this header file from the internal gcc source files instead of searching for a system-level file 'unwind.h'. This commit also unaliases two 'ls' commands in some build recipes of 'basic.mk' in case the host system (normally at user level) has aliased the command to something like 'ls -F'. In the situation that sometimes occurs of library files being given executable status, the '-F' decorative option could lead to an asterisk being included in a string that is not expected to contain asterisks. If the system shell does not contain the 'alias' command at all, then a fallback of 'true' should provide safe behaviour. The notation of the 'sed' command is also clarified. This solves bug #61240: https://savannah.nongnu.org/bugs/index.php?61240
2021-07-12Configuration: fixed bugs in building of OpenSSL and GettextMohammad Akhlaghi-4/+5
Until now, the 'RPATH' variable (specifying where to look for shared libraries) wasn't being set in the 'libcrypto' library of OpenSSL (it was only set for the 'libssl' library). Also, Gettext used the host Emacs for some operations during installation that could cause the following crash (because we are giving priority to local libraries, which the host Emacs doesn't recognize): emacs: /BDIR/libcrypto.so.1.1: version `OPENSSL_1_1_1b' not found (required by /lib64/libk5crypto.so.3) With this commit both these bugs have been fixed: 1) Patchelf is run on the 'libcrypto' library also and 2) we pass the '--without-emacs' configuration option to the configure script of Gettext. These bugs were found by Elham Saremi.
2021-03-24Maneage installation: removed TCL as a dependency of SWIGMohammad Akhlaghi-1/+2
Until now the SWIG software would use the host operating system's packages to find the TCL configuraiton (which we don't install yet in Maneage). In particular, you can see the error during its configuration here: .... checking for pkg-config... pkg-config checking for Tcl configuration... found /usr/lib/tclConfig.sh /usr/lib/tclConfig.sh: line 2: dpkg-architecture: command not found /usr/lib//tcl8.6/tclConfig.sh: line 2: dpkg-architecture: com. not found With this commit, TCL has been disabled when building SWIG with the '--without-tcl' option. Later, when we add TCL in Maneage, we can remove this option.
2021-03-20Configuration: nullability-completeness warnings suppressedRaul Infante-Sainz-2/+3
With a recent update of macOS systems (macOS Big Sur 11.2.3 and Xcode 12.4), there are many warnings when building C programs (for example the simple program we compile to check the compiler, or some of the software like `gzip'). It prints hundreds of warning lines for every source file that are irrelevant for our builds, but really clutters the output. With this commit, these warnings are disabled by adding `-Wno-nullability-completeness' to the 'CPPFLAGS' environment variable. This has also been added to the very first check of the C compiler in the configure step.
2021-02-12Installation: minor correction in links to system librariesMohammad Akhlaghi-2/+6
Until now when making a link to the system's 'dl' and 'pthread' libraries we were simply linking the installed location on the system (in '/usr/lib'). However, in some systems, these may themselves be links to other locations and this could cause linking problems. With this commit, we now use 'realpath' to extract the absolute address of the final file that the libraries may link to, and directly link to them. A minor cosmetic correction was also made in the build rule for CFITSIO: the long line was broken into two!
2021-01-09IMPORTANT: analysis outputs written in BDIR/analysisMohammad Akhlaghi-2/+2
Until now, the build directory contained a 'software/' directory (that hosted all the built software), a 'tex/' subdirectory for the final building of the paper, and many other directories containing intermediate/final data of the specific project. But this mixing of built software and data is against our modularity and minimal complexity principles: built software and built data are separate things and keeping them separate will enable many optimizations. With this commit, the build directory of the core Maneage branch will only contain two sub-directories: 'software/' and 'analysis/'. The 'software/' directory has the same contents as before and is not touched in this commit. However, the 'analysis/' directory is new and everything created in the './project make' phase of the project will be created inside of this directory. To facilitate easy access to these top-level built directories, two new variables are defined at the top of 'initialize.mk': 'badir', which is short for "built-analysis directory" and 'bsdir', which is short for "built-software directory". HOW TO IMPLEMENT THIS CHANGE IN YOUR PROJECT. It is easy: simply replace all occurances of '$(BDIR)' in your project's subMakefiles (except the ones below) to '$(badir)'. To confirm if everything is fine before building your project from scratch after merging, you can run the following command to see where 'BDIR' is used and confirm the only remaning cases. $ grep -r BDIR reproduce/analysis/* --> make/verify.mk: innobdir=$$(echo $$infile | sed -e's|$(BDIR)/||g'); \ --> make/initialize.mk:badir=$(BDIR)/analysis --> make/initialize.mk:bsdir=$(BDIR)/software --> make/initialize.mk: $$sys_rm -rf $(BDIR) --> make/top-prepare.mk:all: $(BDIR)/software/preparation-done.mk 'BDIR' should only be present in lines of the files above. If you see '$(BDIR)' used anywhere else, simply change it to '$(badir)'. Ofcourse, if your project assumes BDIR in other contexts, feel free to keep it, it will not conflict. If anything un-expected happens, please post a comment on the link below (you need to be registered on Savannah to post a comment): https://savannah.nongnu.org/task/?15855 One consequence of this change is that the 'analysis/' subdirectory can be optionally mounted on a separate partition. The need for this actually came up for some new users of Maneage in a Docker image. Docker can fix portability problems on systems that we haven't yet supported (even Windows!), or had a chance to fix low-level issues on. However, Docker doesn't have a GUI interface. So to see the built PDF or intermediate data, it was necessary to copy the built data to the host system after every change, which is annoying during working on a project. It would also need two copies of the source: one in the host, one in the container. All these frustrations can be fixed with this new feature. To describe this scenario, README.md now has a new section titled "Only software environment in the Docker image". It explains step-by-step how you can make a Docker image to only host the built software environment. While your project's source, software tarballs and 'BDIR/analysis' directories are on your host operating system. It has been tested before this commit and works very nicely.
2021-01-05Configuration: GNU Binutils linking bug on some systems fixedMohammad Akhlaghi-1/+1
Until now, when building GNU Binutils on GNU Linux operating systems, we would simply put a link to the host's core C library components (the '*crt*' files). However, the symbolic link wasn't "forced"! So if it already existed in the build directory, it would crash. With this commit a '-f' option has been added to the 'ln' command and this fixed the problem. This bug was reported by Zahra Sharbaf.
2021-01-04Building of Less program now uses patchelf to ensure good linkingMohammad Akhlaghi-0/+3
After correctly setting Less to depend on 'ncurses', I noticed its still not linking to Maneage's 'ncurses', but pointing to my host system's 'ncurses' (that happens to have the same version! So it would crash on a system with a different version). This shows that like some other software, we need to manually correct the RPATH inside Less. With this command, the necessary call to 'patchelf' has been added and with it, the installed 'less' command properly linked to Maneage's internal build of 'ncurses'.
2021-01-04Building of less software depends on ncursesMohammad Akhlaghi-1/+1
Until now, the 'less' software package (used to view large files easily on the command-line and used by Git for things like 'git diff' or 'git log') only depended on 'patchelf' (which is a very low-level software). However, as Boud reported in bug #59811 [1], building less would crash with an error saying "Cannot find terminal libraries" in some systems (including the proposed Docker image of 'README.md' which I confirmed afterwards). Looking into the 'configure' script of 'less', I noticed that 'less' is actually just checking for some functions provided by the ncurses library! With this commit, 'less' depends on 'ncurses'. I was able to confirm that with this change, 'less' successfully builds within the Docker image. [1] https://savannah.nongnu.org/bugs/?59811
2021-01-02Copyright year updated in all source filesMohammad Akhlaghi-13/+13
Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.
2020-12-09Configuration: not settting C_INCLUDE_PATH on macOSRaul Infante-Sainz-2/+9
Until now, when building the high-level (optional) software, we would give both 'CPPFLAGS' and 'C_INCLUDE_PATH' the same value/directory in 'high-level.mk'. But we recently found that on macOS's C compiler ('clang'), if a directory is included in both 'CPPFLAGS' and 'C_INCLUDE_PATH', then that directory is ignored in 'CPPFLAGS' (which has higher priority). This caused linking problems when the version of a software on the host was different from the Maneage version. With this commit, 'C_INCLUDE_PATH' is not set on macOS any more and this fixed the problem on the reported systems. This bug was fixed with the help of Mohammad Akhlaghi and Mahdieh Navabi.
2020-12-02Less is now built as a basic softwareMohammad Akhlaghi-0/+10
Less is rarely used in non-interactive mode and is primarily intended for interactively viewing large files. So its need within Maneage (for batch processing) wasn't often felt until now. However, when running './project shell' (which completely closes-off the outside environment), or building a Maneage'd project within a minimal container that doesn't have less, it becomes hard to use Git (and in particular its 'diff' output which depends on 'less'). With this commit, Less has been added as a dependency of Git in 'basic.mk'. In total its built product is roughly 800KB and builds within a second or two. So it isn't a burden on any project. But it can be very useful when the projects are being developed within the Maneage environment itself.
2020-12-01Installation: m4 no longer depends on TexinfoMohammad Akhlaghi-29/+67
In a recent build on a macOS, we recognized that Texinfo needs the 'libintl.h' headers of Gettext. However, Gettext depends on M4, and until now we had set M4 to depend on Texinfo. Therefore adding Gettext as a dependency of Texinfo would cause a circular dependency. On the macOS, we temporarily disabled M4's Texinfo dependency, and the build went through. I also checked on my GNU/Linux system: temporarily renamed all Texinfo built files from my system and done a clean build of M4 and it succeeded. To be further safe, I built Maneage from this commit (where M4 doesn't depend on Texinfo) in a Docker container, and it went through with no problems. So the current M4 version indeed doesn't need Texinfo. I think adding Texinfo as a dependency of M4 was a historic issue from the early days. In the process, I also cleaned 'basic.mk' a little: - A "# Level N" comment was added on top of each group of software that can be built in parallel (generally). - GNU Nano was moved to the end of the file (to be "Level 6"). - Some comments were edited in some places.
2020-09-09R is built without tcl/tk (for GUI) dependenceBoud Roukema-0/+19
Tcl/Tk are a set of tools to provide Graphic User Interface (GUI) support in some software. But they are not yet natively built within Maneage, primarily because we have higher-priority work right now. GUI tools in general aren't high on our priority list right now because GUI tools are generally good for human interaction (which is contrary to the reproducible philosophy), not automatic analysis (a core concept in reproducibility). So even later, when we do include Tcl/Tk in Maneage, their direct usage will be discouraged. Until this commit, because we don't yet build Tcl/Tk, the default maneage install of the statistical package R failed on a Debian Stretch, with 6227 repeats of the line: '/usr/lib//tcl8.5/tclConfig.sh: line 2: dpkg-architecture: command not found' To fix this problem (atleast until Tcl/Tk is installed within Maneage), R is now configured with the '--without-tcltk' option which fixed the problem. Please see the description above the R installation instructions in 'reproduce/software/make/high-level.mk' for more.
2020-08-28Plain text editors: nano in basic, emacs and vim in high-levelMohammad Akhlaghi-6/+64
While a project is under development, the raw analysis software are not the only necessary software in a project. We also need tools to all the edit plain-text files within the Maneaged project. Usually people use their operating system's plain-text editor. However, when working on the project on a new computer, or in a container, the plain-text editors will have different versions, or may not be present at all! This can be very annoying and frustrating! With this commit, Maneage now installs GNU Nano as part of the basic tools. GNU Nano is a very simple and small plain text editor (the installed size is only ~3.5MB, and it is friendly to new users). Therefore, any Maneaged project can assume atleast Nano will be present (in particular when no editor is available on the running system!). GNU Emacs and VIM (both without extra dependencies, in particular without GUI support) are also optionally available in 'high-level.mk' (by adding them to 'TARGETS.conf'). The basic idea for the more advanced editors (Emacs and VIM) is that project authors can add their favorite editor while they are working on the project, but upon publication they can remove them from 'TARGETS.conf'. A few other minor things came up during this work and are now also fixed: - The 'file' program and its libraries like 'libmagic' were linking to system's 'libseccomp'! This dependency then leaked into Nano (which depends on 'libmagic'). But this is just an extra feature of 'file', only for the Linux kernel. Also, we have no dependency on it so far. So 'file' is not configured to not build with 'libseccomp'. - A typo was fixed in the line where the physical core information is being read on macOS. - The top-level directories when running './project shell' are now quoted (in case they have special characters).
2020-08-27Machine architecture and byte-order available as LaTeX macroMohammadreza Khellat-4/+4
Until now, no machine-related specifications were being documented in the workflow. This information can become helpful when observing differences in the outcome of both software and analysis segments of the workflow by others (some software may behave differently based on host machine). With this commit, the host machine's 'hardware class' and 'byte-order' are collected and now available as LaTeX macros for the authors to use in the paper. Currently it is placed in the acknowledgments, right after mentioning the Maneage commit. Furthermore, the project and configuration scripts are now capable of dealing with input directory names that have SPACE (and other special characters) by putting them inside double-quotes. However, having spaces and metacharacters in the address of the build directory could cause build/install failure for some software source files which are beyond the control of Maneage. So we now check the user's given build directory string, and if the string has any '@', '#', '$', '%', '^', '&', '*', '(', ')', '+', ';', and ' ' (SPACE), it will ask the user to provide a different directory.
2020-08-08Software tarballs saved as symlinks if already in filesystemMohammad Akhlaghi-2/+7
Until now, if the software source tarballs already existed on the system they would be copied inside the project. However, the software source tarballs are sometimes/mostly larger than their actual product and can consume significant space (~375 MB in the core branch!). With this commit, when the software are present on the system, their symbolic link will be placed in 'BDIR/software/tarballs', not a full copy. Also, because the tarballs in software tarball directory may themselves be links, we use 'realpath' to find the final place of the actual file and link to that location. Therefore if 'realpath' can't be found (prior to installing Coreutils in Maneage), we will copy the tarballs from the given software tarball directory. After Maneage has installed Coreutils, the project's own 'realpath' will be used. Of course, if the software are downloaded, their full downloaded copy will be kept in 'BDIR/software/tarballs', nothing has changed in the downloading scenario.
2020-08-08IMPORTANT: New software versions (17 basic, 16 high-level and 7 Python)Mohammad Akhlaghi-28/+81
It was a long time that the Maneage software versions hadn't been updated. With this commit, the versions of all basic software were checked and 17 of that had newer versions were updated. Also, 16 high-level programs and libraries were updated as well as 7 Python modules. The full list is available below. Basic Software (affecting all projects) --------------------------------------- bash 5.0.11 -> 5.0.18 binutils 2.32 -> 2.35 coreutils 8.31 -> 8.32 curl 7.65.3 -> 7.71.1 file 5.36 -> 5.39 gawk 5.0.1 -> 5.1.0 gcc 9.2.0 -> 10.2.0 gettext 0.20.2 -> 0.21 git 2.26.2 -> 2.28.0 gmp 6.1.2 -> 6.2.0 grep 3.3 -> 3.4 libbsd 0.9.1 -> 0.10.0 ncurses 6.1 -> 6.2 perl 5.30.0 -> 5.32.0 sed 4.7 -> 4.8 texinfo 6.6 -> 6.7 xz 5.2.4 -> 5.2.5 Custom programs/libraries ------------------------- astrometrynet 0.77 -> 0.80 automake 0.16.1 -> 0.16.2 bison 3.6 -> 3.7 cfitsio 3.47 -> 3.48 cmake 3.17.0 -> 3.18.1 freetype 2.9 -> 2.10.2 gdb 8.3 -> 9.2 ghostscript 9.50 -> 9.52 gnuastro 0.11 -> 0.12 libgit2 0.28.2 -> 1.0.1 libidn 1.35 -> 1.36 openmpi 4.0.1 -> 4.0.4 R 3.6.2 -> 4.0.2 python 3.7.4 -> 3.8.5 wcslib 6.4 -> 7.3 yaml 0.2.2 -> 0.2.5 Python modules -------------- cython 0.29.6 -> 0.29.21 h5py 2.9.0 -> 2.10.0 matplotlib 3.1.1 -> 3.3.0 mpi4py 3.0.2 -> 3.0.3 numpy 1.17.2 -> 1.19.1 pybind11 2.4.3 -> 2.5.0 scipy 1.3.1 -> 1.5.2
2020-08-02OpenMPI build with slurm compatibilityBoud Roukema-1/+4
Prior to this commit, compilation of OpenMPI used the default OpenMPI choices of deciding which libraries should be used in relating to a job scheduler [1] (such as Slurm [2]). Given that the user on a multi-user cluster has to accept the sysadmin's choice of a job scheduler, the question of whether to (1) link with OpenMPI's own libraries (and increase the reproducibility of the science project) or rather (2) link with the sysadmin managed libraries (more likely to be compatible with the host's job scheduler), is an open question of which the best strategy for reproducibility needs to be debated and studied. In this commit, strategy (1) is adopted. The options '--withpmix=internal' and '--with-hwloc=internal' are added to the configure command. The working assumption is that the Maneage version of OpenMPI is likely to be modern enough to be compatible with the native job scheduler such as Slurm. Compilation without any 'pmix' option gave a fail in at least one case; it appears that an external pmix library was sought by the configure script. As of OpenMPI 4.0.1, the internal libevent library is used by default, so there appears to be no option to force it to be chosen internally. This commit also includes the option '--without-verbs'. This option removes a library related to "infiniband", "verbs", "openib" and "BTL"; this library appears to be deprecated. See [3], [4] for discussion. Please add feedback and discussion to the Maneage task about openmpi linking strategies (1) (internal) and (2) (external) at Savannah [5]. [1] https://en.wikipedia.org/wiki/Job_scheduler#Batch_queuing_for_HPC_clusters [2] https://en.wikipedia.org/wiki/Slurm_Workload_Manager - To avoid a name clash, 'slurm-wlm' is the metapackage in Debian for the client commands, the compute node daemon, and the central node daemon. An unrelated package 'slurm' also exists. [3] https://www-lb.open-mpi.org/faq/?category=openfabrics#ofa-device-error [4] https://www-lb.open-mpi.org/faq/?category=building [5] https://savannah.nongnu.org/task/index.php?15737
2020-07-07Fixed typo that lead to crash when building healpyMarius Peper-1/+1
Until now, if a project needed the healpy software package, Maneage would crash with the following error message (abridged for full name in build directory). This was caused by a typo in the version of 'healpix' (the dependency of 'healpy'). make: *** No rule to make target '.../version-info/proglib/healpix-' With this commit, the typo in line 334 of 'python.mk' is fixed, so that when '$(ipydir)/healpy-$(healpy-version)' gets called it correctly searches for a rule to make '$(ibidir)/healpix-$(healpix-version)'.
2020-07-05Configure script prefers clang for macOS systemsMohammad Akhlaghi-0/+1
In the previous commit (Commit 1bc00c9: Only using clang in macOS systems that also have GCC) we set the used C compiler for high-level programs to be 'clang' on macOS systems. But I forgot to do the same kind of change in the configure script (to prefer 'clang' when we are testing for a C compiler on the host). With this commit, the compiler checking phases of the configure script have been improved, so on macOS systems, we now first search for 'clang', then search for 'gcc'. While doing this, I also noticed that the 'rpath' checking command was done before we actually define 'instdir'!!! So in effect, the 'rpath' directory was being set to '/lib'! So with this commit, this test has been taken to after defining 'instdir'.
2020-07-05Only using clang in macOS systems that also have GCCMohammad Akhlaghi-27/+58
Until now, when Maneage was built on a macOS that had both a clang and GCC, we would make links to both. But this cause many conflicts in some high-level programs (for example Numpy and etc, all the programs where we have explicity set 'export CC=clang' before the build recipe). This happens because the GCC that is built on a macOS isn't complete for some operations. To fix this problem, when we are on a macOS, we explicity set 'gcc' to point to 'clang' and 'g++' to point to 'clang++'. We also don't link to the host's C-preprocessor ('cpp') on macOS systems because this is only a GNU feature and using the GNU CPP is also known to have some basic problems. For example this was reported by Mahdieh Nabavi (which was the main trigger for this work): ld: Symbol not found: ___keymgr_global Referenced from: /Users/Mahdieh/build/software/installed/bin/cpp Expected in: /usr/lib/libSystem.B.dylib Also, to avoid linking to another link on the host tools (in the 'makelink' function of 'basic.mk'), we are now using 'realpath'.
2020-07-01Properly accounting for space characters in host's PATHMohammad Akhlaghi-12/+20
Until now, when reading the host's PATH environment variable we weren't accounting for directory names with a space character. This was most prominently visible in the 'low-level-links' step where we put links to some core system components into the project's build directory (mainly for prorietary systems like macOS). To address the problem, double quotations have been placed around the part that we extract 'ccache' from the PATH, and the part where we make the symbolic link. In the process the comments above 'makelink' were made more clear and 'low-level-links' now depends on 'grep' (which is the highest-level program it uses). This bug was reported by Mahdieh Navabi.
2020-07-01Minor typo corrected in referencing LibidnRaul Infante-Sainz-1/+2
Until this commit, once Libidn was installed, insted of its own name and version, the name and version of Libjpeg were saved (in the target if Libidn). This robably come from a copy/paste of the rule. With this commit, this minor bug has been corrected. I also added my name as an author of `reproduce/software/make/xorg.mk' Makefile since I added some code there.
2020-06-30Proper deletion of util-linux source after successfully building itMohammad Akhlaghi-1/+3
After recently adding util-linux to Maneage build-tree, we had forgot to delete the unpacked and built source directory after it was installed! This has been corrected with this commit.
2020-06-30Core Xorg libraries necessary for Ghostscript now includedMohammad Akhlaghi-47/+322
Until now, in order to build Ghostscript, the project used the host's Xorg libraries. This was because we hadn't yet added the necessary build rules for them. With this commit, the instructions to build the necessary Xorg libraries for Ghostscript have also been added. Also, the shared Ghostscript library has been built with this commit and two sets of standard fonts are also included, setting us on the path to build TeXLive from source later. This task was done with the help and support of Raul Infante-Sainz.
2020-06-27IMPORTANT: many improvements to low-level software building phaseMohammad Akhlaghi-2091/+2002
POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause conflicts to your project if you have changed the software building Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and 'python.mk'). If your project has only added analysis, it shouldn't be affected. This is a large commit, involving a long series of corrections in a differnt branch which is now finally being merged into the core Maneage branch. All changes were related and came up naturally as the low-level infrastructure was improved. So separating them in the end for the final merge would have been very time consuming and we are merging them as one commit. In general, the software building Makefiles are now much more easier to read, modify and use, along with several new features that have been added. See below for the full list. - Until now, Maneage needed the host to have a 'make' implementation because Make was necessary to build Lzip. Lzip is then used to uncompress the source of our own GNU Make. However, in the minimalist/slim versions of operating systems (for example used to build Docker images) Make isn't included by default. Since Lzip was the only program before our own GNU Make was installed, we consulting Antonio Diaz Diaz (creator of Lzip) and he kindly added the necessary functionality to a new version of Lzip, which we are using now. Hence we don't need to assume a Make implementation on the host any more. With this commit, Lzip and GNU Make are built without Make, allowing everything else to be safely built with our own custom version of GNU Make and not using the host's 'make' at all. - Until recently (Commit 3d8aa5953c4) GNU Make was built in 'basic.mk'. Therefore 'basic.mk' was written in a way that it can be used with other 'make' implementations also (i.e., important shell commands starting with '&&' and ending in '\' without any comments between them!). Furthermore, to help in style uniformity, the rules in 'high-level.mk' and 'python.mk' also followed a similar structure. But due to the point above, we can now guarantee that GNU Make is used from the very first Makefile, so this hard-to-read structure has been removed in the software build recipes and they are much more readable and edit-friendly now. - Until now, the default backup servers where at some fixed URLs, on our own pages or on Gitlab. But recently we uploaded all the necessary software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is more suitable for this task (it promises longevity, has a fixed DOI, while allowing us to add new content, or new software tarball versions). With this commit, a small script has been written to extract the most recent Zenodo upload link from the Zenodo DOI and use it for downloading the software source codes. - Until now, we primarily used the webpage of each software for downloading its tarball. But this caused many problems: 1) Some of them needed Javascript before the download, 2) Some URLs had a complex dependency on the version number, 3) some servers would be randomly down for maintenance and etc. So thanks to the point above, we now use the Zenodo server as the primary download location. However, if a user wants to use a custom software that is not (yet!) in Zenodo, the download script gives priority to a custom URL that the users can give as Make variables. If that variable is defined, then the script will use that URL before going onto Zenodo. We now have a special place for such URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a good documentation themselves) are preserved here, but are commented by default. - The software source code downloading and checksum verification step has been moved into a Make function called 'import-source' (defined in the 'build-rules.mk' and loaded in all software Makefiles). Having taken all the low-level steps there, I noticed that there is no more need for having the tarball as a separate target! So with this commit, a single rule is the only place that needs to be edited/added (greatly simplifying the software building Makefiles). - Following task #15272, A new option has been added to the './project' script called '--all-highlevel'. When this option is given, the contents of 'TARGETS.conf' are ignored and all the software in Maneage are built (selected by parsing the 'versions.conf' file). This new option was added to confirm the extensive changes made in all the software building recipes and is great for development/testing purposes. - Many of the software hadn't been tested for a long time! So after using the newly added '--all-highlevel', we noticed that some need to be updated. In general, with this commit, 'libpaper' and 'pcre' were added as new software, and the versions of the following software was updated: 'boost', 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in' shell script was added in 'reproduce/software/shell/' which is installed with 'libpaper'. - Even though we intentionally add the necessary flags to add RPATH inside the built executable at compilation time, some software don't do it (different software on different operating systems!). Until now, for historical reasons this check was done in different ways for different software on GNU/Linux sytems. But now it is unified: if 'patchelf' is present we apply it. Because of this, 'patchelf' has been put as a top-level prerequisite, right after Tar and is installed before anything else. - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in 'basic.mk', it was 'glibtool'! This caused many confusions and is corrected with this commit (in 'basic.mk', it is also 'libtool'). - A new argument is added to the './project' script to allow easy loading of the project's shell and environment for fast/temporary testing of things in the same environment as the project. Before activating the project's shell, we completely remove all host environment variables to simulate the project's environment. It can be called with this command: './project shell'. A simple prompt has also been added to highlight that the user is using the Maneage shell!
2020-06-18Fixed small bug that was introduced four commits agoRaul Infante-Sainz-4/+4
In Commit 105467fe6402 (Software tarballs are downloaded even if not built), we introduced tests to download the tarballs of software even if they don't need to be built on the respective host. However some small typos in the checks existed that could cause a crash on macOS. In particular in the building of PatchELF and libbsd we had forgot to add the necessary 'x' before the 'yes' in the conditional to check if a we are on macOS or not. With this commit these two checks have been corrected. Also, in the building of 'isl' and 'mpc', we now check for 'host_cc' (signifying that the user wants to use their host C compiler for the high-level step) instead of 'on_mac_os'. The reason is that even on non-macOS systems, a user may not want to build the C compiler from scratch and use the '--host-cc' option. In such cases, they don't need to compile 'isl' and 'mpc'.
2020-06-17Software tarballs are downloaded even if not builtMohammad Akhlaghi-56/+36
Some low-level software aren't necessary on some operating systems, for example GCC can't be built on macOS, hence we don't build it and the GCC-only dependencies. Also, on GNU/Linux systems users could configure with '--host-cc' to avoid all the time it takes to build GCC when doing a fast test. Until now, in such cases not only was the software not installed, but the tarballs of the software were also not downloaded. Hence making the output of '--dist-software' incomplete (as in bug #58561). With this commit, we now import all the necessary tarballs, when the software isn't necessary for the particular system, it won't be built or cited, but its tarball will be present anyway, thus allowing the output of '--dist-software' to be complete.
2020-06-16XLSX I/O properly accounts for local buildMohammad Akhlaghi-2/+2
Until now, when adding the necessary library flags to the build of XLSX I/O, we were effectively over-writing the 'LDFLAGS' variables. So the compiler was effectively not being told where to look for the necessary libraries. With this commit, to fix the problem, we now append the new linking flags to LDFLAGS in XLSX I/O's build, not over-write it.
2020-06-15OpenSSL now built after PerlMohammad Akhlaghi-1/+1
After trying a clean build of Maneage in a Docker image (with a minimal debian:stable-20200607-slim OS), I noticed that the building of OpenSSL is failing because it doesn't find the proper Perl functionality. To fix it, with this commit, Perl is set as a prerequisite of OpenSSL and this fixed the problem.
2020-06-02Core software build before using Make to build other softwareMohammad Akhlaghi-237/+282
Until now, Maneage would only build Flock before building everything else using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid parallel downloads during the building of software (which could cause network problems). But after recently trying Maneage on FreeBSD (which is not yet complete, see bug #58465), we noticed that the BSD implemenation of Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq' parts) and its shell also had some peculiarities. It was thus decided to also install our own minimalist shell, Make and compressor program before calling 'basic.mk'. In this way, 'basic.mk' can now assume the same GNU Make features that high-level.mk and python.mk assume. The pre-make building of software is now organized in 'reproduce/software/shell/pre-make-build.sh'. Another nice feature of this commit is for macOS users: until now the default macOS Make had problems for parallel building of software, so 'basic.mk' was built in one thread. But now that we can build the core tools with GNU Make on macOS too, it uses all threads. Furthermore, since we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have to finish every line of a long rule with a backslash to keep variables and such. Generally, the pre-make software are now organized like this: first we build Lzip before anything else: it is downloaded as a simple '.tar' file that is not compressed (only ~400kb). Once Lzip is built, the pre-make phase continues with building GNU Make, Dash (a minimalist shell) and Flock. All of their tarballs are in '.tar.lz'. Maneage then enters 'basic.mk' and the first program it builds is GNU Gzip (itself packaged as '.tar.lz'). Once Gzip is built, we build all the other compression software (all downloaded as '.tar.gz'). Afterwards, any compression standard for other software is fine because we have it. In the process, a bug related to using backup servers was found in 'reproduce/analysis/bash/download-multi-try' for calling outside of 'basic.mk' and removed Bash-specific features. As a result of that bug-fix, because we now have multiple servers for software tarballs, the backup servers now have their own configuration file in 'reproduce/software/config/servers-backup.conf'. This makes it much easier to maintain the backup server list across the multiple places that we need it. Some other minor fixes: - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'. - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD (which doesn't have GCC). - We are now using 'uname -s' to specify if we are on a Linux kernel or not, if not, we are still using the old 'on_mac_os' variable. - While I was trying to build on FreeBSD, I noticed some further corrections that could help. For example the 'makelink' Make-function now takes a third argument which can be a different name compared to the actual program (used for examle to make a link to '/usr/bin/cc' from 'gcc'. - Until now we didn't know if the host's Make implementation supports placing a '@' at the start of the recipe (to avoid printing the actual commands to standard output). Especially in the tarball download phase, there are many lines that are printed for each download which was really annoying. We already used '@' in 'high-level.mk' and 'python.mk' before, but now that we also know that 'basic.mk' is called with our custom GNU Make, we can use it at the start for a cleaner stdout. - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a system where we can't install GCC (or has activated the '--host-cc' option), it may not be present and the project shouldn't break because of this. So with this commit, when a Fortran compiler isn't present, WCSLIB will be built with the '--disable-fortran' configuration option. This commit (task #15667) was completed with help/checks by Raul Infante-Sainz and Boud Roukema.
2020-05-28Fixed TeXLive crash because of differing local and server versionsRaul Infante-Sainz-3/+80
Until this commit, when the user had a previous TeXLive tarball already present (in their software-tarball directory) compared to the CTAN server, the project crashed in the configure phase. This was because TeXLive is updated yearly and we don't yet install TeXLive from source (currently we use its own package manager, but we plan to fix this in task #15267). With this commit, we fix the problem by checking the cause of the crash during the installation of TeX. If the crash is due to this particular error, we ignore the old tarball and download the new one and install it (the old one is still kept in '.build/software/tarballs', but will get a '-OLD' in its name. This probem was recurrent, and every year that TeXLive is updated, the previous tarball had to be removed manually! But with this commit, this is done automatically. The detection and fix of this bug has been possible with the help of Mohammad Akhlaghi, thanks!
2020-05-25Unified reference to GNU/Linux and free softwareMohammad Akhlaghi-3/+3
One of the main reasons to building Maneage is to properly acknowledge/attribute the authors of software in research. So we have adopted a standard of never referring to the GNU-based operating systems running the Linux kernel simply as "Linux", we avoid terms like "Open Sourse" and use Free Software instead (in the same spirit). With this commit, a few instances of the cases above have been corrected, they had slipped through our fingers when we initially imported them into the project. In the special case of the "Journal for Open Source Software", we simply replaced it with its abbreviation (JOSS). This was done because in effect we were generally using journal name abbreviations in almost all the citations already. To avoid any inconsistancies, the names of the three other journals that weren't abbreviated are also abbreviated.
2020-05-23New software: Valgrind and PatchBoud Roukema-0/+37
With this commit, Maneage now includes instructions to build the memory tracing tool Valgrind and the program 'patch' (to apply corrections/patches in text files and in particular the sources of programs). For this version of Valgrind, some patches were necessary for an interface with OpenMPI 2.x (which is the case now). Also note that this version of Valgrind's checks can fail with GCC 10.1.x (when using '--host-cc'), and the failures aren't due to internal problems but due to how the tests are designed (https://bugs.gentoo.org/707598). So currently if any of Valgrind's checks fail, Maneage still assumes that Valgrind was built and installed successfully. While testing on macOS, we noticed that it needs the macOS-specific 'mig' program which we can't build in Maneage. DESCRIPTION: The mig command invokes the Mach Interface Generator to generate Remote Procedure Call (RPC) code for client-server style Mach IPC from specification files. So a symbolic link to the system's 'mig' is now added to the project's programs on macOS systems. This commit's build of Patch and Valgrind has been tested on two GNU/Linux distributions (Debian and ArchLinux) as well as macOS. Work on this commit started by Boud Roukema, but also involved tests and corrections by Mohammad Akhlaghi and Raul Infante-Sainz.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-52/+92
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-05-08Installing Scamp with the option --enable-plplot=noRaul Infante-Sainz-1/+3
Until this commit, Scamp was installed with the option `--enable-plplot=yes' (the default). However, Maneage does not have PLplot included. As it is possible to install Scamp without PLplot (in that case it won't generate plots), with this commit this option has been set to `no'. As a consequence, Scamp will be installed even if the host system does not have PLplot without crashing (but it won't make any plot).
2020-05-08GNU Gettext built as a dependency of BashRaul Infante-Sainz-53/+58
Until now Maneage used the host's GNU Gettext if it was present. Gettext is a relatively low-level software that enables programs to print messages in different languages based on the host environment. Even though it has not direct effect on the running of the software for Maneage and the lanugage environment in Maneage is pre-determined, it is necessary to have it because if the basic programs see it in the host they will link with it and will have problems if/when the host's Gettext is updated. With this commit (which is actually a squashed rebase of 9 commits by Raul and Mohammad), Gettext and its two extra dependencies (libxml2 and libunistring) are now installed within Maneage as a basic software and built before GNU Bash. As a result, all programs built afterwards will successfully link with our own internal version of Gettext and libraries. To get this working, some of the basic software dependencies had to updated and re-ordered and it has been tested in both GNU/Linux and macoS. Some other minor issues that are fixed with this commit - Until this commit, when TeX was not installed, the warning message saying how to run the configure step in order to re-configure the project was not showing the option `-e'. However, the use of this option is more convenient than entering the top-build directory and etc every time. So with this commit, the warning message has been changed in order use the option `-e' in the re-configure of the project. - Until now, on macOS systems, Bash was not linking with our internally built `libncurses'. With this commit, this has been fixed by setting `--withcurses=yes' for Bash's configure script.