aboutsummaryrefslogtreecommitdiff
path: root/reproduce
AgeCommit message (Collapse)AuthorLines
2020-06-17New target --dist-software to package all necessary software tarballsMohammad Akhlaghi-6/+2
When publishing a project, it is necessary to also publish the source code of all necessary software of the project. We had recently added a new './project make' target called 'dist-software' for this job, but had forgotten to add it in the output of './project --help'! There was also a small bug inside of it that didn't allow the successful copying of the created tarball to the top project directory. With this commit, an explanation for this target has been added in the output of './project --help' and that bug has been fixed.
2020-06-17Corrected symbolic link to Gnuastro's configuration filesMohammad Akhlaghi-1/+1
Until now, when making the link to Gnuastro's configuration files, the 'configure.sh' script would incorrectly link to the old configuration directory under the 'reproduce/software' directory. With this commit, it is moved to the proper directory under 'reproduce/analysis'.
2020-06-16Imported recent improvements in Maneage, minor conflicts fixedMohammad Akhlaghi-30/+56
Some minor conflicts came up in 'download.mk' and 'verify.mk' (as expected in the "IMPORTANT" commits) and were easily fixed by choosing this branch's values.
2020-06-16XLSX I/O properly accounts for local buildMohammad Akhlaghi-2/+2
Until now, when adding the necessary library flags to the build of XLSX I/O, we were effectively over-writing the 'LDFLAGS' variables. So the compiler was effectively not being told where to look for the necessary libraries. With this commit, to fix the problem, we now append the new linking flags to LDFLAGS in XLSX I/O's build, not over-write it.
2020-06-15OpenSSL now built after PerlMohammad Akhlaghi-1/+1
After trying a clean build of Maneage in a Docker image (with a minimal debian:stable-20200607-slim OS), I noticed that the building of OpenSSL is failing because it doesn't find the proper Perl functionality. To fix it, with this commit, Perl is set as a prerequisite of OpenSSL and this fixed the problem.
2020-06-15Configure script now accounts for non-interactive shellsMohammad Akhlaghi-6/+33
The project configuration requires a build-directory at configuration time, two other directories can optionally be given to avoid downloading the project's necessary data and software. It is possible to give these three directories as command-line options, or by interactively giving them after running the configure script. Until now, when these directories weren't given as command-line options, and the running shell was non-interactive, the configure script would crash on the line trying to interactively read the user's given directories (the 'read' command). With this commit, all the 'read' commands for these three directories are now put within an 'if' statement. Therefore, when 'read' fails (the shell is non-interactive), instead of a quiet crash, a descriptive message is printed, telling the user that cause of the problem, and suggesting a fix. This bug was found by Michael R. Crusoe.
2020-06-14Better description for input data directory, pointing to INPUTS.confMohammad Akhlaghi-19/+13
Until now, the description of the input-data directory at configure time included a description of the input data (created by reading the values of 'INPUTS.conf'). Maintaining this is easy for a single dataset, but it becomes hard for a general project which may need many input datasets. To avoid extra complexity (for maintaining this list), the description now points a user of the project to the 'INPUTS.conf' file and asks them to look inside of it for seeing the necessary data. This infact helps with the users becoming familiar with the internal structure of Maneage and will allow the authors to focus on not having to worry about updating the low-level 'configure.sh' script.
2020-06-14Better explanation in the start of project configurationMohammad Akhlaghi-3/+7
When './project configure' is run, after the basic checks of the compiler, a small statement is printed telling the user that some configuration questions will now be asked to start building Maneage on the system. Until now this description was confusing: it lead the reader to think that the local configuration (which was recommended to read before continuing) is in another file. With this commit, the text has been edited to explictly mention that the description of the steps following this notice should be read carefully. Thus avoiding that confusion. This issue was mentioned by Michael R. Crusoe.
2020-06-10Corrected bug in using local copy of input datasetMohammad Akhlaghi-13/+47
As described in Maneage's commit 2bd2e2f18 (which I found while testing this project), the existing download recipe had problems when using a local copy of the input dataset. It was first fixed here, then implemented there. Also, to clarify things for a new user, some long comments were added at the top of 'INPUTS.conf' to describe each of the variables, that comment has also been put here (and is also in commit 2bd2e2f18 of Maneage).
2020-06-10IMPORTANT: bug fix in default data download script of download.mkMohammad Akhlaghi-14/+54
Summary of possible semantic conflicts 1. The recipe to download input datasets has been modified. You have to re-set the old 'origname' variable to 'localname' (to avoid confusion) and the default dataset URL should now be complete (including the actual filename). See the newly added descriptions in 'INPUTS.conf' for more on this. Until now, when the dataset was already present on the host system, a link couldn't be made to it, causing the project to crash in the checksum phase. This has been fixed with properly naming the main variable as 'localname' to avoid the confusion that caused it. Some other problems have been fixed in this recipe in the meantime: - When the checksum is different, the expected and calculated checksums are printed. - In the default paper, we now print the full URL of the dataset, not just the server, so the checksum of the 'download.tex' step has been updated.
2020-06-09Minor edit printing arXiv URL in plain text metadataMohammad Akhlaghi-1/+1
Until now, in the 'print-copyright' function of 'initialize.mk' (that prints a fixed set of common meta necessary in plain-text files), we were simply printing this line: # Pre-print server: arXiv:1234.56789 But given that all the other elements are click-able URLs, it now prints: # Pre-print server: https://arxiv.org/abs/1234.56789
2020-06-09Two minor corrections to avoid warnings in make and make cleanMohammad Akhlaghi-10/+2
There were two small warnings that are removed with this commit: - In the end, when we print the number of words in the PDF, we hadn't accounted for the fact that 'paper.pdf' doesn't always exist (for example when './project make clean' is run). So a check was added to only print the number of words when a PDF exists. - I noticed that the '$(texdir)/to-publish' directory was being built both in 'initialize.mk' and in 'demo-plot.mk'. So the one in 'demo-plot.mk' has been removed.
2020-06-09Imported Maneage, minor conflicts fixed, a bug found and fixedMohammad Akhlaghi-54/+160
Some minor conflicts came up in 'initialize.mk' and 'verify.mk'. For the former, I chose the version on Maneage, for the latter, I kept the 'master' version on the checksums of this project, but kept the Maneage version for the rest of the improvements there (like printing the verified files as LaTeX comments in 'verify.tex'. While testing the conflicts, I noticed a bug (in the LaTeX macro for the number of years in the Menke+20 paper) in the previous build, thanks to the verification step :-)! Fortunately it wasn't actually printed in the PDF, so a normal reader won't recognize. The bug was caused by the recently added meta-data/commented lines in the 'tools-per-year.txt' file: when calculating the number of years studied in that paper, we were simply counting all the lines and we had forgot to correct this after adding comments. As a result, the un-used LaTeX macro file was saying that they have studied 47 years instead of the real 31 years! This element was actually used in the very first (+40 page!) draft of the paper that was summarized to fit into the journal limits.
2020-06-06IMPORTANT: Added publication checklist, improved relevant infrastructureMohammad Akhlaghi-68/+258
Possible semantic conflicts (that may not show up as Git conflicts but may cause a crash in your project after the merge): 1) The project title (and other basic metadata) should be set in 'reproduce/analysis/conf/metadata.conf'. Please include this file in your merge (if it is ignored because of '.gitattributes'!). 2) Consider importing the changes in 'initialize.mk' and 'verify.mk' (if you have added all analysis Makefiles to the '.gitattributes' file (thus not merging any change in them with your branch). For example with this command: git diff master...maneage -- reproduce/analysis/make/initialize.mk 3) The old 'verify-txt-no-comments-leading-space' function has been replaced by 'verify-txt-no-comments-no-space'. The new function will also remove all white-space characters between the columns (not just white space characters at the start of the line). Thus the resulting check won't involve spacing between columns. A common set of steps are always necessary to prepare a project for publication. Until now, we would simply look at previous submissions and try to follow them, but that was prone to errors and could cause confusion. The internal infrastructure also didn't have some useful features to make good publication possible. Now that the submission of a paper fully devoted to the founding criteria of Maneage is complete (arXiv:2006.03018), it was time to formalize the necessary steps for easier submission of a project using Maneage and implement some low-level features that can make things easier. With this commit a first draft of the publication checklist has been added to 'README-hacking.md', it was tested in the submission of arXiv:2006.03018 and zenodo.3872248. To help guide users on implementing the good practices for output datasets, the outputs of the default project shown in the paper now use the new features). After reading the checklist, please inspect these. Some other relevant changes in this commit: - The publication involves a copy of the necessary software tarballs. Hence a new target ('dist-software') was also added to package all the project's software tarballs in one tarball for easy distribution. - A new 'dist-lzip' target has been defined for those who want to distribute an Lzip-compressed tarball. - The '\includetikz' LaTeX macro now has a second argument to allow configuring the '\includegraphics' call when the plot should not be built, but just imported.
2020-06-04tex/build and tex/tikz treated properly in tarballMohammad Akhlaghi-1/+14
When the project is being re-built from the tarball (not the Git repository), the 'tex/build' and 'tex/tikz' addresses are actual directories, not symbolic links. In this case, when someone runs './project configure', it will complain about not being able to delete them (it assumes they are symbolic links!). So with this commit, we first check if they are deletable without '-r'. If so, then they are full directories and we rename them to a backup directory to allow the rest of the project to continue building a link there.
2020-06-04Minor improvements in the make dist command for this paperMohammad Akhlaghi-7/+12
This paper doesn't use pdflatex or biblatex, so it was necessary to make some small corrections in the make-dist rule of initialize.mk. Also, while testing the upload on arXiv, I noticed that it complains about an empty 'verify.tex' file, so that is also corrected.
2020-06-04Verification activated, README added, Proper metadata in plot dataMohammad Akhlaghi-18/+89
All the steps following the to-be-added (in 'README-hacking.md') publication checklist prior to the final check from new clone have been added: - 'README.md' file has been set. - "Reproducible supplement" was added just above the keywords, pointing to Zenodo. - A link to the to-be-uploaded data underlying the plot was added in the caption of the tools-per-year plot. - A new meta-data configuration file was added to store basic project metadata to be used throughout the project. This will later be taken into Maneage. For examle the project title is now stored here and written into the paper's LaTeX source and output datasets automatically. - Verification was activated and plot's data and LaTeX macro files are now automatically verified. - A complete metadata was added for the data underlying the plot. - A generic function was added in 'initialize.mk' that will automatically write project info and copyright in all plain-text outputs.
2020-06-03Imported recent updated in Maneage, minor conflict fixedMohammad Akhlaghi-864/+1297
The minor conflict was with 'reproduce/software/make/high-level.mk', and in particular because we implemented the fix to Maneage's Task #15664 in this project first. After it was moved to the main Maneage branch some minor stylistic corrections were done to it, thus causing the conflict. To resolve the conflict, I simply imported the full Maneage version of the file with this command: git checkout maneage -- reproduce/software/make/high-level.mk The other conflicts were due to the deleted files (that were resolved as described in 'README-hacking.md') and the LaTeX files that I had told '.gitattributes' to ignore from the Maneage branch.
2020-06-03README-hacking.md: Improved section on ignoring some files in ManeageMohammad Akhlaghi-2/+2
When some files should not be merged, until now we were suggesting to also add deleted files to the '.gitattributes' file. However, this feature of Git doesn't work for deleted files and they would still show up in the 'master' branch after a merge. So with this commit, we have added a simple AWK command to run after a merge that will automatically detect and delete such files (using the output of 'git status --porcelain'). Also, two minor typos were corrected in the newly added 'servers-backup.conf' file: the copyright year was wrong and there was no new-line at the end of the file (a good convention!).
2020-06-02Core software build before using Make to build other softwareMohammad Akhlaghi-364/+635
Until now, Maneage would only build Flock before building everything else using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid parallel downloads during the building of software (which could cause network problems). But after recently trying Maneage on FreeBSD (which is not yet complete, see bug #58465), we noticed that the BSD implemenation of Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq' parts) and its shell also had some peculiarities. It was thus decided to also install our own minimalist shell, Make and compressor program before calling 'basic.mk'. In this way, 'basic.mk' can now assume the same GNU Make features that high-level.mk and python.mk assume. The pre-make building of software is now organized in 'reproduce/software/shell/pre-make-build.sh'. Another nice feature of this commit is for macOS users: until now the default macOS Make had problems for parallel building of software, so 'basic.mk' was built in one thread. But now that we can build the core tools with GNU Make on macOS too, it uses all threads. Furthermore, since we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have to finish every line of a long rule with a backslash to keep variables and such. Generally, the pre-make software are now organized like this: first we build Lzip before anything else: it is downloaded as a simple '.tar' file that is not compressed (only ~400kb). Once Lzip is built, the pre-make phase continues with building GNU Make, Dash (a minimalist shell) and Flock. All of their tarballs are in '.tar.lz'. Maneage then enters 'basic.mk' and the first program it builds is GNU Gzip (itself packaged as '.tar.lz'). Once Gzip is built, we build all the other compression software (all downloaded as '.tar.gz'). Afterwards, any compression standard for other software is fine because we have it. In the process, a bug related to using backup servers was found in 'reproduce/analysis/bash/download-multi-try' for calling outside of 'basic.mk' and removed Bash-specific features. As a result of that bug-fix, because we now have multiple servers for software tarballs, the backup servers now have their own configuration file in 'reproduce/software/config/servers-backup.conf'. This makes it much easier to maintain the backup server list across the multiple places that we need it. Some other minor fixes: - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'. - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD (which doesn't have GCC). - We are now using 'uname -s' to specify if we are on a Linux kernel or not, if not, we are still using the old 'on_mac_os' variable. - While I was trying to build on FreeBSD, I noticed some further corrections that could help. For example the 'makelink' Make-function now takes a third argument which can be a different name compared to the actual program (used for examle to make a link to '/usr/bin/cc' from 'gcc'. - Until now we didn't know if the host's Make implementation supports placing a '@' at the start of the recipe (to avoid printing the actual commands to standard output). Especially in the tarball download phase, there are many lines that are printed for each download which was really annoying. We already used '@' in 'high-level.mk' and 'python.mk' before, but now that we also know that 'basic.mk' is called with our custom GNU Make, we can use it at the start for a cleaner stdout. - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a system where we can't install GCC (or has activated the '--host-cc' option), it may not be present and the project shouldn't break because of this. So with this commit, when a Fortran compiler isn't present, WCSLIB will be built with the '--disable-fortran' configuration option. This commit (task #15667) was completed with help/checks by Raul Infante-Sainz and Boud Roukema.
2020-05-29TeX installation crash because of different tarball versions fixedRaul Infante-Sainz-2/+58
Until this commit, when the user have previous TeX tarball already present, the project crashed when trying to re-configure, if there was a newer version of TeX. This is because TeX are updated yearly. With this commit, this bug has been fixed. Now, during the installation of TeX, it checks if this problem happens. If this is the case, then it moves the old tarball, download the new one and install it. If not, it will just install the already present tarball or crash because of any other reason. This probem was recurrent, and each time TeX was updated, the previous tarball had to be removed manually. But now, with this commit, it is done automatically. The detection and fix of this bug has been possible with the help of Mohammad Akhlaghi, thanks!
2020-05-29Reproducible research based on open-access papersBoud Roukema-0/+17
Publishing a paper on reproducible research without making it easy for readers to read the references would defeat the point. Of course we have to make some compromises with some journals' reluctance to shift towards the free world, but to satisfy scientific ethics, we should at least provide clickable URLs to the references, preferably to the ArXiv version if available [1], and also to the DOI, again, preferably to an open-access version of the URL if available. I was not able to fully get this done in the .bst file, so there's an sed/tr hack done to the .bbl file in `reproduce/analysis/make/paper.mk` to tidy up commas and spaces. This commit also reverts some of the hacks in the Akhlaghi IAU Symposium `tex/src/references.tex` entry, to match the improved .bst file, `tex/src/IEEEtran_openaccess.bst`, provided here with a different name to the original, in order to satisfy the LaTeX licence. [1] https://cosmo.torun.pl/blog/arXiv_refs
2020-05-28Fixed TeXLive crash because of differing local and server versionsRaul Infante-Sainz-3/+80
Until this commit, when the user had a previous TeXLive tarball already present (in their software-tarball directory) compared to the CTAN server, the project crashed in the configure phase. This was because TeXLive is updated yearly and we don't yet install TeXLive from source (currently we use its own package manager, but we plan to fix this in task #15267). With this commit, we fix the problem by checking the cause of the crash during the installation of TeX. If the crash is due to this particular error, we ignore the old tarball and download the new one and install it (the old one is still kept in '.build/software/tarballs', but will get a '-OLD' in its name. This probem was recurrent, and every year that TeXLive is updated, the previous tarball had to be removed manually! But with this commit, this is done automatically. The detection and fix of this bug has been possible with the help of Mohammad Akhlaghi, thanks!
2020-05-25Unified reference to GNU/Linux and free softwareMohammad Akhlaghi-8/+8
One of the main reasons to building Maneage is to properly acknowledge/attribute the authors of software in research. So we have adopted a standard of never referring to the GNU-based operating systems running the Linux kernel simply as "Linux", we avoid terms like "Open Sourse" and use Free Software instead (in the same spirit). With this commit, a few instances of the cases above have been corrected, they had slipped through our fingers when we initially imported them into the project. In the special case of the "Journal for Open Source Software", we simply replaced it with its abbreviation (JOSS). This was done because in effect we were generally using journal name abbreviations in almost all the citations already. To avoid any inconsistancies, the names of the three other journals that weren't abbreviated are also abbreviated.
2020-05-23New software: Valgrind and PatchBoud Roukema-1/+105
With this commit, Maneage now includes instructions to build the memory tracing tool Valgrind and the program 'patch' (to apply corrections/patches in text files and in particular the sources of programs). For this version of Valgrind, some patches were necessary for an interface with OpenMPI 2.x (which is the case now). Also note that this version of Valgrind's checks can fail with GCC 10.1.x (when using '--host-cc'), and the failures aren't due to internal problems but due to how the tests are designed (https://bugs.gentoo.org/707598). So currently if any of Valgrind's checks fail, Maneage still assumes that Valgrind was built and installed successfully. While testing on macOS, we noticed that it needs the macOS-specific 'mig' program which we can't build in Maneage. DESCRIPTION: The mig command invokes the Mach Interface Generator to generate Remote Procedure Call (RPC) code for client-server style Mach IPC from specification files. So a symbolic link to the system's 'mig' is now added to the project's programs on macOS systems. This commit's build of Patch and Valgrind has been tested on two GNU/Linux distributions (Debian and ArchLinux) as well as macOS. Work on this commit started by Boud Roukema, but also involved tests and corrections by Mohammad Akhlaghi and Raul Infante-Sainz.
2020-05-23Added TeXLive's ulem package to also be builtMohammad Akhlaghi-1/+1
David reported this problem, it happened right after importing IEEEtran, but for some reason, it didn't happen for me.
2020-05-23Corrected name of listings package when installing it with texliveMohammad Akhlaghi-1/+1
When entering the name of the "listings" package, I had forgot to add the final 's', so it wasn't being installed on a clean system! I didn't have a problem until now, because it remained from previous builds.
2020-05-22Software citation: removed abstract entry from BibTeX infoMohammad Akhlaghi-7/+0
Until now, two of the software BibTeX sources (Matplolib and Sympy) had an "abstract" entry that was long, not similar to the rest, and not relevant in this context, so they are removed with this commit.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-162/+246
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-05-22Re-write of the paper to fit in ~6000 words and IEEE formatMohammad Akhlaghi-7/+14
Following the fact that the DSJ editor decided that this paper doesn't fit into their scope, we decided to submit it to IEEE's Computing in Science and Engineering (CiSE). So with this commit the text was re-written to fit into their style and word-count limitations.
2020-05-08Added the option -e in the configure Gnuastro warning messageRaul Infante-Sainz-1/+1
Until this commit, when the version of Gnuastro doesn't match with the version that the project was designed to use, the warning message saying how to run the configure step was not showing the option `-e'. This situation is normal when updating the version of Gnuastro to the most recent one (with the project already configured). However, the use of this option is more convenient than giving the top-build directory, etc, every time. With this commit, the warning message has been changed in order show also the option `-e' in the re-configure of the project.
2020-05-08Installing Scamp with the option --enable-plplot=noRaul Infante-Sainz-1/+3
Until this commit, Scamp was installed with the option `--enable-plplot=yes' (the default). However, Maneage does not have PLplot included. As it is possible to install Scamp without PLplot (in that case it won't generate plots), with this commit this option has been set to `no'. As a consequence, Scamp will be installed even if the host system does not have PLplot without crashing (but it won't make any plot).
2020-05-08GNU Gettext built as a dependency of BashRaul Infante-Sainz-98/+69
Until now Maneage used the host's GNU Gettext if it was present. Gettext is a relatively low-level software that enables programs to print messages in different languages based on the host environment. Even though it has not direct effect on the running of the software for Maneage and the lanugage environment in Maneage is pre-determined, it is necessary to have it because if the basic programs see it in the host they will link with it and will have problems if/when the host's Gettext is updated. With this commit (which is actually a squashed rebase of 9 commits by Raul and Mohammad), Gettext and its two extra dependencies (libxml2 and libunistring) are now installed within Maneage as a basic software and built before GNU Bash. As a result, all programs built afterwards will successfully link with our own internal version of Gettext and libraries. To get this working, some of the basic software dependencies had to updated and re-ordered and it has been tested in both GNU/Linux and macoS. Some other minor issues that are fixed with this commit - Until this commit, when TeX was not installed, the warning message saying how to run the configure step in order to re-configure the project was not showing the option `-e'. However, the use of this option is more convenient than entering the top-build directory and etc every time. So with this commit, the warning message has been changed in order use the option `-e' in the re-configure of the project. - Until now, on macOS systems, Bash was not linking with our internally built `libncurses'. With this commit, this has been fixed by setting `--withcurses=yes' for Bash's configure script.
2020-05-06Corrected latex citation of sip_tpv program (no more \ before _)Mohammad Akhlaghi-1/+1
Until now there we had manually inserted a `\' before the `_' of sip_tpv program. However, we also recently added a step in the configure script to add a `\' before every `_' when writing the final LaTeX macro. This was because some C compilers (when the host's is used) have an `_' in their version that we had no control over. With this commit, the `\' is removed from `sip_tpv' in its build-rule and we let the backslash be inserted automatically.
2020-05-06Software are rebuilt automatically with change of versionMohammad Akhlaghi-227/+232
Until now, when you changed the version of a software in an already-built system, its tarball would be downloaded, but it wouldn't actually build. The only way would be to force the build by deleting the main target of that file (under `.local/version-info/TYPE/PROGRAM'). This was because the tarballs were an order-only prerequisite which was implemented some time ago based on some theoretical argument that if the tarball dates changes, the software should not be rebuilt (because we check the checksum). However, the problems this causes are more than those it solves: Users may forget to delete the main target of the program and mistakenly think that they are using the new version. The fact that all the numbers going into the paper also contain this number further hides this. With this commit, tarballs are no longer order-only and any time a version of a software is updated, it will be automatically built and not cause confusion and manual intervention by the users. As a result of this change, I also had to correct the way we find the tarball from the list of prerequisites.
2020-05-02First implementation of style in IEEEtran styleMohammad Akhlaghi-18/+20
The paper is no longer using LuaLaTeX, but raw LaTeX (that saves a DVI), it is so much faster! Initially I had used LuaLaTeX to use special fonts to resemble the CODATA Data Science Journal, but all that overhead is no longer necessary. Therefore I also removed the MANY extra LaTeX packages we were importing. The paper builds and is able to construct one of its images (the git-branching figure) with only 7 packages beyond the minimal TeX/LaTeX installation. Also in terms of processing it is so much faster. The text is just temporary now, and mainly just a place holder. With the next commit, I'll fill it with proper text.
2020-05-01Imported recent changes in Maneage, minor conflicts fixedMohammad Akhlaghi-248/+359
A few small conflicts showed up here and there. They are fixed with this merge.
2020-05-01Fixed OpenSSL deprecation bug on some OSs, causing problems in libgit2Boud Roukema-1/+16
Until this commit, the configure step would fail with an error when compiling libgit2 on a test system. The origin of this bug, on the OS that was tested, appears to be that in OpenSSL Version 1.1.1a, openssl/ec.h fails to include openssl/openconf.h. The bug is described in more detail at https://savannah.nongnu.org/bugs/index.php?58263 With this commit, this is fixed by manually inserting a necessary components. In particular, `sed` is used to insert a preprocessor instruction into `openssl/openconf.h`, defining `DEPRECATED_1_2_0(f)`, for an arbitrary section of code `f`, to include that code rather than exclude it or warn about it. This commit is valid provided that openssl remains at a version earlier than 1.2.0. Starting at version 1.2.0, deprecation warnings should be run normally. We have thus moved the version of OpenSSL in `versions.conf' to the section for programs that need to be manually checked for version updates with a note to remind the user when reaching that version. Other packages that use OpenSSL may benefit from this commit, not just libgit2.
2020-04-29Reactivated --host-cc config option to use host C compilerMohammad Akhlaghi-5/+23
Until now, if GCC couldn't be built for any reason, Maneage would crash and the user had no way forward. Since GCC is complicated, it may happen and is frustrating to wait until the bug is fixed. Also, while debugging Maneage, when we know GCC has no problem, because it takes so long, it discourages testing. With this commit, we have re-activated the `--host-cc' option. It was already defined in the options of `./project', but its affect was nullified by hard-coding it to zero in the configure script on GNU/Linux systems. So with this commit that has been removed and the user can use their own C compiler on a GNU/Linux operating system also. Furthermore, to inform the user about this option and its usefulness, when GCC fails to build, a clear warning message is printed, instructing the user to post the problem as a bug and telling them how to continue building the project with the `--host-cc' option.
2020-04-28Better explanation at the end of the configurationMohammad Akhlaghi-4/+8
Until now, at the end of the configuration step, we would tell the user this: "To change the configuration later, please re-run './project configure', DO NOT manually edit the relevant files". However, as Boud suggested in Bug #58243, this is against our principle to encourage users to modify Maneage. With this commit, that explanation has been expanded by a few sentences to tell the users what to change and warn them in case they decide to change the build-directory.
2020-04-28Astropy will no longer be installed by defaultMohammad Akhlaghi-1/+1
Until now Gnuastro and Astropy where installed by default in any clean build of Maneage. Gnuastro is used to do the demonstration analysis that is reported in the paper and Astropy was just there to help in testing the building of the MANY tools it depends on! It (and its dependencies) also had several papers that helped show software citation. However, as Boud suggested in task #15619, the burden of installing them for a new user may be too much and any future changes will cause merge conflicts. It may also give the impression that Maneage is only/mainly written for astronomers. So with this commit, I am removing Astropy as a default target. But we can only remove Gnuastro after we include an alternative analysis in the demonstration `delete-me' files. Following Boud's suggestion in that task, `TARGETS.conf' was also added to the files to be ignored in any future merge (in the checklist of `README-hacking.mk'). The solution was already described there, but mainly focused on the deleted `delete-me' files. So with this commit, I brought out this item as a more prominent item in the list. Maybe we can later add the analysis done in the Maneage paper (not yet published). In terms of testing the software builds, we already have task #15272 (Single target to build all high-level software, for testing) that aims to have a single configure option to install ALL high-level software and we can ask people to try if they like and report errors.
2020-04-28Configration bug fixed: other problematic software names from tarballBoud Roukema-5/+4
Similar to the previous commit (e43e3291483699), following a change made yesterday in the identification of software names from their tarballs, a few other problematic names are corrected with this commit: `apr-util', HDF5, TeX Live's installation tarball and `rpcsvc-proto'. Even though we have visually checked the list of software, other unidentified similar cases may remain and will be fixed when found in practice.
2020-04-28Configration bug fixed: identify pkg-config from its tarball nameBoud Roukema-1/+1
Until Commit 3409a54 (from yesterday), pkg-config was found correctly in `reproduce/software/make/basic.mk` by searching for `pkg`. However, commit a21ea20 made an improvement in the regular expression for relating package names and download filenames, and the string `pkg-config` with the new regex no longer simplifies to `pkg`. The result of this was that the basic.mk could not find `pkg-config` in the list of packages, since it was still listed as `pkg`. This blocked downloading for a system without pkg-config preloaded. With this commit (of just a few bytes), the bug is fixed.
2020-04-27Aborting with informative error when GNU gettext not foundMohammad Akhlaghi-1/+39
Until now, we wouldn't explicity check for GNU gettext. If it was present on the system, we would just add a link to it in Maneage's installation directory. However, in bug #58248, Boud noticed that Git (a basic software) actually needs it to complete its installation. Unfortunately we haven't had the tiem to include a build of Gettext in Maneage. Because it is mostly available on many systems, it hasn't been reported too commonly, it also has many dependencies which make it a little time consuming to install. So with this commit, we actually check for GNU gettext right after checking the compiler and if its not available an informative error message is written to inform the user of the problem, along with suggestions on fixing it (how to install GNU gettext from their package manager).
2020-04-27Configuration: improved version separation from tarball nameBoud Roukema-27/+36
Until now, the sed script for determining URL download rules in the three software building Makefiles (`basic.mk', `high-level.mk' and `python.mk') considered package names such as `fftw-3...` and `fftw2-2.1...` to be identical. As the example above shows, this would make it hard to include some software that may hav conflicting non-number names. With this commit, the SED script that is used to separate the version from the tarball name only matches numbers that are after a dash (`-'). Therefore considers `fftw-3...` and `fftw-2...` to be identical, but `fftw-3-...` and `fftw2-2.1...` to be different. As a result of this change, the `elif' check for some of the other programs like `m4', or `help2man' was also corrected in all three Makefiles. While doing this check on all the software, we noticed that `zlib-version' is being repeated two times in `version.conf' so it was removed. It caused no complications, because both were the same number, but could lead to bugs later.
2020-04-26Corrected Gnuastro configuration directory in initialize.mkZahra Sharbaf-1/+1
Recently (in Commit 8eb0892e) the Gnuastro configuration files moved under "reproduce/analysis/config/gnuastro" directory (before that they were in `reproduce/software/config/gnuastro)'. But this hadn't been reflected in it the variable that defines this directory in `initialize.mk'. With this commit, the address of the Gnuastro configuration files directory is corrected, allowing Gnuastro programs to operate properly when it is used.
2020-04-26verify-outputs.conf: typo correction in comment to avoid confusionBoud Roukema-1/+1
Until now, the comment in the file said that setting the `verify-outputs` variable to `yes` disables the verification. Looking at `reproduce/analysis/make/verify.mk` shows that the opposite is true. With this commit, the word `disable` is replaced with `enable` so that the user is not confused by the conflict between the source code in the other file and this comment.
2020-04-26Configure.sh: build directory checked for ability to modify permissionsPedram Ashofteh Ardakani-11/+81
Until now we only checked for the existance and write-ability of the build directory. But we recently discovered that if the specified build-directory is in a non-POSIX compatible partition (for example NTFS), permissions can't be modified and this can cause crashs in some programs (in particular, while building Perl, see [1]). The thing that makes this problem hard to identify is that on such partitions, `chmod' will still return 0 (so it was hard to find). With this commit, a check has been added after the user specifies the build-directory. If the proposed build directory is not able to handle permissions as expected, the configure script will not continue and will let the user know and will ask them for another directory. Also, the two printed characters at the start of error messages were changed to `**' (instead of `--'). When everything is good, we'll use `--' to tell the user that their given directory will be used as the build directory. And since there are multiple checks now, the final message to specify a new build directory is now moved to the end and not repeated in every check. [1] https://savannah.nongnu.org/support/?110220
2020-04-24TypoBoud Roukema-1/+1
2020-04-23Further edits to summarize the parts corrected by BoudMohammad Akhlaghi-1/+4
[Compared to first submission to DSJ last week with 11436 words in raw PDF, we have decreased the paper by ~1000 words to 10493 :-)] As with the previous commits, the moment Boud changed the structure of sentences, I was able to find the redundancies and remove them! This is a fascinating feature of collaboration I had never felt before: it is so hard to find redundancies in my own raw text, but even a minor correction by someone else suddeny breaks my mental memories/barrier on the sentence, allowing me to be more critical to it! Anyway, besides such corrections, I fixed a few other things: 1) In the DSJ's recently published papers, ther is no `~' between "Figure" and its number. 2) I noticed that in `tex/src/figure-src-inputconf.tex' I was actually using manually input strings for the filename, checksum and size! This was contrary to the whole philosophy of Maneage(!), I must have rushed and forgot! So LaTeX variables are now defined and used.