aboutsummaryrefslogtreecommitdiff
path: root/reproduce
AgeCommit message (Collapse)AuthorLines
2021-01-09Imported recent changes in Maneage, minor single conflict fixedMohammad Akhlaghi-69/+80
There was a single conflict in the comments of one part of 'configure.sh' that has been fixed. There was also a single place that needed to convert 'BDIR' to 'badir' in this project (so after the merge, it also built easily).
2021-01-09IMPORTANT: analysis outputs written in BDIR/analysisMohammad Akhlaghi-54/+78
Until now, the build directory contained a 'software/' directory (that hosted all the built software), a 'tex/' subdirectory for the final building of the paper, and many other directories containing intermediate/final data of the specific project. But this mixing of built software and data is against our modularity and minimal complexity principles: built software and built data are separate things and keeping them separate will enable many optimizations. With this commit, the build directory of the core Maneage branch will only contain two sub-directories: 'software/' and 'analysis/'. The 'software/' directory has the same contents as before and is not touched in this commit. However, the 'analysis/' directory is new and everything created in the './project make' phase of the project will be created inside of this directory. To facilitate easy access to these top-level built directories, two new variables are defined at the top of 'initialize.mk': 'badir', which is short for "built-analysis directory" and 'bsdir', which is short for "built-software directory". HOW TO IMPLEMENT THIS CHANGE IN YOUR PROJECT. It is easy: simply replace all occurances of '$(BDIR)' in your project's subMakefiles (except the ones below) to '$(badir)'. To confirm if everything is fine before building your project from scratch after merging, you can run the following command to see where 'BDIR' is used and confirm the only remaning cases. $ grep -r BDIR reproduce/analysis/* --> make/verify.mk: innobdir=$$(echo $$infile | sed -e's|$(BDIR)/||g'); \ --> make/initialize.mk:badir=$(BDIR)/analysis --> make/initialize.mk:bsdir=$(BDIR)/software --> make/initialize.mk: $$sys_rm -rf $(BDIR) --> make/top-prepare.mk:all: $(BDIR)/software/preparation-done.mk 'BDIR' should only be present in lines of the files above. If you see '$(BDIR)' used anywhere else, simply change it to '$(badir)'. Ofcourse, if your project assumes BDIR in other contexts, feel free to keep it, it will not conflict. If anything un-expected happens, please post a comment on the link below (you need to be registered on Savannah to post a comment): https://savannah.nongnu.org/task/?15855 One consequence of this change is that the 'analysis/' subdirectory can be optionally mounted on a separate partition. The need for this actually came up for some new users of Maneage in a Docker image. Docker can fix portability problems on systems that we haven't yet supported (even Windows!), or had a chance to fix low-level issues on. However, Docker doesn't have a GUI interface. So to see the built PDF or intermediate data, it was necessary to copy the built data to the host system after every change, which is annoying during working on a project. It would also need two copies of the source: one in the host, one in the container. All these frustrations can be fixed with this new feature. To describe this scenario, README.md now has a new section titled "Only software environment in the Docker image". It explains step-by-step how you can make a Docker image to only host the built software environment. While your project's source, software tarballs and 'BDIR/analysis' directories are on your host operating system. It has been tested before this commit and works very nicely.
2021-01-05Configuration: GNU Binutils linking bug on some systems fixedMohammad Akhlaghi-1/+1
Until now, when building GNU Binutils on GNU Linux operating systems, we would simply put a link to the host's core C library components (the '*crt*' files). However, the symbolic link wasn't "forced"! So if it already existed in the build directory, it would crash. With this commit a '-f' option has been added to the 'ln' command and this fixed the problem. This bug was reported by Zahra Sharbaf.
2021-01-05appendix.bbl is now included in make dist tarballMohammad Akhlaghi-4/+7
Since the addition of the appendix bibliography we hadn't checked the 'make dist' command, as a result the PDF couldn't be built. With this commit, in the 'dist' rule, we are now also copying 'appendix.bbl' and the created tarball could build the PDF properly. Also the 'peer-review' directory is now also included in the tarball created by './project make dist'. I also found a small typo in the description of Occam (an 'a' was missing) and fixed it.
2021-01-04Imported recent updates in Maneage, no conflictsMohammad Akhlaghi-1/+4
There weren't any conflicts in this merge; either technical conflicts that can be found by Git, or logical conflicts (that will cause a crash in the project).
2021-01-04Building of Less program now uses patchelf to ensure good linkingMohammad Akhlaghi-0/+3
After correctly setting Less to depend on 'ncurses', I noticed its still not linking to Maneage's 'ncurses', but pointing to my host system's 'ncurses' (that happens to have the same version! So it would crash on a system with a different version). This shows that like some other software, we need to manually correct the RPATH inside Less. With this command, the necessary call to 'patchelf' has been added and with it, the installed 'less' command properly linked to Maneage's internal build of 'ncurses'.
2021-01-04Building of less software depends on ncursesMohammad Akhlaghi-1/+1
Until now, the 'less' software package (used to view large files easily on the command-line and used by Git for things like 'git diff' or 'git log') only depended on 'patchelf' (which is a very low-level software). However, as Boud reported in bug #59811 [1], building less would crash with an error saying "Cannot find terminal libraries" in some systems (including the proposed Docker image of 'README.md' which I confirmed afterwards). Looking into the 'configure' script of 'less', I noticed that 'less' is actually just checking for some functions provided by the ncurses library! With this commit, 'less' depends on 'ncurses'. I was able to confirm that with this change, 'less' successfully builds within the Docker image. [1] https://savannah.nongnu.org/bugs/?59811
2021-01-03Updated copyrights of project-specific copyrightsMohammad Akhlaghi-4/+4
Having entered 2021, it was necessary to update the years of all the copyright statements.
2021-01-03Imported recent updates in Maneage, minor conflicts fixedMohammad Akhlaghi-112/+189
There were only three very small conflicts that have been fixed.
2021-01-02Supplement (containing appendices) optionally built separatelyMohammad Akhlaghi-47/+55
Until now, the build strategy of the paper was to have a single output PDF that either contains (1) the full paper with appendices in the same paper (2) only the main body of the paper with no appencies. But the editor in chief of CiSE recently recommended publishing the appendices as supplements that is a separate PDF (on its webpage). So with this commit, the project can make either (1) a single PDF (containing both the main body and the appendices) that will be published on arXiv and will be the default output (this is the same as before). (2) two PDFs: one that is only the main body of the paper and another that is only the appendices. Since the appendices will be printed as a PDF in any case now, the old '--no-appendix' option has been replaced by '--supplement'. Also, the internal shell/TeX variable 'noappendix' has been renamed to 'separatesupplement'.
2021-01-02./project make: new texclean targetMohammad Akhlaghi-1/+6
Until now there was only a 'clean' (to delete all files created during the 'make' phase) and the 'distclean' (to delete all files during configuration and make). But sometimes we don't want to delete all the files created during the full 'make' phase, we only want to delete the files that were created by LaTeX for building the paper. Witht this commit, a new target has been added for this job. You can now run the following command for this job: ./project make texclean Only the files in '$(BDIR)/tex/build' will be deleted (and the 'tikz' directory under that location is recreated, ready for a future build).
2021-01-02Copyright year updated in all source filesMohammad Akhlaghi-71/+71
Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.
2020-12-14Better warnings when maneage branch not present and PDF not builtMohammad Akhlaghi-12/+31
Until now, there was no warning when the 'maneage' branch didn't exist in the Git history. This can happen when you forget to push the 'maneage' branch to a remote for your project, and you later clone your project from that remote (for example on another computer). We use the 'maneage' branch to report the latest commit hash and date in the final paper (which can greatly help future readers). Since we check the 'maneage' branch on every run of './project make' (in 'initialize.mk') this would result in a printed statement like this: fatal: Not a valid object name maneage Also until now, the description of what to do when TeXLive wasn't installed properly wasn't complete: it didn't mention that it is necessary to delete the TeXLive target files. This could confuse users (they would re-run './project configure -e', but with no effect). With this commit, for the 'maneage' branch issue a complete warning will be printed. Telling the user what to do to get the 'maneage' branch (and thus fix this warning). Also, the LaTeX macros that go in the paper are now red when the 'maneage' branch doesn't exist, telling the user to see the printed warning (thus encouraging the user to get the branch). For the TeXLive issue, the necessary commands to run are now also printed in the warning.
2020-12-09Configuration: not settting C_INCLUDE_PATH on macOSRaul Infante-Sainz-2/+9
Until now, when building the high-level (optional) software, we would give both 'CPPFLAGS' and 'C_INCLUDE_PATH' the same value/directory in 'high-level.mk'. But we recently found that on macOS's C compiler ('clang'), if a directory is included in both 'CPPFLAGS' and 'C_INCLUDE_PATH', then that directory is ignored in 'CPPFLAGS' (which has higher priority). This caused linking problems when the version of a software on the host was different from the Maneage version. With this commit, 'C_INCLUDE_PATH' is not set on macOS any more and this fixed the problem on the reported systems. This bug was fixed with the help of Mohammad Akhlaghi and Mahdieh Navabi.
2020-12-02Less is now built as a basic softwareMohammad Akhlaghi-0/+13
Less is rarely used in non-interactive mode and is primarily intended for interactively viewing large files. So its need within Maneage (for batch processing) wasn't often felt until now. However, when running './project shell' (which completely closes-off the outside environment), or building a Maneage'd project within a minimal container that doesn't have less, it becomes hard to use Git (and in particular its 'diff' output which depends on 'less'). With this commit, Less has been added as a dependency of Git in 'basic.mk'. In total its built product is roughly 800KB and builds within a second or two. So it isn't a burden on any project. But it can be very useful when the projects are being developed within the Maneage environment itself.
2020-12-01Installation: m4 no longer depends on TexinfoMohammad Akhlaghi-29/+67
In a recent build on a macOS, we recognized that Texinfo needs the 'libintl.h' headers of Gettext. However, Gettext depends on M4, and until now we had set M4 to depend on Texinfo. Therefore adding Gettext as a dependency of Texinfo would cause a circular dependency. On the macOS, we temporarily disabled M4's Texinfo dependency, and the build went through. I also checked on my GNU/Linux system: temporarily renamed all Texinfo built files from my system and done a clean build of M4 and it succeeded. To be further safe, I built Maneage from this commit (where M4 doesn't depend on Texinfo) in a Docker container, and it went through with no problems. So the current M4 version indeed doesn't need Texinfo. I think adding Texinfo as a dependency of M4 was a historic issue from the early days. In the process, I also cleaned 'basic.mk' a little: - A "# Level N" comment was added on top of each group of software that can be built in parallel (generally). - GNU Nano was moved to the end of the file (to be "Level 6"). - Some comments were edited in some places.
2020-12-01Imported recent work in Maneage, minor conflicts fixedMohammad Akhlaghi-160/+92
Some minor conflicts that came up during the merge were fixed.
2020-12-01Default paper: macros available for date of commits citedMohammad Akhlaghi-1/+5
Until now, Maneage only provided the commit hashes (of the project and Maneage) as LaTeX macros to use in your paper. However, they are too cryptic and not really human friendly (unless you have access to the Git history on a computer). With this commit, to make things easier for the readers, the date of both commits are also available as LaTeX macros for use in the paper. The date of the Maneage commit is also included in the acknowledgements. Also, the paragraph above the acknowledgements has been updated with better explanation on why adding this acknowledgement in the science papers is good/necessary.
2020-12-01IMPORTANT: organizational improvements in Maneage TeX sourcesMohammad Akhlaghi-2/+15
This only concerns the TeX sources in the default branch. In case you don't use them, there should only be a clean conflict in 'paper.tex' (that is obvious and easy to fix). Conflicts may only happen in some of the 'tex/src/preamble-*.tex' files if you have actually changed them for your project. But generally any conflict that does arise by this commit with your project branch should be very clear and easy to fix and test. In short, from now on things will even be easier: any LaTeX configuration that you want to do for your project can be done in 'tex/src/preamble-project.tex', so you don't have to worry about any other LaTeX preamble file. They are either templates (like the ones for PGFPlots and BibLaTeX) or low-level things directly related to Maneage. Until now, this distinction wasn't too clear. Here is a summary of the improvements: - Two new options to './project make': with '--highlight-new' and '--highlight-notes' it is now possible to activate highlighting on the command-line. Until now, there was a LaTeX macro for this at the start of 'paper.tex' (\highlightchanges). But changing that line would change the Git commit hash, making it hard for the readers to trust that this is the same PDF. With these two new run-time options, the printed commit hash will not changed. - paper.tex: the sentences are formatted as one sentence per line (and one line per sentence). This helps in version controlling narrative and following the changes per sentence. A description of this format (and its advantages) is also included in the default text. - The internal Maneage preambles have been modified: - 'tex/src/preamble-header.tex' and 'tex/src/preamble-style.tex' have been merged into one preamble file called 'tex/src/preamble-maneage-default-style.tex'. This helps a lot in simply removing it when you use a journal style file for example. - Things like the options to highlight parts of the text are now put in a special 'tex/src/preamble-maneage.tex'. This helps highlight that these are Maneage-specific features that are independent of the style used in the paper. - There is a new 'tex/src/preamble-project.tex' that is the place you can add your project-specific customizations.
2020-11-27Fix paper.bbl flaw; reduce long author lists: save 70 wordsBoud Roukema-10/+21
This commit fixes the error of trying to run bibtex on appendix.tex when the --no-appendix option is selected. A hardwired hack, appropriate only for this specific paper, replaces the more-than-three-author parts of two long author lists by "et al." To test this without having to redownload the menke file, first do "rm -fv .build/tex/build/*.aux .build/tex/build/*.bbl" and then "./project make --no-appendix" a few times. This commit should reduce the word length by about 70 words.
2020-11-26All the referee points have been answeredMohammad Akhlaghi-1/+1
There is an answer for all the referee points now. I also did some minor edits in the paper. But we are still over the limit by around 250 words. The only remaining point that is not yet addressed (and has '####' around it) is the discussion on parallelization and its effect on reproducibility.
2020-11-25Reviewer points 1-15; appendix clickable linksBoud Roukema-0/+7
This commit updates "paper.tex" and "peer-review/1-answer.txt" for the first 15 (out of 59!) reviewer points, excluding points 2 (not yet done) and 9 (README-hacking.md needs tidying). A fix to "reproduce/analysis/make/paper.mk" for the links in the appendices is also done in this commit (the same algorithm as for paper.tex is added). The links in the appendices are not (yet) clickable.
2020-11-23Minor edits and correctionsMohammad Akhlaghi-1/+1
Raul's added point on the answer to the referee was very good, so I edited it a little to be more clear (and removed his name). Also, after looking in a few parts of the text, I fixed a few typos.
2020-11-23First draft of all the points addressed by the refereesMohammad Akhlaghi-2/+8
A new directory has been added at the top of the project's source called 'peer-review'. The raw reviews of the paper by the editors and referees has been added there as '1-review.txt'. All the main points raised by the referees have been listed in a numbered list and addressed (mostly) in '1-answers.txt'. The text of the paper now also includes all the implemented answers to the various points.
2020-11-20Highlighting changes can now be toggled at run-timeMohammad Akhlaghi-1/+11
Until now, the core Maneage 'paper.tex' had a '\highlightchanges' macro that defines two LaTeX macros: '\new' and '\tonote'. When '\highlightchanges' was defined, anything that was written within '\new' became dark green (highlighting new things that have been added). Also, anything that was written in '\tonote' was put within a '[]' and became dark red (to show that there is a note here that should be addressed later). When '\highlightchanges' wasn't defined, anything within the '\new' element would be black (like the rest of the text), and the things in '\tonote' would not be shown at all. Commenting the '\newcommand{\highlightchanges}{}' line within 'paper.tex' (to toggle the modes above) would create a different Git hash and has to be committed. But this different commit hash could create a false sense in the reader that other things have also been changed and the only way they could confirm was to actually go and look into the project history (which they will not usually have time to do, and thus won't be able to trust the two modes of the text). Also, the added highlights and the note highlights were bundeled together into one macro, so you couldn't only have one of them. With this commit, the choice of highlighting either one of the two is now done as two new run-time options to the './project' script (which are passed to the Makefiles, and written into the 'project.tex' file which is loaded into 'paper.tex'). In this way, we can generate two PDFs with the same Git commit (project's state): one with the selected highlights and another one without it. This issue actually came up for me while implementing the changes here: we need to submit one PDF to the journal/referees with highlights on the added features. But we also need to submit another PDF to arXiv and Zenodo without any highlights. If the PDFs have different commit hashes, the referees may associate it with other changes in any part of the work. For example https://oadoi.org/10.22541/au.159724632.29528907 that mentions "Another version of the manuscript was published on arXiv: 2006.03018", while the only difference was a few words in the abstract after the journal complained on the abstract word-count of our first submission (where the commit hashes matched with arXiv/Zenodo).
2020-11-15First edits on the newly added appendices in new formMohammad Akhlaghi-2/+3
With the optional appendices added recently to the paper, it was important to go through them and make them more fitting into the paper.
2020-11-15Building final PDF: pdf-build-final has to be given an explicit yesMohammad Akhlaghi-15/+15
Until now, when the 'pdf-build-final' configuration variable (defined in 'reproduce/analysis/config/pdf-build.conf') was given any string a PDF would be built. This was very confusing, because people could put a 'no' and the PDF would still be built! With this commit, only when this variable has a value of 'yes' will the PDF be built. If given any other string (or no string at all), it will not produce a PDF. This issue was reported by Zahra Sharbaf.
2020-11-04Appendix of long paper added, optionally we can disable itMohammad Akhlaghi-3/+17
Given the referee reports, after discussing with the editors of CiSE, we decided that it is important to include the complete appendix we had before that included a thorough review of existing tools and methods. However, the appendix will not be published in the paper (due to the strict word-count limit). It will only be used in the arXiv/Zenodo versions of the paper. This actually created a technical problem: we want the commit hash of the project source to remain the same when the paper is built with an appendix or without it. To fix this problem the choice of including an appendix has gone into the 'project' script as a run-time option called '--no-appendix'. So by default (when someone just runs './project make'), the PDF will have an appendix, but when we want to submit to the journal, or when the appendix isn't needed for a certain reason, we can use this new option. The appendix also has its own separate bibliography. Some other corrections made in this commit: 1. Some new references were added that had an '_' in their source, they were corrected in 'references.tex'. 2. I noticed that 'preamble-style.tex' is not actually used in this paper, so it has been deleted.
2020-10-18Recipes for final initialize and verify targets not on stdoutMohammad Akhlaghi-4/+11
The LaTeX macro files for these two subMakefiles are created on every run of './project make'. So their commands are also printed every time and hardly ever will a normal user want to modify or change these. So to avoid populating the standard output of a Maneaged project with all these extra lines every time (possibly getting mixed with the important analysis or LaTeX outputs), an '@' has been placed at the start of the recipes. With an '@' at the start of the recipe, Make is instructed to not print the commands it wants to run in the standard output.
2020-10-02TexLive's xstring package is now necessaryMohammad Akhlaghi-4/+4
After a fresh build of Maneage with a newly downloaded TeXLive, I noticed that it is complaining about not finding 'xstring.sty', apparently some package that depeneded on it is no longer including it itself! It is thus now added to the packages that are built by Maneage's TeXLive.
2020-09-24Gnuastro's analysis configuration files removedMohammad Akhlaghi-134/+2
Until now, the core Maneage branch included some configuration files for Gnuastro's programs. This was actually a remnant of the distant past when Maneage didn't actually build its own software and we had to rely on the host's software versions. This file contained the configuration files specific to Gnuastro for this project and also had a feature to avoid checking the host's own configuration files. However, we now build all our software ourselves with fixed configuration files (for the version that is being installed and its version is stored). So those extra configuration files were just extra and caused confusion and problems in some scenarios. With this commit, those extra files are now removed. Also, two small issues are also addressed in parallel with this commit: - When running './project make clean', the 'hardware-parameters.tex' macro file (which is created by './project configure' is not deleted. - The project title is now written into the default output's PDF's properties (through 'hypersetup' in 'tex/src/preamble-header.tex') through the LaTeX macro. All these issues were found and fixed with the help of Samane Raji.
2020-09-15Checking Xcode installation for macOS systemsRaul Infante-Sainz-1/+60
Until now, during the configure step it was checked if the host Operative System were GNU/Linux, and if not, we assumed it is macOS. However, it can be any other different OS! With this commit, now we explicity check if the system is GNU/Linux or Darwin (macOS). If it is not any of them, a warning message says to the user that the host system is different from which we have checked so far (and invite to contact us if there is any problem). In addition to this, if the system is macOS, now it checks if Xcode is already installed in the host system. If it is not installed, a warning message informs the user to do that in case a problem/crash in the configure step occurs. We have found that it is convenient to have Xcode installed in order to avoid some problems.
2020-09-09Imported recent important fixes in Maneage, no conflictsMohammad Akhlaghi-7/+26
There weren't any conflicts in this merge.
2020-09-09R is built without tcl/tk (for GUI) dependenceBoud Roukema-0/+19
Tcl/Tk are a set of tools to provide Graphic User Interface (GUI) support in some software. But they are not yet natively built within Maneage, primarily because we have higher-priority work right now. GUI tools in general aren't high on our priority list right now because GUI tools are generally good for human interaction (which is contrary to the reproducible philosophy), not automatic analysis (a core concept in reproducibility). So even later, when we do include Tcl/Tk in Maneage, their direct usage will be discouraged. Until this commit, because we don't yet build Tcl/Tk, the default maneage install of the statistical package R failed on a Debian Stretch, with 6227 repeats of the line: '/usr/lib//tcl8.5/tclConfig.sh: line 2: dpkg-architecture: command not found' To fix this problem (atleast until Tcl/Tk is installed within Maneage), R is now configured with the '--without-tcltk' option which fixed the problem. Please see the description above the R installation instructions in 'reproduce/software/make/high-level.mk' for more.
2020-09-08Removed all occurances of IFS in low-level scriptsMohammad Akhlaghi-3/+3
Following the previous commit, we recognized that the 'IFS' terms are not necessary and can be even cause problems. So all their occurances in the scripts of Maneage have been removed with this commit.
2020-09-07Software installation: removed IFS statements in pre-make-build.shMohammad Akhlaghi-4/+4
Until a recent commit, the IFS='"' was added at the start of the variables in this shell script and as a result, the SPACE character wasn't being used as a delimiter. This caused a major problem when downloading the tarballs (all the backup servers were considered as the top link). With this commit we removed these 'IFS' statements). Because we now check for the existance of meta-characters in the build directory name, there is no more problem, and also generally both the calling command and internally, we have double-qutations around the variable names. So removal of IFS will not affect the result in this scenario. This bug was found by Mohammadreza Khellat.
2020-09-03Imported recent work in Maneage, minor conflicts fixedMohammad Akhlaghi-136/+306
Only two small conflicts came up: * The addition of the hardware architecture macro in 'paper.tex' (which was removed for now, but will be added as the referee has requested within the text). * The usage of "" around directory variables in 'paper.mk'.
2020-08-28Plain text editors: nano in basic, emacs and vim in high-levelMohammad Akhlaghi-7/+74
While a project is under development, the raw analysis software are not the only necessary software in a project. We also need tools to all the edit plain-text files within the Maneaged project. Usually people use their operating system's plain-text editor. However, when working on the project on a new computer, or in a container, the plain-text editors will have different versions, or may not be present at all! This can be very annoying and frustrating! With this commit, Maneage now installs GNU Nano as part of the basic tools. GNU Nano is a very simple and small plain text editor (the installed size is only ~3.5MB, and it is friendly to new users). Therefore, any Maneaged project can assume atleast Nano will be present (in particular when no editor is available on the running system!). GNU Emacs and VIM (both without extra dependencies, in particular without GUI support) are also optionally available in 'high-level.mk' (by adding them to 'TARGETS.conf'). The basic idea for the more advanced editors (Emacs and VIM) is that project authors can add their favorite editor while they are working on the project, but upon publication they can remove them from 'TARGETS.conf'. A few other minor things came up during this work and are now also fixed: - The 'file' program and its libraries like 'libmagic' were linking to system's 'libseccomp'! This dependency then leaked into Nano (which depends on 'libmagic'). But this is just an extra feature of 'file', only for the Linux kernel. Also, we have no dependency on it so far. So 'file' is not configured to not build with 'libseccomp'. - A typo was fixed in the line where the physical core information is being read on macOS. - The top-level directories when running './project shell' are now quoted (in case they have special characters).
2020-08-27Machine architecture and byte-order available as LaTeX macroMohammadreza Khellat-129/+232
Until now, no machine-related specifications were being documented in the workflow. This information can become helpful when observing differences in the outcome of both software and analysis segments of the workflow by others (some software may behave differently based on host machine). With this commit, the host machine's 'hardware class' and 'byte-order' are collected and now available as LaTeX macros for the authors to use in the paper. Currently it is placed in the acknowledgments, right after mentioning the Maneage commit. Furthermore, the project and configuration scripts are now capable of dealing with input directory names that have SPACE (and other special characters) by putting them inside double-quotes. However, having spaces and metacharacters in the address of the build directory could cause build/install failure for some software source files which are beyond the control of Maneage. So we now check the user's given build directory string, and if the string has any '@', '#', '$', '%', '^', '&', '*', '(', ')', '+', ';', and ' ' (SPACE), it will ask the user to provide a different directory.
2020-08-20Imported recent updates in Maneage, minor conflicts fixedMohammad Akhlaghi-354/+524
Some very minor conflicts came up and were easily corrected. They were mostly in parts that are also shared with the demonstration in the core Maneage branch.
2020-08-17Minor typo correction in comments of paper.mkMohammad Akhlaghi-1/+1
The '.bbl' suffix in the comment of one call to LaTeX was incorrectly written as '.bb'.
2020-08-08Software tarballs saved as symlinks if already in filesystemMohammad Akhlaghi-5/+18
Until now, if the software source tarballs already existed on the system they would be copied inside the project. However, the software source tarballs are sometimes/mostly larger than their actual product and can consume significant space (~375 MB in the core branch!). With this commit, when the software are present on the system, their symbolic link will be placed in 'BDIR/software/tarballs', not a full copy. Also, because the tarballs in software tarball directory may themselves be links, we use 'realpath' to find the final place of the actual file and link to that location. Therefore if 'realpath' can't be found (prior to installing Coreutils in Maneage), we will copy the tarballs from the given software tarball directory. After Maneage has installed Coreutils, the project's own 'realpath' will be used. Of course, if the software are downloaded, their full downloaded copy will be kept in 'BDIR/software/tarballs', nothing has changed in the downloading scenario.
2020-08-08IMPORTANT: New software versions (17 basic, 16 high-level and 7 Python)Mohammad Akhlaghi-170/+187
It was a long time that the Maneage software versions hadn't been updated. With this commit, the versions of all basic software were checked and 17 of that had newer versions were updated. Also, 16 high-level programs and libraries were updated as well as 7 Python modules. The full list is available below. Basic Software (affecting all projects) --------------------------------------- bash 5.0.11 -> 5.0.18 binutils 2.32 -> 2.35 coreutils 8.31 -> 8.32 curl 7.65.3 -> 7.71.1 file 5.36 -> 5.39 gawk 5.0.1 -> 5.1.0 gcc 9.2.0 -> 10.2.0 gettext 0.20.2 -> 0.21 git 2.26.2 -> 2.28.0 gmp 6.1.2 -> 6.2.0 grep 3.3 -> 3.4 libbsd 0.9.1 -> 0.10.0 ncurses 6.1 -> 6.2 perl 5.30.0 -> 5.32.0 sed 4.7 -> 4.8 texinfo 6.6 -> 6.7 xz 5.2.4 -> 5.2.5 Custom programs/libraries ------------------------- astrometrynet 0.77 -> 0.80 automake 0.16.1 -> 0.16.2 bison 3.6 -> 3.7 cfitsio 3.47 -> 3.48 cmake 3.17.0 -> 3.18.1 freetype 2.9 -> 2.10.2 gdb 8.3 -> 9.2 ghostscript 9.50 -> 9.52 gnuastro 0.11 -> 0.12 libgit2 0.28.2 -> 1.0.1 libidn 1.35 -> 1.36 openmpi 4.0.1 -> 4.0.4 R 3.6.2 -> 4.0.2 python 3.7.4 -> 3.8.5 wcslib 6.4 -> 7.3 yaml 0.2.2 -> 0.2.5 Python modules -------------- cython 0.29.6 -> 0.29.21 h5py 2.9.0 -> 2.10.0 matplotlib 3.1.1 -> 3.3.0 mpi4py 3.0.2 -> 3.0.3 numpy 1.17.2 -> 1.19.1 pybind11 2.4.3 -> 2.5.0 scipy 1.3.1 -> 1.5.2
2020-08-08Configuration fail if gfortran necessary, but not built or availableBoud Roukema-25/+41
When the host C compiler is used (either by calling '--host-cc' or on OSs that we can't build the GNU C Compiler), Maneage will also not build the Fortran compiler 'gfortran'. Until now, the './project configure' script would give a big warning about the need for 'gfortran' and the fact that it is missing, and would for 5 seconds, but it would continue anyway. For projects that don't need 'gfortran', this can be confusing to the users and for those that need 'gfortran', it means that a lot of time and cpu cycles are wasted compiling non-fortran software that are unusable in the end. With this commit, the 'need_gfortarn' variable has been added 'reproduce/software/shell/configure.sh', in a new part that is devoted to project-specific features. If it equals '0', then the 'gfortran' test (and message!) isn't done at all, but if it is set to '1', then the configure stage will halt immediately gfortran is not found and not built. The default operations of the core Maneage branch don't need 'gfortran', so by default it is set to 0. But 'gfortran' is necessary for all projects that use Numpy (Python's numeric library) for example. So if your project needs 'gfortran', please set this new variable to 1. As mentioned in the comments of 'configure.sh', ideally we should detect this automatically, but we haven't had the time to implement it yet.
2020-08-02initialize.mk: accounting for no maneage branchBoud Roukema-3/+8
One of the LaTeX macros reported by 'initialize.mk' is the git commit hash of the most recent 'maneage' branch that the project has been branched from. However, not all projects will retain the maneage reference. This can happen for example when people don't push the 'maneage' reference to their repository and then clone from their own repository to a second computer. Therefore, until now, in such situations, Maneage would break with an error. With this commit, in such scenarios, a place holder string is used instead, clearly highlighting that there is no 'maneage' reference.
2020-08-02OpenMPI build with slurm compatibilityBoud Roukema-1/+4
Prior to this commit, compilation of OpenMPI used the default OpenMPI choices of deciding which libraries should be used in relating to a job scheduler [1] (such as Slurm [2]). Given that the user on a multi-user cluster has to accept the sysadmin's choice of a job scheduler, the question of whether to (1) link with OpenMPI's own libraries (and increase the reproducibility of the science project) or rather (2) link with the sysadmin managed libraries (more likely to be compatible with the host's job scheduler), is an open question of which the best strategy for reproducibility needs to be debated and studied. In this commit, strategy (1) is adopted. The options '--withpmix=internal' and '--with-hwloc=internal' are added to the configure command. The working assumption is that the Maneage version of OpenMPI is likely to be modern enough to be compatible with the native job scheduler such as Slurm. Compilation without any 'pmix' option gave a fail in at least one case; it appears that an external pmix library was sought by the configure script. As of OpenMPI 4.0.1, the internal libevent library is used by default, so there appears to be no option to force it to be chosen internally. This commit also includes the option '--without-verbs'. This option removes a library related to "infiniband", "verbs", "openib" and "BTL"; this library appears to be deprecated. See [3], [4] for discussion. Please add feedback and discussion to the Maneage task about openmpi linking strategies (1) (internal) and (2) (external) at Savannah [5]. [1] https://en.wikipedia.org/wiki/Job_scheduler#Batch_queuing_for_HPC_clusters [2] https://en.wikipedia.org/wiki/Slurm_Workload_Manager - To avoid a name clash, 'slurm-wlm' is the metapackage in Debian for the client commands, the compute node daemon, and the central node daemon. An unrelated package 'slurm' also exists. [3] https://www-lb.open-mpi.org/faq/?category=openfabrics#ofa-device-error [4] https://www-lb.open-mpi.org/faq/?category=building [5] https://savannah.nongnu.org/task/index.php?15737
2020-07-21Printing location when downloaded input data checksum is differentBoud Roukema-0/+1
There are many different directory trees involved in Maneage system: the top directory, the 'reproduce/' directory and its sub-directories, '.build/' (that point to a user-defined build area), and a possibly user-defined input directory. Until now, in the case of a download checksum failure, it was not immediately obvious [1] to the user *where* the file with a failed checksum is. To clarify to the user *where* the suspicious file is now located, this commit adds a line to 'reproduce/analysis/make/download.mk' to print out this full path location: '$$unchecked' along with the expected and calculated checksums. [1] Euphemism for me spending lots of time debugging and being confused.
2020-07-20README-hacking.md: clarify Zenodo usage in publication checklistBoud Roukema-2/+2
This commit clarifies the initial usage of Zenodo for reserving a Zenodo identifier and starting an 'unpublished' upload. Some other minor wording changes are done here.
2020-07-20make dist: only archive files that are under version controlBoud Roukema-17/+31
Until this commit, the '$(project-package-contents)' rules in 'reproduce/analysis/make/initialize.mk' included a line to provide all contents, recursively, of the directory 'reproduce/' in the package for further distribution. This could potentially lead to the distribution of private working files that are used during development and not intended for general distribution. With this commit, only those files in 'reproduce/' and 'tex/src' that are under version control are copied to the temporary directory (that is later used for creating an archive). With this change, the archiving commands actually became more clean (we don't have to manually remove 'LOCAL.conf' or other temporary files). Extensive comments have also been added above each step to clarify each step's purpose and method.
2020-07-07Fixed typo that lead to crash when building healpyMarius Peper-1/+1
Until now, if a project needed the healpy software package, Maneage would crash with the following error message (abridged for full name in build directory). This was caused by a typo in the version of 'healpix' (the dependency of 'healpy'). make: *** No rule to make target '.../version-info/proglib/healpix-' With this commit, the typo in line 334 of 'python.mk' is fixed, so that when '$(ipydir)/healpy-$(healpy-version)' gets called it correctly searches for a rule to make '$(ibidir)/healpix-$(healpix-version)'.
generated by cgit v1.2.1 (git 2.18.0) at 2024-12-26 19:51:36 +0000