aboutsummaryrefslogtreecommitdiff
path: root/project
AgeCommit message (Collapse)AuthorLines
2020-06-06IMPORTANT: Added publication checklist, improved relevant infrastructureMohammad Akhlaghi-1/+2
Possible semantic conflicts (that may not show up as Git conflicts but may cause a crash in your project after the merge): 1) The project title (and other basic metadata) should be set in 'reproduce/analysis/conf/metadata.conf'. Please include this file in your merge (if it is ignored because of '.gitattributes'!). 2) Consider importing the changes in 'initialize.mk' and 'verify.mk' (if you have added all analysis Makefiles to the '.gitattributes' file (thus not merging any change in them with your branch). For example with this command: git diff master...maneage -- reproduce/analysis/make/initialize.mk 3) The old 'verify-txt-no-comments-leading-space' function has been replaced by 'verify-txt-no-comments-no-space'. The new function will also remove all white-space characters between the columns (not just white space characters at the start of the line). Thus the resulting check won't involve spacing between columns. A common set of steps are always necessary to prepare a project for publication. Until now, we would simply look at previous submissions and try to follow them, but that was prone to errors and could cause confusion. The internal infrastructure also didn't have some useful features to make good publication possible. Now that the submission of a paper fully devoted to the founding criteria of Maneage is complete (arXiv:2006.03018), it was time to formalize the necessary steps for easier submission of a project using Maneage and implement some low-level features that can make things easier. With this commit a first draft of the publication checklist has been added to 'README-hacking.md', it was tested in the submission of arXiv:2006.03018 and zenodo.3872248. To help guide users on implementing the good practices for output datasets, the outputs of the default project shown in the paper now use the new features). After reading the checklist, please inspect these. Some other relevant changes in this commit: - The publication involves a copy of the necessary software tarballs. Hence a new target ('dist-software') was also added to package all the project's software tarballs in one tarball for easy distribution. - A new 'dist-lzip' target has been defined for those who want to distribute an Lzip-compressed tarball. - The '\includetikz' LaTeX macro now has a second argument to allow configuring the '\includegraphics' call when the plot should not be built, but just imported.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-8/+11
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-04-25IMPORTANT: Primary Maneage repositories are now under maneage.orgMohammad Akhlaghi-1/+1
Until now, the primary Maneage URLs were under GitLab, but since we now have a dedicated URL and Git repository, its better to transfer to this as soon as possible. Therefore with this commit, throughout Maneage, any place that Maneage was referenced through GitLab has been corrected. Please correct your project's remote to point to the new repository at `git.maneage.org/project.git', and please make sure it follows the `maneage' branch. There is no more `master' branch on Maneage.
2020-04-20Maneage instead of Template in README-hacking.md and copyright noticesMohammad Akhlaghi-8/+8
Until now, throughout Maneage we were using the old name of "Reproducible Paper Template". But we have finally decided to use Maneage, so to avoid confusion, the name has been corrected in `README-hacking.md' and also in the copyright notices. Note also that in `README-hacking.md', the main Maneage branch is now called `maneage', and the main Git remote has been changed to `https://gitlab.com/maneage/project' (this is a new GitLab Group that I have setup for all Maneage-related projects). In this repository there is only one `maneage' branch to avoid complications with the `master' branch of the projects using Maneage later.
2020-02-29IMPORTANT: re-preparation can only be done with --prepare-redoMohammad Akhlaghi-5/+19
Until now, the preparation phase was always executed before the final build phase when running `./project make'. But when it becomes necessary, project preparation can be slow and will un-necessarily slow down the project while the project is growing (focus is on the analysis that is done after preparation). With this commit, preparation will be done automatically the first time that the project is run (`.build/software/preparation-done.mk' doesn't exist). However, after preperation is complete once, future runs of `./project make' won't do preparation any more (by calling `top-prepare.mk'). They will directly call `top-make.mk' for the analysis. To manually invoke preparation after the first attempt, the `./project make' script should be run with the new `--prepare-redo' option. Also, since the preparation phase is now automatically done before the analysis phase, the long notice that describes running `./project make' at the end of the preparation phase has been removed in `top-prepare.mk'. It now just prints a short line, saying the preparation has been complete. Finally, when the project has not been run with the proper group configuration, it ends with an `exit 1' so the main `./project' script doesn't proceed any further.
2020-02-01Better message for analysis when configuration wasn't completeMohammad Akhlaghi-4/+13
Until now, when `./project make' was run after an insuccessful run of `./project configure', it would just say to run `./project configure'. But for a first time user, this could be confusing because when the configuration is done in parallel, the error message can be very high on the command-line outputs and not seen clearly. With this commit, the error message is more complete and describes the problem and what the users should do in which circumstance.
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-2/+2
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-27Initial scripts compatible with Dash (minimalistic POSIX)Mohammad Akhlaghi-5/+5
Until now, the initial project scripts were primarily tested with GNU Bash. But Bash is not generally available on all systems (it has many features beyond POSIX). Because of this, effectively we were imposing the requirement on the user that they must have Bash installed. We recently started this with setting the shebang of `project' and `reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul and Gaspar reported an error on their systems. To fix the problem, I installed Dash (a minimalist POSIX-compliant shell) on my computer and temporarily set the shebangs to `/bin/dash', ran the project configuration step and fixed all issues that came up. With this commit, it can go all the way to building GCC on my system's Dash. After this stage (when `high-level.mk' is called), there is no problem, because we have our own version of GNU Bash and that installed version is used. Probably some more issues still remain and will hopefully be found in the future. While doing this, I also noticed the following two minor issues: - The `./project configure' option `--input-dir' was not recognized because it was mistakenly checking `--inputdir'. It has been corrected. - The test C programs now use the `<<EOF' method instead of `echo'. - In `basic.mk', the extra space between `syspath' and `:=' was removed (it was an ancient relic!).
2020-01-23Hashbangs of project and configure.sh set to /bin/shMohammad Akhlaghi-1/+1
Until now, the hashbang of these two shell scripts was set to `/bin/bash', hence assuming that GNU Bash exists on the host system! But this is an extra requirement on the host operating system and these two scripts should be written such that they operate on a POSIX shell (the generic `/bin/sh' which can point to any shell program). With this commit this has been implemented! We may confront some errors as the system is run on other systems, but we should fix such errors and work hard to make these two scripts as POSIX-compatible as possible (runnable on any shell, so as not to force users to install Bash before running the project). This completes Task #15525.
2020-01-23IMPORTANT: Project preparation is now also done with project makeMohammad Akhlaghi-60/+22
Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system.
2020-01-19New --check-config option to ./project to check software build statusMohammad Akhlaghi-0/+42
Until now, it was necessry to run a long `while true' loop to see what is currently being built at configure time. So with this commit, a new `--checkconfig' option has been added to `./project' that can be called to run that loop and make it easier to check.
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-1/+1
Now that its 2020, its necessary to include this year in the copyright statements.
2019-11-24Project's Makefiles run with no builtin rules or variablesMohammad Akhlaghi-1/+2
Until now, after removing all environment variables, we were just giving Make the top Makefile to execute. By default, for every target, Make will try many built-in rules (which is good when compiling programs, but redundant in other cases). All these checkings also populate the debugging output of Make (with `-d'). So its easier and slightly faster to just tell Make to ignore builtin rules and variables. With this commit, to address this issue, the `project' script runs `.local/bin/make' with `--no-builtin-rules' and `--no-builtin-variables'.
2019-10-01Infrastructure to keep preparation resultsMohammad Akhlaghi-1/+1
A special directory is now defined in `initialize.mk' that can be used in both the preparation and build phases. Also, the contents of prepared results can now be conditionally read during `./project make'.
2019-10-01Preparation phase added before final buildingMohammad Akhlaghi-31/+97
In many real-world scenarios, `./project make' can really benefit from having some basic information about the data before being run. For example when quering a server. If we know how many datasets were downloaded and their general properties, it can greatly optmize the process when we are designing the solution to be run in `./project make'. Therefore with this commit, a new phase has been added to the template's design: `./project prepare'. In the raw template this is empty, because the simple analysis done in the template doesn't warrant it. But everything is ready for projects using the template to add preparation phases prior to the analysis.
2019-09-28Main project script sets executable flags at configure timeMohammad Akhlaghi-0/+20
Until now, when the project's source was downloaded from something like arXiv, in `README.md', we were instructing them to set the executable flags of all the files that need it. But except for `./project', the reader shouldn't have to worry about the project internals! Once its executable, `./project' can easily fix the executable flags of all the files that need it automatically. With this commit, in `README.md', we just instruct the reader to set the executable flag of `./project' and any other file that needs an executable flag is given one at the start of the set of commands for `./project configure'. In customized projects, if an author needs executable flags on any other files, they can easily add it there without involving the user.
2019-09-26Working project when downloaded from arXivMohammad Akhlaghi-0/+13
Until now, we were assuming that the users would just clone the project in Git. But after submitting arXiv:1909.11230, and trying to build directly from the arXiv source, I noticed several problems that wouldn't allow users to build it automatically. So I tried the build step by step and was able to find a fix for the several issues that came up. The scripting parts of the fix were primarily related to the fact that the unpacked arXiv tarball isn't under version control, so some checks had to be put there. Also, we wanted to make it easy to remove the extra files, so an extra `--clean-texdit' option was added to `./project'. Finally, some manual corrections were necessary (prior to running `./project', which are now described in `README.md'. Most of the later steps can be automated and we should do it later, I just don't have enough time now.
2019-09-16Distribution tarball now builds in arXivMohammad Akhlaghi-11/+20
`./project make dist' will package all the LaTeX-specific files (and analysis source files) into one `tar.gz' file that is ready to upload to servers like arXiv. However, it wasn't updated for some time, so running it would complain about not having a `configure' script in the top of the project. With this commit, it now works with the new file-structure of the project and also copies all the BibLaTeX source files and `paper.bbl' into the top tarball directory, which allows arXiv to build the paper as intended. The output of `./project make dist' has been uploaded and tested on arXiv and it is built by arXiv perfectly. Also, a short description of all the special `make' targets was added to the output of `./project --help'.
2019-07-28Corrected typo in environment before running makeMohammad Akhlaghi-1/+1
We recently moved the system's `rm' program absolute address to a shell variable that is found during the `./project' script. But I had forgot to account for the difference between the Make and Bash variable naming differences. I had also forgot to add a value to the HOME variable. With this commit both are corrected: the system's `rm' path is now called `sys_rm' and the HOME variable is set.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-0/+291
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.