aboutsummaryrefslogtreecommitdiff
path: root/README-hacking.md
AgeCommit message (Collapse)AuthorLines
2020-06-06IMPORTANT: Added publication checklist, improved relevant infrastructureMohammad Akhlaghi-16/+330
Possible semantic conflicts (that may not show up as Git conflicts but may cause a crash in your project after the merge): 1) The project title (and other basic metadata) should be set in 'reproduce/analysis/conf/metadata.conf'. Please include this file in your merge (if it is ignored because of '.gitattributes'!). 2) Consider importing the changes in 'initialize.mk' and 'verify.mk' (if you have added all analysis Makefiles to the '.gitattributes' file (thus not merging any change in them with your branch). For example with this command: git diff master...maneage -- reproduce/analysis/make/initialize.mk 3) The old 'verify-txt-no-comments-leading-space' function has been replaced by 'verify-txt-no-comments-no-space'. The new function will also remove all white-space characters between the columns (not just white space characters at the start of the line). Thus the resulting check won't involve spacing between columns. A common set of steps are always necessary to prepare a project for publication. Until now, we would simply look at previous submissions and try to follow them, but that was prone to errors and could cause confusion. The internal infrastructure also didn't have some useful features to make good publication possible. Now that the submission of a paper fully devoted to the founding criteria of Maneage is complete (arXiv:2006.03018), it was time to formalize the necessary steps for easier submission of a project using Maneage and implement some low-level features that can make things easier. With this commit a first draft of the publication checklist has been added to 'README-hacking.md', it was tested in the submission of arXiv:2006.03018 and zenodo.3872248. To help guide users on implementing the good practices for output datasets, the outputs of the default project shown in the paper now use the new features). After reading the checklist, please inspect these. Some other relevant changes in this commit: - The publication involves a copy of the necessary software tarballs. Hence a new target ('dist-software') was also added to package all the project's software tarballs in one tarball for easy distribution. - A new 'dist-lzip' target has been defined for those who want to distribute an Lzip-compressed tarball. - The '\includetikz' LaTeX macro now has a second argument to allow configuring the '\includegraphics' call when the plot should not be built, but just imported.
2020-06-04README-hacking.md: minor edits in description of merging with ManeageMohammad Akhlaghi-7/+15
The recently added description for this step in the last commit needed some edits to be more clear and encourage re-building the project from scratch anytime authors merge with Maneage.
2020-06-03README-hacking.md: Improved section on ignoring some files in ManeageMohammad Akhlaghi-22/+53
When some files should not be merged, until now we were suggesting to also add deleted files to the '.gitattributes' file. However, this feature of Git doesn't work for deleted files and they would still show up in the 'master' branch after a merge. So with this commit, we have added a simple AWK command to run after a merge that will automatically detect and delete such files (using the output of 'git status --porcelain'). Also, two minor typos were corrected in the newly added 'servers-backup.conf' file: the copyright year was wrong and there was no new-line at the end of the file (a good convention!).
2020-05-25Unified reference to GNU/Linux and free softwareMohammad Akhlaghi-2/+2
One of the main reasons to building Maneage is to properly acknowledge/attribute the authors of software in research. So we have adopted a standard of never referring to the GNU-based operating systems running the Linux kernel simply as "Linux", we avoid terms like "Open Sourse" and use Free Software instead (in the same spirit). With this commit, a few instances of the cases above have been corrected, they had slipped through our fingers when we initially imported them into the project. In the special case of the "Journal for Open Source Software", we simply replaced it with its abbreviation (JOSS). This was done because in effect we were generally using journal name abbreviations in almost all the citations already. To avoid any inconsistancies, the names of the three other journals that weren't abbreviated are also abbreviated.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-10/+10
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-04-28Astropy will no longer be installed by defaultMohammad Akhlaghi-23/+30
Until now Gnuastro and Astropy where installed by default in any clean build of Maneage. Gnuastro is used to do the demonstration analysis that is reported in the paper and Astropy was just there to help in testing the building of the MANY tools it depends on! It (and its dependencies) also had several papers that helped show software citation. However, as Boud suggested in task #15619, the burden of installing them for a new user may be too much and any future changes will cause merge conflicts. It may also give the impression that Maneage is only/mainly written for astronomers. So with this commit, I am removing Astropy as a default target. But we can only remove Gnuastro after we include an alternative analysis in the demonstration `delete-me' files. Following Boud's suggestion in that task, `TARGETS.conf' was also added to the files to be ignored in any future merge (in the checklist of `README-hacking.mk'). The solution was already described there, but mainly focused on the deleted `delete-me' files. So with this commit, I brought out this item as a more prominent item in the list. Maybe we can later add the analysis done in the Maneage paper (not yet published). In terms of testing the software builds, we already have task #15272 (Single target to build all high-level software, for testing) that aims to have a single configure option to install ALL high-level software and we can ask people to try if they like and report errors.
2020-04-26README-hacking.md: described why automatic preparation only occurs onceZahra Sharbaf-1/+20
Recently (since Commit 7d0c5ef77), the preparation is not run automatically every time. It is only run automatically the first time and needs to be manually called with the `--prepare-redo' option. But this wasn't explained in `README-hacking.md' (currently the main documentation of Maneage). With this commit, a description about invoking the preparation process after the first attempt of the running project has been added to `README-hacking.md'.
2020-04-25IMPORTANT: Primary Maneage repositories are now under maneage.orgMohammad Akhlaghi-11/+10
Until now, the primary Maneage URLs were under GitLab, but since we now have a dedicated URL and Git repository, its better to transfer to this as soon as possible. Therefore with this commit, throughout Maneage, any place that Maneage was referenced through GitLab has been corrected. Please correct your project's remote to point to the new repository at `git.maneage.org/project.git', and please make sure it follows the `maneage' branch. There is no more `master' branch on Maneage.
2020-04-21README-hacking.md: removed any mention of tagsMohammad Akhlaghi-94/+68
Tags are not a fixed piece of history (they can easily be moved and not imported in a different repository), so they are only confusing in the context of Maneage (where people should branch-off the main project). the raw commit hashes are a much more robust way to store a precise moment in history. Before this commit, I removed all Tags from the main Git repositories of Maneage and thus removed any mention of Tags with `README-hacking.md'. Ofcourse, if a project decides to use tags is upto them, but we won't implement it in the main branch.
2020-04-21README-hacking.md: minor clarifications in checklistMohammad Akhlaghi-28/+32
Roberto Baena recently tried building a new project with Maneage and provided the following suggestions to make it more clear for a new user: 1) In the part where we talk about creating a Git repository, we should highlight that it must be empty. This is because some (for example Gitlab) propose to include a `README' file. But if the project is not empty, Git will not allow pushing to it. 2) The `(can be done later)' comment was removed from the "Delete dummy parts") to avoid confusion about applying some of them, but not others: if only some are done, it may cause problems in the build.
2020-04-20README-hacking.md: Removed TeXLive year problem and numberd checklistMohammad Akhlaghi-21/+11
We recently fixed the problem of TeXLive that hard-codes the year of its build in its installation directory. But the note on this problem was still kept in `README-hacking.md'. That part is now removed. Also, to help in following the checklist, it is now an ordered list.
2020-04-20Maneage instead of Template in README-hacking.md and copyright noticesMohammad Akhlaghi-217/+211
Until now, throughout Maneage we were using the old name of "Reproducible Paper Template". But we have finally decided to use Maneage, so to avoid confusion, the name has been corrected in `README-hacking.md' and also in the copyright notices. Note also that in `README-hacking.md', the main Maneage branch is now called `maneage', and the main Git remote has been changed to `https://gitlab.com/maneage/project' (this is a new GitLab Group that I have setup for all Maneage-related projects). In this repository there is only one `maneage' branch to avoid complications with the `master' branch of the projects using Maneage later.
2020-04-17IMPORTANT: software config directly under reproduce/software/configMohammad Akhlaghi-30/+29
Until now the software configuration parameters were defined under the `reproduce/software/config/installation/' directory. This was because the configuration parameters of analysis software (for example Gnuastro's configurations) were placed under there too. But this was terribly confusing, because the run-time options of programs falls under the "analysis" phase of the project. With this commit, the Gnuastro configuration files have been moved under the new `reproduce/analysis/config/gnuastro' directory and the software configuration files are directly under `reproduce/software/config'. A clean build was done with this change and it didn't crash, but it may cause crashes in derived projects, so after merging with Maneage, please re-configure your project to see if anything has been missed. Please let us know if there is a problem.
2020-04-01Corrected reference for Infante-Sainz+2020 in README-hacking.mdRaul Infante-Sainz-1/+1
Until this commit, the year of this paper was 2019 and the linking url was the temporal one. However, the final official publication year is 2020. With this commit, the year and the url have been changed to the final ones.
2020-02-18README-hacking.md: corrected typo in project commandMohammad Akhlaghi-1/+1
I had forgot to add a `./' before the call to `project' for the `--check-config'.
2020-02-05Imported Raul's additions to README-hacking.md, no conflictsMohammad Akhlaghi-2/+16
There were no conflicts in this merge.
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-2/+2
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-27Moving basic configuration of Git section in README-hacking.mdRaul Infante-Sainz-13/+13
Until this commit, the small section of `README-hacking.md' in which it is explained how to do the first configuration of Git was at the beginning of the section `First custom commit'. However, it is better to have it just before the item `Your first commit' in that section. With this commit, this change has been done. Now the reader has the necessary steps for configuring Git just before it is needed for making the first commit.
2020-01-23IMPORTANT: Project preparation is now also done with project makeMohammad Akhlaghi-21/+12
Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system.
2020-01-22Adding Raul as contributor of README-hacking.mdRaul Infante-Sainz-1/+2
Since I (Raul) did some changes (and I hope to do more :-)) in the `README-hacking.md', I am adding my information at the beginning of this file.
2020-01-22Adding basic configuration of Git in README-hacking.mdRaul Infante-Sainz-1/+14
Until this commit, we were asuming that Git was already properly configured. However, in order to be as complete as possible, it would be good if the basic commands to configure Git were in the `README-hacking.md'. With this commit, a small paragraph has been added in order to have the basic Git configuration commands (i.e. to configure the name, email, and favorite text editor).
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-33/+33
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19New --check-config option to ./project to check software build statusMohammad Akhlaghi-31/+14
Until now, it was necessry to run a long `while true' loop to see what is currently being built at configure time. So with this commit, a new `--checkconfig' option has been added to `./project' that can be called to run that loop and make it easier to check.
2020-01-18README-hacking.md: edits and corrections for easier customizationMohammad Akhlaghi-8/+10
The checklist descriptions were slightly edited to be more clear. Also, while following them, I noticed that while removing the "delete-me" parts on `verify.mk', would cause an error: the `if [ $$m == delete-me ];' statement we were saying to delete cause an error because `elif' was the first statement Bash would see. So with this commit, the `download' conditional (which isn't instructed to be deleted) was set to be the top (with an `if') and the `delete-me' conditional now has an `elif'.
2020-01-17README-hacking.md: script to list installed programs before configureMohammad Akhlaghi-10/+18
Until now, the small one-line script that lists programs was introduced in the checklist after running `./project configure'. But people would mostly miss it because they would wait until the configuration is complete. With this commit, that point has been put above the `./project configure' step. Readers are instructed to open a new terminal and run that script, then go to the next step so they see the directories get filled actively. It will also help them understand what is going on.
2020-01-13Minor corrections in referencing Infante-Sainz+2019Mohammad Akhlaghi-6/+5
Until now the actual journal webpage was used for Raul's paper. However, the journal webpage needs authorized access for people to read it, therefore its will be inaccessible for many people. A better and more well known place for the paper (atleast in astronomy) is the ADS link. In the ADS link, if someone has access to the journal, they will get the journal's version and if not, they will get the arXiv version. It also has a common BibTeX export tool for all journals. We had also done this for the other papers in that list. With this commit, I thus changed the URL for the paper, and also removed the "issue" number (4 in this case), since that is mostly irrelevant, only the volume and page numbers are usually used for the other papers too.
2020-01-13Added Infante-Sainz et al. 2019 as most recent paper using this templateRaul Infante-Sainz-3/+4
The "SDSS extended PSFs" paper was already included as an example of papers wich uses this template. However, the reference was the arXiv one. With this commit, since the paper has been finally published, it has been added the final reference to the journal.
2020-01-01Verification function checks if file existsMohammad Akhlaghi-5/+7
Until now, if the file to be verified didn't exist, a different checksum would be generated, and it would stop, but it wasn't immediately clear if the differing checksum is because the file doesn't exist at all! With this commit, before calculating the checksum, we first make sure if the file exists. If it doesn't exist an explicit error is printed and thus will help the project editor to find the cause of the problem.
2020-01-01Minor corrections in `README-hacking.md' after verificationMohammad Akhlaghi-4/+7
In the previous commit, I had forgot to update a small part in the checklist (when modifying `top-make.mk') which is now corrected. I also added a few sentences in the description of how to customize the verification to make it easier to understand.
2020-01-01Verification of output values and data added within templateMohammad Akhlaghi-9/+51
Until now, the only verification that the template provided was the published PDF. Users had to manually compare the published and generated PDFs (numbers, plots, tables) and see if they obtained the same result. However, this type of manual verification is not good and is prone to frustration and missing important differences. With this commit, a new Makefile has been added in the analysis steps: `verify.mk'. It provides facilities to easily verify the results that go into the paper. For example tables that go into making the paper's plots, or the LaTeX macros that blend into the text. See the updated parts in `README-hacking.md` for a more complete explanation. This completes task #15497.
2020-01-01README-hacking.md checklist now also ignores changes in paper.texMohammad Akhlaghi-7/+12
In the previous commit, we added the files to ignore from the template branch, but only the files that had been deleted. With this commit, `paper.tex' is also added to the files that must be ignored from the template branch (the file remains in the project, but in the template branch, its contents are just dummy place-holders).
2020-01-01Added step README-hacking's checklist to avoid merging dummy filesMohammad Akhlaghi-0/+15
During the checklist we guide the user to delete the dummy `delete-me*' files from their custom branch. Later, if the dummy files are updated in the template's master branch, if the user merges with the template branch, these files will be written back into their project! This is very annoying! With this commit, a step was added in the `README-hacking.md' checklist, just after deleting the dummy files to avoid this problem using the `.gitattributes' file, telling Git to keep the changes as implemented in the merging branch (`ours').
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-3/+3
Now that its 2020, its necessary to include this year in the copyright statements.
2019-11-06Added 1911.01430 as most recent paper using this templateMohammad Akhlaghi-0/+7
Raul's paper (that uses this template) was just published on arXiv today (congratulations Raul!). So it has been added to the list of papers using this template.
2019-10-30Added ./project prepare in the checklist of README-hacking.mdMohammad Akhlaghi-2/+3
Since adding this new step, I had forgot to mention it in the checklist of `README-hacking.md'. It is added with this commit.
2019-10-29Minor edits to README-hacking.mdMohammad Akhlaghi-28/+31
The part on using shared memory was edited for a few things that weren't clear.
2019-10-29Further small edit in using shared memory of README-hacking.mdMohammad Akhlaghi-3/+2
Some typos were fixed.
2019-10-29Minor edits to suggestion on using shared memoryMohammad Akhlaghi-3/+9
The edits help it be more clear, and remind the reader to delete any remaining file in the RAM in the end.
2019-10-29Suggestion on good usage of /dev/shm in README-hacking.mdMohammad Akhlaghi-6/+57
When you are working with large files and there is some good RAM in the system (large/powerful computers), it is beneficial to work in the shared memory directory and not the actual persistent storage (like HDD or SSD). With this commit, a fully working demo has been added to `README-hacking.md' (under the tips of "Make programming") to show how to effectively work in situations like this.
2019-10-01Preparation phase added before final buildingMohammad Akhlaghi-35/+53
In many real-world scenarios, `./project make' can really benefit from having some basic information about the data before being run. For example when quering a server. If we know how many datasets were downloaded and their general properties, it can greatly optmize the process when we are designing the solution to be run in `./project make'. Therefore with this commit, a new phase has been added to the template's design: `./project prepare'. In the raw template this is empty, because the simple analysis done in the template doesn't warrant it. But everything is ready for projects using the template to add preparation phases prior to the analysis.
2019-09-26Minor edit in README-hacking.mdMohammad Akhlaghi-3/+2
The description of arXiv:1909.11230 was slightly modified to be in the same format as the other papers.
2019-09-26arXiv:1909.11230 added as example paper using this templateMohammad Akhlaghi-4/+11
This paper was published on arXiv today and is a good example for people to see the application of this system in practice.
2019-09-15Minor edits on the git bundle suggestion of README-hacking.mdMohammad Akhlaghi-8/+9
After a re-read on Gitlab, it has been slightly edited to be more clear.
2019-09-15Added tip on bundling Git history in one fileMohammad Akhlaghi-0/+24
When you want to publish your project, it is very convenient to have a single file that contains the whole history. So a tip is added to `README-hacking.md' that describes how to do this with `git bundle'.
2019-08-22Paper's title and author information moved to main paper.texMohammad Akhlaghi-6/+8
Until now, the paper's title and author information were set it `tex/src/preamble-header.tex'. But they are actually shown in the final PDF paper and a much better place to keep them is the top-level `paper.tex'. With this commit, the setting of the title and author names has been moved to `paper.tex', just after importing all the preambles. However, the basic package importation and low-level settings are still set in `tex/src/preamble-header.tex', because they are relatively low-level. This task was suggested by Deepak (Indian Institute of Astrophysics).
2019-08-20Corrected typo in README-hacking.mdMohammad Akhlaghi-1/+1
Until now, when describing the sections to remove for customizing a project, I had mistakenly repeated the `%% Start of main body.' statement. With this commit, the second one is changed to `%% End of main body.' This issue was reported by Deepak.
2019-08-18Updated README-hacking.md's checklist for better usabilityMohammad Akhlaghi-122/+156
After the checklist was applied in the 5th Indo-French Astronomy School, we found some cases in the checklist that were extra (and thus had to be removed), or were needed (and thus were added). Also the non-necessary steps for a first commit were moved to a separate/new section in the checklist for the people to add after doing their first commit. Also, the software part of the paper was moved to an appendix.
2019-08-01Bash startup script for every recipeMohammad Akhlaghi-0/+8
Until now the only way to define the environment of the Make recipes was through the exported Make variables (mostly in `initialize.mk' for the analysis steps for example). However, there is only so much you can do with environment variables! In some situations you want slightly more complicated environment control, like setting an alias or running of scripts (things that are commonly done in the `~/.bashrc' file of users to configure their interactive, non-login shells). With this commit, a `reproduce/software/bash/bashrc.sh' has been defined for this job (which is currently empty!). Every major Make step of the project adds this file as the `BASH_ENV' environment variable, so the shell that is created to execute a recipe first executes this file, then the recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment variable that enables users to limit their envirnoment setup based on the condition it is being setup (in particular in the early phase of `basic.mk', where the user can't make any assumption about the programs and has to write a portable shell script).
2019-07-29Added explanation on .file-metadata within README-hacking.mdMohammad Akhlaghi-0/+36
Until now there was no clear explanation on `.file-metadata' within the project. But since it sometimes appears in the Git changed files and its binary, it was necessary to add a short explanation to inform users. With this commit a section has been added to the "Project architecture" section of `README-hacking.md' to give some context on what it is and how to deal with it. This was suggested by Hamed Altafi.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-72/+68
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.