aboutsummaryrefslogtreecommitdiff
path: root/README-hacking.md
AgeCommit message (Collapse)AuthorLines
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-2/+2
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-23IMPORTANT: Project preparation is now also done with project makeMohammad Akhlaghi-21/+12
Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system.
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-33/+33
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19New --check-config option to ./project to check software build statusMohammad Akhlaghi-31/+14
Until now, it was necessry to run a long `while true' loop to see what is currently being built at configure time. So with this commit, a new `--checkconfig' option has been added to `./project' that can be called to run that loop and make it easier to check.
2020-01-18README-hacking.md: edits and corrections for easier customizationMohammad Akhlaghi-8/+10
The checklist descriptions were slightly edited to be more clear. Also, while following them, I noticed that while removing the "delete-me" parts on `verify.mk', would cause an error: the `if [ $$m == delete-me ];' statement we were saying to delete cause an error because `elif' was the first statement Bash would see. So with this commit, the `download' conditional (which isn't instructed to be deleted) was set to be the top (with an `if') and the `delete-me' conditional now has an `elif'.
2020-01-17README-hacking.md: script to list installed programs before configureMohammad Akhlaghi-10/+18
Until now, the small one-line script that lists programs was introduced in the checklist after running `./project configure'. But people would mostly miss it because they would wait until the configuration is complete. With this commit, that point has been put above the `./project configure' step. Readers are instructed to open a new terminal and run that script, then go to the next step so they see the directories get filled actively. It will also help them understand what is going on.
2020-01-13Minor corrections in referencing Infante-Sainz+2019Mohammad Akhlaghi-6/+5
Until now the actual journal webpage was used for Raul's paper. However, the journal webpage needs authorized access for people to read it, therefore its will be inaccessible for many people. A better and more well known place for the paper (atleast in astronomy) is the ADS link. In the ADS link, if someone has access to the journal, they will get the journal's version and if not, they will get the arXiv version. It also has a common BibTeX export tool for all journals. We had also done this for the other papers in that list. With this commit, I thus changed the URL for the paper, and also removed the "issue" number (4 in this case), since that is mostly irrelevant, only the volume and page numbers are usually used for the other papers too.
2020-01-13Added Infante-Sainz et al. 2019 as most recent paper using this templateRaul Infante-Sainz-3/+4
The "SDSS extended PSFs" paper was already included as an example of papers wich uses this template. However, the reference was the arXiv one. With this commit, since the paper has been finally published, it has been added the final reference to the journal.
2020-01-01Verification function checks if file existsMohammad Akhlaghi-5/+7
Until now, if the file to be verified didn't exist, a different checksum would be generated, and it would stop, but it wasn't immediately clear if the differing checksum is because the file doesn't exist at all! With this commit, before calculating the checksum, we first make sure if the file exists. If it doesn't exist an explicit error is printed and thus will help the project editor to find the cause of the problem.
2020-01-01Minor corrections in `README-hacking.md' after verificationMohammad Akhlaghi-4/+7
In the previous commit, I had forgot to update a small part in the checklist (when modifying `top-make.mk') which is now corrected. I also added a few sentences in the description of how to customize the verification to make it easier to understand.
2020-01-01Verification of output values and data added within templateMohammad Akhlaghi-9/+51
Until now, the only verification that the template provided was the published PDF. Users had to manually compare the published and generated PDFs (numbers, plots, tables) and see if they obtained the same result. However, this type of manual verification is not good and is prone to frustration and missing important differences. With this commit, a new Makefile has been added in the analysis steps: `verify.mk'. It provides facilities to easily verify the results that go into the paper. For example tables that go into making the paper's plots, or the LaTeX macros that blend into the text. See the updated parts in `README-hacking.md` for a more complete explanation. This completes task #15497.
2020-01-01README-hacking.md checklist now also ignores changes in paper.texMohammad Akhlaghi-7/+12
In the previous commit, we added the files to ignore from the template branch, but only the files that had been deleted. With this commit, `paper.tex' is also added to the files that must be ignored from the template branch (the file remains in the project, but in the template branch, its contents are just dummy place-holders).
2020-01-01Added step README-hacking's checklist to avoid merging dummy filesMohammad Akhlaghi-0/+15
During the checklist we guide the user to delete the dummy `delete-me*' files from their custom branch. Later, if the dummy files are updated in the template's master branch, if the user merges with the template branch, these files will be written back into their project! This is very annoying! With this commit, a step was added in the `README-hacking.md' checklist, just after deleting the dummy files to avoid this problem using the `.gitattributes' file, telling Git to keep the changes as implemented in the merging branch (`ours').
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-3/+3
Now that its 2020, its necessary to include this year in the copyright statements.
2019-11-06Added 1911.01430 as most recent paper using this templateMohammad Akhlaghi-0/+7
Raul's paper (that uses this template) was just published on arXiv today (congratulations Raul!). So it has been added to the list of papers using this template.
2019-10-30Added ./project prepare in the checklist of README-hacking.mdMohammad Akhlaghi-2/+3
Since adding this new step, I had forgot to mention it in the checklist of `README-hacking.md'. It is added with this commit.
2019-10-29Minor edits to README-hacking.mdMohammad Akhlaghi-28/+31
The part on using shared memory was edited for a few things that weren't clear.
2019-10-29Further small edit in using shared memory of README-hacking.mdMohammad Akhlaghi-3/+2
Some typos were fixed.
2019-10-29Minor edits to suggestion on using shared memoryMohammad Akhlaghi-3/+9
The edits help it be more clear, and remind the reader to delete any remaining file in the RAM in the end.
2019-10-29Suggestion on good usage of /dev/shm in README-hacking.mdMohammad Akhlaghi-6/+57
When you are working with large files and there is some good RAM in the system (large/powerful computers), it is beneficial to work in the shared memory directory and not the actual persistent storage (like HDD or SSD). With this commit, a fully working demo has been added to `README-hacking.md' (under the tips of "Make programming") to show how to effectively work in situations like this.
2019-10-01Preparation phase added before final buildingMohammad Akhlaghi-35/+53
In many real-world scenarios, `./project make' can really benefit from having some basic information about the data before being run. For example when quering a server. If we know how many datasets were downloaded and their general properties, it can greatly optmize the process when we are designing the solution to be run in `./project make'. Therefore with this commit, a new phase has been added to the template's design: `./project prepare'. In the raw template this is empty, because the simple analysis done in the template doesn't warrant it. But everything is ready for projects using the template to add preparation phases prior to the analysis.
2019-09-26Minor edit in README-hacking.mdMohammad Akhlaghi-3/+2
The description of arXiv:1909.11230 was slightly modified to be in the same format as the other papers.
2019-09-26arXiv:1909.11230 added as example paper using this templateMohammad Akhlaghi-4/+11
This paper was published on arXiv today and is a good example for people to see the application of this system in practice.
2019-09-15Minor edits on the git bundle suggestion of README-hacking.mdMohammad Akhlaghi-8/+9
After a re-read on Gitlab, it has been slightly edited to be more clear.
2019-09-15Added tip on bundling Git history in one fileMohammad Akhlaghi-0/+24
When you want to publish your project, it is very convenient to have a single file that contains the whole history. So a tip is added to `README-hacking.md' that describes how to do this with `git bundle'.
2019-08-22Paper's title and author information moved to main paper.texMohammad Akhlaghi-6/+8
Until now, the paper's title and author information were set it `tex/src/preamble-header.tex'. But they are actually shown in the final PDF paper and a much better place to keep them is the top-level `paper.tex'. With this commit, the setting of the title and author names has been moved to `paper.tex', just after importing all the preambles. However, the basic package importation and low-level settings are still set in `tex/src/preamble-header.tex', because they are relatively low-level. This task was suggested by Deepak (Indian Institute of Astrophysics).
2019-08-20Corrected typo in README-hacking.mdMohammad Akhlaghi-1/+1
Until now, when describing the sections to remove for customizing a project, I had mistakenly repeated the `%% Start of main body.' statement. With this commit, the second one is changed to `%% End of main body.' This issue was reported by Deepak.
2019-08-18Updated README-hacking.md's checklist for better usabilityMohammad Akhlaghi-122/+156
After the checklist was applied in the 5th Indo-French Astronomy School, we found some cases in the checklist that were extra (and thus had to be removed), or were needed (and thus were added). Also the non-necessary steps for a first commit were moved to a separate/new section in the checklist for the people to add after doing their first commit. Also, the software part of the paper was moved to an appendix.
2019-08-01Bash startup script for every recipeMohammad Akhlaghi-0/+8
Until now the only way to define the environment of the Make recipes was through the exported Make variables (mostly in `initialize.mk' for the analysis steps for example). However, there is only so much you can do with environment variables! In some situations you want slightly more complicated environment control, like setting an alias or running of scripts (things that are commonly done in the `~/.bashrc' file of users to configure their interactive, non-login shells). With this commit, a `reproduce/software/bash/bashrc.sh' has been defined for this job (which is currently empty!). Every major Make step of the project adds this file as the `BASH_ENV' environment variable, so the shell that is created to execute a recipe first executes this file, then the recipe. Each top-level Makefile also defines a `PROJECT_STATUS' environment variable that enables users to limit their envirnoment setup based on the condition it is being setup (in particular in the early phase of `basic.mk', where the user can't make any assumption about the programs and has to write a portable shell script).
2019-07-29Added explanation on .file-metadata within README-hacking.mdMohammad Akhlaghi-0/+36
Until now there was no clear explanation on `.file-metadata' within the project. But since it sometimes appears in the Git changed files and its binary, it was necessary to add a short explanation to inform users. With this commit a section has been added to the "Project architecture" section of `README-hacking.md' to give some context on what it is and how to deal with it. This was suggested by Hamed Altafi.
2019-07-28Single wrapper instead of old ./configure, Makefile and ./for-groupMohammad Akhlaghi-72/+68
Until now, to work on a project, it was necessary to `./configure' it and build the software. Then we had to run `.local/bin/make' to run the project and do the analysis every time. If the project was a shared project between many users on a large server, it was necessary to call the `./for-group' script. This way of managing the project had a major problem: since the user directly called the lower-level `./configure' or `.local/bin/make' it was not possible to provide high-level control (for example limiting the environment variables). This was especially noticed recently with a bug that was related to environment variables (bug #56682). With this commit, this problem is solved using a single script called `project' in the top directory. To configure and build the project, users can now run these commands: $ ./project configure $ ./project make To work on the project with other users in a group these commands can be used: $ ./project configure --group=GROUPNAME $ ./project make --group=GROUPNAME The old options to both configure and make the project are still valid. Run `./project --help' to see a list. For example: $ ./project configure -e --host-cc $ ./project make -j8 The old `configure' script has been moved to `reproduce/software/bash/configure.sh' and is called by the new `./project' script. The `./project' script now just manages the options, then passes control to the `configure.sh' script. For the "make" step, it also reads the options, then calls Make. So in the lower-level nothing has changed. Only the `./project' script is now the single/direct user interface of the project. On a parallel note: as part of bug #56682, we also found out that on some macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set to blank. This is no problem because RPATH is automatically set in macOS and the executables and libraries contain the absolute address of the libraries they should link with. But having `DYLD_LIBRARY_PATH' can conflict with some low-level system libraries and cause very hard to debug linking errors (like that reported in the bug report). This fixes bug #56682.
2019-07-26Better explanation of suggested commit messages in REAME-hacking.mdMohammad Akhlaghi-19/+25
Until now the description of the commit message guidelines wasn't clear enough and could cause confusion, in particular because it didn't describe why some basic formatting issues are mandatory. With this commit, it is explained that the main reason we require contributors for follow this format is "consistency" within the project. Also generally it was edited to become more elaborate and explain the points more clearly. I also ran a spell check over the whole file and fixed a few typos. This correction was suggested by Mohammad-reza Khellat.
2019-07-07Commit message format tip added to README-hacking.mdMohammad Akhlaghi-0/+16
Until now there was no guideline in `README-hacking.md' to describe/suggest a good format for commit messages. With this commit a point has been added in the "Tips" section to help new developers contribute more smoothly. The necessity of this paragraph was pointed out by Mohammad-reza Khellat.
2019-06-14Minor correction in README-hacking.md for better web-renderingMohammad Akhlaghi-5/+5
The new command-box wasn't being rendered properly, so another correction is made here. I also added the prompt `$' sign in another box of commands.
2019-06-14Minor correction is code demonstration in previous commitMohammad Akhlaghi-1/+2
After checking the previous commit on Gitlab (to see how it is rendered), I noticed that the code has come in the same line, not as a sperate box. Hopefully this commit will fix it.
2019-06-14Software building status check tip added to README-hacking.mdMohammad Akhlaghi-0/+18
It is useful to visually see how the building of software is progressing when running configure. I have been using a simple Bash `while' loop for this, so I added it in the `README-hacking.md' to be useful for others too.
2019-05-21README-hacking.md: corrected typoMohammad Akhlaghi-1/+1
A title in the checklist was mistakenly using "project" (customized template) instead of "template".
2019-05-02Updated to TeXLive 2019Mohammad Akhlaghi-0/+10
Since we just download the binary source of TeXLive, we need to keep it up to date with the server. So it has been incremented to 2019 (TeXLive 2019 was released April 29th). A note was also added in the Checklist to keep the users informed on how to update TeXLive if necessary.
2019-04-22High-level software now specified in TARGETS.mkMohammad Akhlaghi-26/+18
Until now, to specify which high-level software you want the project to contain, it was necessary to go into the `high-level.mk' Makefile that is complicated and can create bugs. With this commit, a new `reproduce/software/config/installation/TARGETS.mk' file has been created that is easily/cleanly in charge of documenting the final high-level software that must be built for the project. Also, until now, FFTW was set as a dependency of Numpy while we couldn't actually get Numpy to use it! It was just there for future reference and to justify its build rule. But now that many software won't be built and there is no problem with having rules even though a project might not use them, it has been removed.
2019-04-15Minor typo corrections in README-hacking.mdMohammad Akhlaghi-3/+7
In two places, I had mistakenly put a <'> instead of a <`>, causing bad highlighting in the markdown rendering. They have been corrected. Also, one long line in was broken up into several.
2019-04-15Correction in customization checklistMohammad Akhlaghi-22/+22
Until now, the customization checklist of `README-hacking.md' had the same name for the base template's remote and branch. This was confusing and would cause Git to print a warning. With this commit, like before, the template's remote is now called `template-origin', and `template' is only the branch name.
2019-04-15Template's main branch is now masterMohammad Akhlaghi-25/+44
Until now, the main template branch was called `template'. However, the standard Git convention is that the main branch of a project be called `master'. Many systems rely on this default and it is also easier for new users (who have been accustomed to this convention). So with this commit, the main template branch is `master', but in `README-hacking.mk', we instruct the users on how to rename it to `template' as part of their customization. This is infact better, because when we are actually developing the template in a separate fork, we can refer/use the `master' branch like any other project. And when we are working on a project that uses this template, we will be referring to the main template branch as `template'.
2019-04-15New architecture to separate software-building and analysis stepsMohammad Akhlaghi-125/+133
Until now, the software building and analysis steps of the pipeline were intertwined. However, these steps (of how to build a software, and how to use it) are logically completely independent. Therefore with this commit, the pipeline now has a new architecture (particularly in the `reproduce' directory) to emphasize this distinction: The `reproduce' directory now has the two `software' and `analysis' subdirectories and the respective parts of the previous architecture have been broken up between these two based on their function. There is also no more `src' directory. The `config' directory for software and analysis is now mixed with the language-specific directories. Also, some of the software versions were also updated after some checks with their webpages. This new architecture will allow much more focused work on each part of the pipeline (to install the software and to run them for an analysis).
2019-04-14Replaced all occurances of pipeline in textMohammad Akhlaghi-242/+238
All occurances of "pipeline" have been chanaged to "project" or "template" withint the text (comments, READMEs, and comments) of the template. The main template branch is now also named `template'. This was all because `pipeline' is too generic and couldn't be distinguished from the base, and customized project.
2019-04-13Corrected copyright notices and info about adding copyright infoMohammad Akhlaghi-20/+23
Until now, the files where the people were meant to change didn't have a proper copyright notice (for example `Copyright (C) YOUR NAME.'). This was wrong because the license does not convey copyright ownership. So the name of the file's original author must always be included and when people modify it (and add their own copyright-able modifications). With this commit, the file's original author (and email) are added to the copyright notice and when more than one person modified a file, both names have their individual copyright notice. Based on this, the description for adding a copyright notice in `README-hacking.md' has also been modified.
2019-04-11.file-metadata also given a copyright in top READMEMohammad Akhlaghi-16/+16
Since `.file-metadata' is a binary file and we couldn't put a copyright notice within it, it has been mentioned in `README.md' to have the same copyright. Also, the copyright modification step in `README-hacking.md' was brought to a later step to be more clear that it should always be done (on new files or files that are changed).
2019-04-07Copyright notice added to all files missing oneMohammad Akhlaghi-10/+12
Until now, for short files, we only had a license notice, not an actual copyright notice. With this commit, a copyright notice has also been added. We use this new command to find these files, suggested by `ineiev@gnu.org'.
2019-04-04Better option-reading in configure, using .build to access BDIRMohammad Akhlaghi-76/+70
Until now, the steps to manage the command-line options of the configure script were limited (couldn't accept an equal sign or space between the option name and value). With this commit, it can now also accept optional equal signs between the option name and value. Thus not causing many confusions. Also, it is more logically consistent for the link to the build-directory to be placed in the top directory (as a hidden file like `.local' until now), and not as a visible directory like `reproduce/build' (which we used until now). Therefore, with this commit, the link to easily access the build-directory is `.build' in the top source directory. Finally, because `minmapsize' is too specific to Gnuastro and has now been given its default value at the start of the configure script, the description for `minmapsize' has been removed (to not confuse users who don't use Gnuastro). If anyone is familiar enough with Gnuastro to change it, they already know it from its book.
2019-03-29Added Copyright to all TeX and README filesMohammad Akhlaghi-0/+25
In order to be more clear, a copyright statement was added to all the LaTeX and README files.
2019-03-01Elaboration in README-hacking.mk's future improvements sectionMohammad Akhlaghi-30/+43
This section was a little outdated and since then, a more clear/exact image of using the Nix experience for the reproducible paper template has been added.