aboutsummaryrefslogtreecommitdiff
path: root/README-hacking.md
AgeCommit message (Collapse)AuthorLines
2023-05-07Copyright years: updated to 2023, accompanied by some minor fixesMohammad Akhlaghi-4/+4
SUMMARY: just house-cleaning, no need to do anything major in your branch. Just update the copyright years in files that you have added. Until now, the latest copyright years of the whole Maneage source code was 2022! As of this commit, we have already moved to 2023 for 5 months! Furthermore, there were a few other minor issues that needed correction: - The URL to download input datasets wasn't quoted in 'initialize.mk' or the download script! As a result, when the input URL had characters that are meaningful to the shell (like '&'), the download command would not work. - The only program that had 'make check' in the 'basic.mk' programs was MPFR. At that stage, we still haven't built our own compiler at this stage, this is not accurate. - The 'pyerfa' and 'extension-helpers' packages in Python need 'setuptools_scm' on some systems. But until now, it was not in the list of their prerequisites. With this commit, all the issues above have been corrected.
2022-06-11IMPORTANT: download.mk removed, content moved to initialize.mkMohammad Akhlaghi-14/+13
SUMMARY: no special action should be necessary; but its an important update in low-level Maneage infra-structure (related with downloading and setting input checksums). Until now, we had a separate 'download.mk' as one of the default sub-Makefiles that should have been loaded in all the 'top-*.mk' files after 'initialize.mk'. This was due to historic reasons: until Commit 91799fe4b6d, we had to manually make some changes in 'download.mk' for every input file we defined in 'INPUTS.mk' (which was very inconvenient, and not easily possible for a large number of files!). But since Commit 91799fe4b6d, those manual changes are no longer necessary, and a normal user will hardly ever need to touch the contents of 'download.mk' (which also had one effective rule). Furthermore, based on shared projects with Zohre Ghaffari and Sepideh Eskandarlou (which involved a large number of large files), we recognized that it is very inconvenient to download a file once, update its checksum, and re-run Maneage (so the validation works). A robust solution was necesary to let project authors download the data and automatically update the checksum. With this commit, to help in high-level project management in Maneage, the single, and generic rule of 'download.mk' has been moved to 'initialize.mk', enabling us to fully remove this extra sub-Makefile from Maneage's source. Furthermore, with this commit, a usable solution to the automatic updating of the checksum has also been implemented (which has been described in the comments of 'INPUTS.conf'): the users can now set the checksum to '--auto-replace--'. In this case, the download rule (now in 'initialize.mk') will automatically update that line of 'INPUTS.conf' and add the checksum instead. After './project make' is complete, when the user runs 'git diff', they can see all the updated checksums in the source of their project and commit the updated 'INPUTS.conf' into the source so this will not be necessary later. Two other smaller issues have also been addressed in this commit: - There was an extra ',' in the call to 'filter-out' when we defined 'prepare-dep' in 'reproduce/analysis/make/prepare.mk'. This would cause a crash (with Make complaining that there is no rule for target 'initialize.mk,': notice the extra ','). With this commit, that extra ',' has been removed and the problem was solved. - The build recipe of Imfit (in 'reproduce/software/make/high-level.mk'), had two SPACE characters after '--no-openmp' which would make the reading hard. They have been updated to one SPACE.
2022-05-10initialize.mk: Git call in variable works with LD_LIBRARY_PATHMohammad Akhlaghi-11/+10
Until now, the '$(project-commit-hash)' Make variable of 'initialize.mk' simply called 'git' to find the commit hash. However, due to one of the recent software updates, we noticed that this command is no longer working (and the project commit hash wasn't getting printed in the PDF)! The problem was that Maneage's Git, couldn't find the 'libiconv' library that it was built with. With this commit, the '$(shell' command that calls Git, first exports 'LD_LIBRARY_PATH' to Maneage's software build directory. As a result, the Git command can work and will report the commit as a LaTeX macro to be used in the paper. To avoid relying on PATH outside of Make recipes, we now also directly call the Git executable with Maneage. Some other minor issues have been found and fixed in this commit: - README-hacking.md: some minor edits and typo corrections. - initialize.mk: the '$(curdir)' variable is now used in several places that we were calling 'pwd'. - versions.conf: 'xlsxio-version' now included with other programs. Until now it was commented because GCC 11.1.0 had issues with it. However, GCC 11.2.0 doesn't have a problem any more, so it has been returned to the list of all high-level programs. - xorg.mk: used same format to comment recipe lines as the other Makefiles (a '#' followed by a TAB). - preamble-pgfplots.tex: lines to comment for building an EPS figure with PGFPlots have been re-formatted to be more human-readable.
2022-04-20Updated Git, Coreutils and Emacs, new script to prepare tarballsPedram Ashofteh Ardakani-1/+1
Until now, one had to follow the instructions from [1] to prepare a standard software tarball before merging with the low-level tarballs-software repository [2]. The script only worked for '.tar.gz' suffix and was only available as a comment on Savannah (in [1]). With this commit, the script has been imported into Maneage as 'reproduce/software/shell/tarball-prepare.sh' to simplify future software updates. It work with all supported '.tar.*' suffixes (of the upstream tarball repository) and will convert the tarballs to Maneage's standard format. Also, this script has a minimal argument parser and can skip the tarballs that are already unpacked, allowing faster tests. This script was used to update the versions of: Coreutiles 9.0 --> 9.1 Git 2.34 --> 2.36 Emacs 27.2 --> 28.1 The main motive behind this update was Git which announced a vulnerability issue [3] and suggested an update to the latest version as soon as possible. More detail is described in this github blog [4], but in summary, it was a security issue on multi-user systems that has been found and fixed by Git developers. Since Maneage is often installed on such shared systems, it was important to make this update. GNU Coreutils and GNU Emacs were also updated because they are also commonly used. The following improvements have also done with this commit: - .gitignore: ignore emacs auto-save files (that end with a '#') - README-hacking.md: In the checklist for updating the Maneage branch, the no-longer-necessary '--decorate' option of Git was removed from the command to check the general branch history. [1] https://savannah.nongnu.org/task/?15699 [2] https://git.maneage.org/tarballs-software.git/ [3] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/ [4] https://github.blog/2022-04-12-git-security-vulnerability-announced/
2022-04-15IMPORTANT: more generic, robust and secure INPUTS.conf and download.mkMohammad Akhlaghi-38/+27
SUMMARY: it is necessary to update your 'INPUTS.conf' and 'download.mk'. Until now, adding an input file involved several steps that needed manual (and inconvenient!) intervention: for every file, you needed to define four variables in 'INPUTS.conf', and in 'reproduce/analysis/make/download.mk' you had to use a (complex for large number of files) shell 'if/elif/else' condition to link the names of the input files to those variables. Besides inconvenience, this could cause bugs (typos!). Furthermore, a basic MD5 checksum was used for verifying the files. With this commit, a new structure has been defined for 'INPUTS.conf' that (thanks to some pretty useful GNU Make features), removes the need for users to manually edit 'reproduce/analysis/make/download.mk', and reduces the number of variables necessary for each file to three (from four). Furthermore, we now use the SHA256 checksum for input data validation. Regarding the trick used in 'INPUTS.conf' (form the newly added description in 'download.mk'): In GNU Make, '.VARIABLES' "... expands to a list of the names of all global variables defined so far" (from the "Other Special Variables" section of the GNU Make manual). Assuming that the pattern 'INPUT-%-sha256' is only used for input files, we find all the variables that contain the input file names (the '%' is the filename). Finally, using the pattern-substitution function ('patsubst'), we remove the fixed string at the start and end of the variable name. Steps you need to take: - INPUTS.conf: translate your old format to the new format (after carefully reading the description in the comments at the start of the file). After applying the new standards, you don't need to use the variables of 'INPUTS.conf' directly in your Makefiles! For example if one of your input datasets is called 'abc.fits', the checksum variable will be 'INPUT-abc.fits-sha256' and in your high-level Makefiles, you can simply set '$(indir)/abc.fits' as a prerequisite (like you probably did already). - reproduce/analysis/make/download.mk: for the definition and rule of 'inputdatasets', simply use the Maneage branch, and remove anything you had added in your project. In the process, I also noticed that 'README-hacking.md' still referred to 'master' as the main project branch, while we have used 'main' in the paper (and is the common convention with Git).
2022-01-21IMPORTANT: Updates to almost all softwareMohammad Akhlaghi-22/+33
This commit primarily affects the configuration step of Maneage'd projects, and in particular, updated versions of the many of the software (see P.S.). So it shouldn't affect your high-level analysis other than the version bumps of the software you use (and the software's possibly improve/changed behavior). The following software (and thus their dependencies) couldn't be updated as described below: - Cryptography: isn't building because it depends on a new setuptools-rust package that has problems (https://savannah.nongnu.org/bugs/index.php?61731), so it has been commented in 'versions.conf'. - SecretStorage: because it depends on Cryptography. - Keyring: because it depends on SecretStorage. - Astroquery: because it depends on Keyring. This is a "squashed" commit after rebasing a development branch of 60 commits corresponding to a roughly two-month time interval. The following people contributed to this branch. - Boudewijn Roukema added all the R software infrastructure and the R packages, as well as greatly helping in fixing many bugs during the update. - Raul Infante-Sainz helped in testing and debugging the build. - Pedram Ashofteh Ardakani found and fixed a bug. - Zahra Sharbaf helped in testing and found several bugs. Below a description of the most noteworthy points is given. - Software tarballs: all updated software now have a unified format tarball (ustar; if not possible, pax) and unified compression (Lzip) in Maneage's software repository in Zenodo (https://doi.org/10.5281/zenodo.3883409). For more on this See https://savannah.nongnu.org/task/?15699 . This won't affect any extra software you would like to add; you can use any format recognized by GNU Tar, and all common compression algorithms. This new requirement is only for software that get merged to the core Maneage branch. - Metastore (and thus libbsd and libmd) moved to highlevel: Metastore (and the packages it depends on) is a high-level product that is only relevant during the project development (like Emacs!): when the user wants the file meta data (like dates) to be unchanged after checking out branches. So it should be considered a high-level software, not basic. Metastore also usually causes many more headaches and error messages, so personally, I have stopped using it! Instead I simply merge my branches in a separate clone, then pull the merge commit: in this way, the files of my project aren't re-written during the checkout phase and therefore their dates are untouched (which can conflict with Make's dates on configuration files). - The un-official cloned version of Flex (2.6.4-91 until this commit) was causing problems in the building of Netpbm, so with this commit, it has been moved back to version 2.6.4. - Netpbm's official page had version 10.73.38 as the latest stable tarball that was just released in late 2021. But I couldn't find our previously-used version 10.86.99 anywhere (to see when it was released and why we used it! Its at last more than one year old!). So the official stable version is being used now. - Improved instructions in 'README.md' for building software environment in a Docker container (while having project source and output data products on the local system; including the usage of the host's '/dev/shm' to speed up temporary operations). - Until now, the convention in Maneage was to put eight SPACE characters before the comment lines within recipes. This was done because by default GNU Emacs (also many other editors) show a TAB as eight characters. However, in other text editors, online browsers, or even the Git diff, a TAB can correspond to a different number of characters. In such cases, the Maneage recipes wouldn't look too interesting (the comments and the recipe commands would show a different indentation!). With this commit, all the comment lines in the Makefiles within the core Maneage branch have a hash ('#') as their first character and a TAB as the second. This allows the comment lines in recipes to have the same indentation as code; making the code much more easier to read in a general scenario including a 'git diff' (editor agnostic!). P.S. List of updated software with their old and new versions - Software with no version update are not mentioned. - The old version of newly added software are shown with '--'. Name (Basic) Old version New version ------------ ----------- ----------- Bzip2 1.0.6 1.0.8 CURL 7.71.1 7.79.1 Dash 0.5.10.2 0.5.11.5 File 5.39 5.41 Flock 0.2.3 0.4.0 GNU Bash 5.0.18 5.1.8 GNU Binutils 2.35 2.37 GNU Coreutils 8.32 9.0 GNU GCC 10.2.0 11.2.0 GNU M4 1.4.18 1.4.19 GNU Readline 8.0 8.1.1 GNU Tar 1.32 1.34 GNU Texinfo 6.7 6.8 GNU diffutils 3.7 3.8 GNU findutils 4.7.0 4.8.0 GNU gmp 6.2.0 6.2.1 GNU grep 3.4 3.7 GNU gzip 1.10 1.11 GNU libunistring 0.9.10 1.0 GNU mpc 1.1.0 1.2.1 GNU mpfr 4.0.2 4.1.0 GNU nano 5.2 6.0 GNU ncurses 6.2 6.3 GNU wget 1.20.3 1.21.2 Git 2.28.0 2.34.0 Less 563 590 Libxml2 2.9.9 2.9.12 Lzip 1.22-rc2 1.22 OpenSLL 1.1.1a 3.0.0 Patchelf 0.10 0.13 Perl 5.32.0 5.34.0 Podlators -- 4.14 Name (Highlevel) Old version New version ---------------- ----------- ----------- Apachelog4cxx 0.10.0-603 0.12.1 Astrometry.net 0.80 0.85 Boost 1.73.0 1.77.0 CFITSIO 3.48 4.0.0 Cmake 3.18.1 3.21.4 Eigen 3.3.7 3.4.0 Expat 2.2.9 2.4.1 FFTW 3.3.8 3.3.10 Flex 2.6.4-91 2.6.4 Fontconfig 2.13.1 2.13.94 Freetype 2.10.2 2.11.0 GNU Astronomy Utilities 0.12 0.16.1-e0f1 GNU Autoconf 2.69.200-babc 2.71 GNU Automake 1.16.2 1.16.5 GNU Bison 3.7 3.8.2 GNU Emacs 27.1 27.2 GNU GDB 9.2 11.1 GNU GSL 2.6 2.7 GNU Help2man 1.47.11 1.48.5 Ghostscript 9.52 9.55.0 ICU -- 70.1 ImageMagick 7.0.8-67 7.1.0-13 Libbsd 0.10.0 0.11.3 Libffi 3.2.1 3.4.2 Libgit2 1.0.1 1.3.0 Libidn 1.36 1.38 Libjpeg 9b 9d Libmd -- 1.0.4 Libtiff 4.0.10 4.3.0 Libx11 1.6.9 1.7.2 Libxt 1.2.0 1.2.1 Netpbm 10.86.99 10.73.38 OpenBLAS 0.3.10 0.3.18 OpenMPI 4.0.4 4.1.1 Pixman 0.38.0 0.40.0 Python 3.8.5 3.10.0 R 4.0.2 4.1.2 SWIG 3.0.12 4.0.2 Util-linux 2.35 2.37.2 Util-macros 1.19.2 1.19.3 Valgrind 3.15.0 3.18.1 WCSLIB 7.3 7.7 Xcb-proto 1.14 1.14.1 Xorgproto 2020.1 2021.5 Name (Python) Old version New version ------------- ----------- ----------- Astropy 4.0 5.0 Beautifulsoup4 4.7.1 4.10.0 Beniget -- 0.4.1 Cffi 1.12.2 1.15.0 Cryptography 2.6.1 36.0.1 Cycler 0.10.0 0.11.0+} Cython 0.29.21 0.29.24 Esutil 0.6.4 0.6.9 Extension-helpers -- 0.1 Galsim 2.2.1 2.3.3 Gast -- 0.5.3 Jinja2 -- 3.0.3 MPI4py 3.0.3 3.1.3 Markupsafe -- 2.0.1 Numpy 1.19.1 1.21.3 Packaging -- 21.3 Pillow -- 8.4.0 Ply -- 3.11 Pyerfa -- 2.0.0.1 Pyparsing 2.3.1 3.0.4 Pythran -- 0.11.0 Scipy 1.5.2 1.7.3 Setuptools 41.6.0 58.3.0 Six 1.12.0 1.16.0 Uncertainties 3.1.2 3.1.6 Wheel -- 0.37.0 Name (R) Old version New version -------- ----------- ----------- Cli -- 2.5.0 Colorspace -- 2.0-1 Cowplot -- 1.1.1 Crayon -- 1.4.1 Digest -- 0.6.27 Ellipsis -- 0.3.2 Fansi -- 0.5.0 Farver -- 2.1.0 Ggplot2 -- 3.3.4 Glue -- 1.4.2 GridExtra -- 2.3 Gtable -- 0.3.0 Isoband -- 0.2.4 Labeling -- 0.4.2 Lifecycle -- 1.0.0 Magrittr -- 2.0.1 MASS -- 7.3-54 Mgcv -- 1.8-36 Munsell -- 0.5.0 Pillar -- 1.6.1 R-Pkgconfig -- 2.0.3 R6 -- 2.5.0 RColorBrewer -- 1.1-2 Rlang -- 0.4.11 Scales -- 1.1.1 Tibble -- 3.1.2 Utf8 -- 1.2.1 Vctrs -- 0.3.8 ViridisLite -- 0.4.0 Withr -- 2.4.2
2021-06-03Update publications: Peper+Roukema publishedBoud Roukema-18/+22
This commit updates some of the publication data in README-hacking.md : Peper+Roukema (2021) is now published in MNRAS and Akhlaghi+ (2021) is published online and very close to getting a conventional volume and page number. :) See task https://savannah.nongnu.org/task/?15736 for ideas of how to make a more systematic publication list instead one managed by prose text. There are already too many non-automated places for publication lists where we have to copy/paste our publication data again and again and again and ... This commit also adds the softwareheritage ID that we have in the content of Akhlaghi+2021 (without the extra context, because as a URL that's very long). There are plenty of arguments to be made each way for different versions of the swh IDS. One advantage of the 'rev' ID is that the hash is the original (full) git hash, which is what I've done for the elaphrocentre and subpoisson papers.
2021-04-17IMPORTANT: print-general-metadata new name for print-copyrightMohammad Akhlaghi-64/+79
Summary: - Use the new name of this variable in your Makefiles. - In 'metadata.conf', remove fixed URL prefixes for DOIs ('https://doi.org/') or arXiv ('https://arxiv.org/abs'). Until now, the Make variable that would print the general metadata (of whole project) into each to-be-published dataset was called 'print-copyright'! But it now does much more than simply printing the copyright, it will also print a lot of metadata like arXiv ID, Zenodo DOI and etc into plain-text outputs. The out-dated name could thus be misleading and cause confusions. With this commit, the variable is therefore called 'print-general-metadata'. After merging your project with the Maneage branch, please replace any usage of 'print-copyright' to 'print-general-metadata'. Also with this commit, 'README-hacking.md' mentions 'metadata.conf' and 'print-general-metadata' in the "Publication checklist" section and reminds you to keep the first up to date, and use the second in your to-be-published datasets.
2021-01-04README-hacking.md: edits and improvements to publication checklistMohammad Akhlaghi-23/+40
After going through the publication checklist, some edits were made to make things more clear. Also, an item was added to remind the project author that the commit hashes on the uploaded data files should be the same.
2021-01-02Copyright year updated in all source filesMohammad Akhlaghi-4/+4
Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also.
2020-12-01README-hacking.md: recommended to push maneage after mergingMohammad Akhlaghi-4/+11
Until now at the end of the updating process, we hadn't explicity talked about pushing the branches. So people would usually only push their 'master' branch to their remote. While the merged 'master' branch does contain the commits from the core Maneage branch, having a no-updated 'maneage' branch reference on their remote can be confusing. With this commit, at the end of the process to merge with the 'maneage' branch we explicitly recommend to push both the 'master' and 'maneage' branches.
2020-11-28README-hacking.md: updated paper to cite for using ManeageMohammad Akhlaghi-4/+3
Until now, we were asking the users of Maneage to cite the first paper that used its primoridal version (arXiv:1505:01664). But there is now a paper that fully describes the concept (arXiv:2006.03018). With this commit, in the 'citation' section of 'README-hacking.md' we now ask to cite the new paper.
2020-10-09Update README-hacking.md with elaphrocentre ArXiv:2010.03742Boud Roukema-2/+11
This commit updates README-hacking.md with the URIs for the 'elaphrocentre' galaxy formation pipeline paper arXiv:2010.03742. This makes three papers currently in the peer review pipeline: arXiv:2006.03018, arXiv:2007.11779, and arXiv:2010.03742, each chronologically corresponding to various stages of the review process.
2020-09-24Gnuastro's analysis configuration files removedMohammad Akhlaghi-5/+3
Until now, the core Maneage branch included some configuration files for Gnuastro's programs. This was actually a remnant of the distant past when Maneage didn't actually build its own software and we had to rely on the host's software versions. This file contained the configuration files specific to Gnuastro for this project and also had a feature to avoid checking the host's own configuration files. However, we now build all our software ourselves with fixed configuration files (for the version that is being installed and its version is stored). So those extra configuration files were just extra and caused confusion and problems in some scenarios. With this commit, those extra files are now removed. Also, two small issues are also addressed in parallel with this commit: - When running './project make clean', the 'hardware-parameters.tex' macro file (which is created by './project configure' is not deleted. - The project title is now written into the default output's PDF's properties (through 'hypersetup' in 'tex/src/preamble-header.tex') through the LaTeX macro. All these issues were found and fixed with the help of Samane Raji.
2020-07-28README-hacking.md: added new paper using Maneage (arXiv:2007.11779)Boud Roukema-4/+15
Roukema+2020 (arXiv:2007.11779) is a newly published (as preprint) paper that uses Maneage, so it is being added to the list of published or submitted papers in 'README-hacking.md'. The Software Heritage URL sticks out way beyond the standard number of columns in the plain text form of the updated 'README-hacking.md' file, when rendered using markdown, it shouldn't look so bad. Also, see the related task https://savannah.nongnu.org/task/index.php?15736 (Raul+2020 should be Infante-Sainz+2020) for a suggestion of a more standard machine-readable format. It should be mentioned and emphasised to the reader that one should very carefully and obediently note and pay attention to the noteworthy fact that a few distracting words [1] such as "Note that" are removed in this commit. ;) [1] https://en.wiktionary.org/wiki/pontification
2020-07-20README-hacking.md: clarify Zenodo usage in publication checklistBoud Roukema-8/+23
This commit clarifies the initial usage of Zenodo for reserving a Zenodo identifier and starting an 'unpublished' upload. Some other minor wording changes are done here.
2020-06-06IMPORTANT: Added publication checklist, improved relevant infrastructureMohammad Akhlaghi-16/+330
Possible semantic conflicts (that may not show up as Git conflicts but may cause a crash in your project after the merge): 1) The project title (and other basic metadata) should be set in 'reproduce/analysis/conf/metadata.conf'. Please include this file in your merge (if it is ignored because of '.gitattributes'!). 2) Consider importing the changes in 'initialize.mk' and 'verify.mk' (if you have added all analysis Makefiles to the '.gitattributes' file (thus not merging any change in them with your branch). For example with this command: git diff master...maneage -- reproduce/analysis/make/initialize.mk 3) The old 'verify-txt-no-comments-leading-space' function has been replaced by 'verify-txt-no-comments-no-space'. The new function will also remove all white-space characters between the columns (not just white space characters at the start of the line). Thus the resulting check won't involve spacing between columns. A common set of steps are always necessary to prepare a project for publication. Until now, we would simply look at previous submissions and try to follow them, but that was prone to errors and could cause confusion. The internal infrastructure also didn't have some useful features to make good publication possible. Now that the submission of a paper fully devoted to the founding criteria of Maneage is complete (arXiv:2006.03018), it was time to formalize the necessary steps for easier submission of a project using Maneage and implement some low-level features that can make things easier. With this commit a first draft of the publication checklist has been added to 'README-hacking.md', it was tested in the submission of arXiv:2006.03018 and zenodo.3872248. To help guide users on implementing the good practices for output datasets, the outputs of the default project shown in the paper now use the new features). After reading the checklist, please inspect these. Some other relevant changes in this commit: - The publication involves a copy of the necessary software tarballs. Hence a new target ('dist-software') was also added to package all the project's software tarballs in one tarball for easy distribution. - A new 'dist-lzip' target has been defined for those who want to distribute an Lzip-compressed tarball. - The '\includetikz' LaTeX macro now has a second argument to allow configuring the '\includegraphics' call when the plot should not be built, but just imported.
2020-06-04README-hacking.md: minor edits in description of merging with ManeageMohammad Akhlaghi-7/+15
The recently added description for this step in the last commit needed some edits to be more clear and encourage re-building the project from scratch anytime authors merge with Maneage.
2020-06-03README-hacking.md: Improved section on ignoring some files in ManeageMohammad Akhlaghi-22/+53
When some files should not be merged, until now we were suggesting to also add deleted files to the '.gitattributes' file. However, this feature of Git doesn't work for deleted files and they would still show up in the 'master' branch after a merge. So with this commit, we have added a simple AWK command to run after a merge that will automatically detect and delete such files (using the output of 'git status --porcelain'). Also, two minor typos were corrected in the newly added 'servers-backup.conf' file: the copyright year was wrong and there was no new-line at the end of the file (a good convention!).
2020-05-25Unified reference to GNU/Linux and free softwareMohammad Akhlaghi-2/+2
One of the main reasons to building Maneage is to properly acknowledge/attribute the authors of software in research. So we have adopted a standard of never referring to the GNU-based operating systems running the Linux kernel simply as "Linux", we avoid terms like "Open Sourse" and use Free Software instead (in the same spirit). With this commit, a few instances of the cases above have been corrected, they had slipped through our fingers when we initially imported them into the project. In the special case of the "Journal for Open Source Software", we simply replaced it with its abbreviation (JOSS). This was done because in effect we were generally using journal name abbreviations in almost all the citations already. To avoid any inconsistancies, the names of the three other journals that weren't abbreviated are also abbreviated.
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-10/+10
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-04-28Astropy will no longer be installed by defaultMohammad Akhlaghi-23/+30
Until now Gnuastro and Astropy where installed by default in any clean build of Maneage. Gnuastro is used to do the demonstration analysis that is reported in the paper and Astropy was just there to help in testing the building of the MANY tools it depends on! It (and its dependencies) also had several papers that helped show software citation. However, as Boud suggested in task #15619, the burden of installing them for a new user may be too much and any future changes will cause merge conflicts. It may also give the impression that Maneage is only/mainly written for astronomers. So with this commit, I am removing Astropy as a default target. But we can only remove Gnuastro after we include an alternative analysis in the demonstration `delete-me' files. Following Boud's suggestion in that task, `TARGETS.conf' was also added to the files to be ignored in any future merge (in the checklist of `README-hacking.mk'). The solution was already described there, but mainly focused on the deleted `delete-me' files. So with this commit, I brought out this item as a more prominent item in the list. Maybe we can later add the analysis done in the Maneage paper (not yet published). In terms of testing the software builds, we already have task #15272 (Single target to build all high-level software, for testing) that aims to have a single configure option to install ALL high-level software and we can ask people to try if they like and report errors.
2020-04-26README-hacking.md: described why automatic preparation only occurs onceZahra Sharbaf-1/+20
Recently (since Commit 7d0c5ef77), the preparation is not run automatically every time. It is only run automatically the first time and needs to be manually called with the `--prepare-redo' option. But this wasn't explained in `README-hacking.md' (currently the main documentation of Maneage). With this commit, a description about invoking the preparation process after the first attempt of the running project has been added to `README-hacking.md'.
2020-04-25IMPORTANT: Primary Maneage repositories are now under maneage.orgMohammad Akhlaghi-11/+10
Until now, the primary Maneage URLs were under GitLab, but since we now have a dedicated URL and Git repository, its better to transfer to this as soon as possible. Therefore with this commit, throughout Maneage, any place that Maneage was referenced through GitLab has been corrected. Please correct your project's remote to point to the new repository at `git.maneage.org/project.git', and please make sure it follows the `maneage' branch. There is no more `master' branch on Maneage.
2020-04-21README-hacking.md: removed any mention of tagsMohammad Akhlaghi-94/+68
Tags are not a fixed piece of history (they can easily be moved and not imported in a different repository), so they are only confusing in the context of Maneage (where people should branch-off the main project). the raw commit hashes are a much more robust way to store a precise moment in history. Before this commit, I removed all Tags from the main Git repositories of Maneage and thus removed any mention of Tags with `README-hacking.md'. Ofcourse, if a project decides to use tags is upto them, but we won't implement it in the main branch.
2020-04-21README-hacking.md: minor clarifications in checklistMohammad Akhlaghi-28/+32
Roberto Baena recently tried building a new project with Maneage and provided the following suggestions to make it more clear for a new user: 1) In the part where we talk about creating a Git repository, we should highlight that it must be empty. This is because some (for example Gitlab) propose to include a `README' file. But if the project is not empty, Git will not allow pushing to it. 2) The `(can be done later)' comment was removed from the "Delete dummy parts") to avoid confusion about applying some of them, but not others: if only some are done, it may cause problems in the build.
2020-04-20README-hacking.md: Removed TeXLive year problem and numberd checklistMohammad Akhlaghi-21/+11
We recently fixed the problem of TeXLive that hard-codes the year of its build in its installation directory. But the note on this problem was still kept in `README-hacking.md'. That part is now removed. Also, to help in following the checklist, it is now an ordered list.
2020-04-20Maneage instead of Template in README-hacking.md and copyright noticesMohammad Akhlaghi-217/+211
Until now, throughout Maneage we were using the old name of "Reproducible Paper Template". But we have finally decided to use Maneage, so to avoid confusion, the name has been corrected in `README-hacking.md' and also in the copyright notices. Note also that in `README-hacking.md', the main Maneage branch is now called `maneage', and the main Git remote has been changed to `https://gitlab.com/maneage/project' (this is a new GitLab Group that I have setup for all Maneage-related projects). In this repository there is only one `maneage' branch to avoid complications with the `master' branch of the projects using Maneage later.
2020-04-17IMPORTANT: software config directly under reproduce/software/configMohammad Akhlaghi-30/+29
Until now the software configuration parameters were defined under the `reproduce/software/config/installation/' directory. This was because the configuration parameters of analysis software (for example Gnuastro's configurations) were placed under there too. But this was terribly confusing, because the run-time options of programs falls under the "analysis" phase of the project. With this commit, the Gnuastro configuration files have been moved under the new `reproduce/analysis/config/gnuastro' directory and the software configuration files are directly under `reproduce/software/config'. A clean build was done with this change and it didn't crash, but it may cause crashes in derived projects, so after merging with Maneage, please re-configure your project to see if anything has been missed. Please let us know if there is a problem.
2020-04-01Corrected reference for Infante-Sainz+2020 in README-hacking.mdRaul Infante-Sainz-1/+1
Until this commit, the year of this paper was 2019 and the linking url was the temporal one. However, the final official publication year is 2020. With this commit, the year and the url have been changed to the final ones.
2020-02-18README-hacking.md: corrected typo in project commandMohammad Akhlaghi-1/+1
I had forgot to add a `./' before the call to `project' for the `--check-config'.
2020-02-05Imported Raul's additions to README-hacking.md, no conflictsMohammad Akhlaghi-2/+16
There were no conflicts in this merge.
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-2/+2
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-27Moving basic configuration of Git section in README-hacking.mdRaul Infante-Sainz-13/+13
Until this commit, the small section of `README-hacking.md' in which it is explained how to do the first configuration of Git was at the beginning of the section `First custom commit'. However, it is better to have it just before the item `Your first commit' in that section. With this commit, this change has been done. Now the reader has the necessary steps for configuring Git just before it is needed for making the first commit.
2020-01-23IMPORTANT: Project preparation is now also done with project makeMohammad Akhlaghi-21/+12
Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system.
2020-01-22Adding Raul as contributor of README-hacking.mdRaul Infante-Sainz-1/+2
Since I (Raul) did some changes (and I hope to do more :-)) in the `README-hacking.md', I am adding my information at the beginning of this file.
2020-01-22Adding basic configuration of Git in README-hacking.mdRaul Infante-Sainz-1/+14
Until this commit, we were asuming that Git was already properly configured. However, in order to be as complete as possible, it would be good if the basic commands to configure Git were in the `README-hacking.md'. With this commit, a small paragraph has been added in order to have the basic Git configuration commands (i.e. to configure the name, email, and favorite text editor).
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-33/+33
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19New --check-config option to ./project to check software build statusMohammad Akhlaghi-31/+14
Until now, it was necessry to run a long `while true' loop to see what is currently being built at configure time. So with this commit, a new `--checkconfig' option has been added to `./project' that can be called to run that loop and make it easier to check.
2020-01-18README-hacking.md: edits and corrections for easier customizationMohammad Akhlaghi-8/+10
The checklist descriptions were slightly edited to be more clear. Also, while following them, I noticed that while removing the "delete-me" parts on `verify.mk', would cause an error: the `if [ $$m == delete-me ];' statement we were saying to delete cause an error because `elif' was the first statement Bash would see. So with this commit, the `download' conditional (which isn't instructed to be deleted) was set to be the top (with an `if') and the `delete-me' conditional now has an `elif'.
2020-01-17README-hacking.md: script to list installed programs before configureMohammad Akhlaghi-10/+18
Until now, the small one-line script that lists programs was introduced in the checklist after running `./project configure'. But people would mostly miss it because they would wait until the configuration is complete. With this commit, that point has been put above the `./project configure' step. Readers are instructed to open a new terminal and run that script, then go to the next step so they see the directories get filled actively. It will also help them understand what is going on.
2020-01-13Minor corrections in referencing Infante-Sainz+2019Mohammad Akhlaghi-6/+5
Until now the actual journal webpage was used for Raul's paper. However, the journal webpage needs authorized access for people to read it, therefore its will be inaccessible for many people. A better and more well known place for the paper (atleast in astronomy) is the ADS link. In the ADS link, if someone has access to the journal, they will get the journal's version and if not, they will get the arXiv version. It also has a common BibTeX export tool for all journals. We had also done this for the other papers in that list. With this commit, I thus changed the URL for the paper, and also removed the "issue" number (4 in this case), since that is mostly irrelevant, only the volume and page numbers are usually used for the other papers too.
2020-01-13Added Infante-Sainz et al. 2019 as most recent paper using this templateRaul Infante-Sainz-3/+4
The "SDSS extended PSFs" paper was already included as an example of papers wich uses this template. However, the reference was the arXiv one. With this commit, since the paper has been finally published, it has been added the final reference to the journal.
2020-01-01Verification function checks if file existsMohammad Akhlaghi-5/+7
Until now, if the file to be verified didn't exist, a different checksum would be generated, and it would stop, but it wasn't immediately clear if the differing checksum is because the file doesn't exist at all! With this commit, before calculating the checksum, we first make sure if the file exists. If it doesn't exist an explicit error is printed and thus will help the project editor to find the cause of the problem.
2020-01-01Minor corrections in `README-hacking.md' after verificationMohammad Akhlaghi-4/+7
In the previous commit, I had forgot to update a small part in the checklist (when modifying `top-make.mk') which is now corrected. I also added a few sentences in the description of how to customize the verification to make it easier to understand.
2020-01-01Verification of output values and data added within templateMohammad Akhlaghi-9/+51
Until now, the only verification that the template provided was the published PDF. Users had to manually compare the published and generated PDFs (numbers, plots, tables) and see if they obtained the same result. However, this type of manual verification is not good and is prone to frustration and missing important differences. With this commit, a new Makefile has been added in the analysis steps: `verify.mk'. It provides facilities to easily verify the results that go into the paper. For example tables that go into making the paper's plots, or the LaTeX macros that blend into the text. See the updated parts in `README-hacking.md` for a more complete explanation. This completes task #15497.
2020-01-01README-hacking.md checklist now also ignores changes in paper.texMohammad Akhlaghi-7/+12
In the previous commit, we added the files to ignore from the template branch, but only the files that had been deleted. With this commit, `paper.tex' is also added to the files that must be ignored from the template branch (the file remains in the project, but in the template branch, its contents are just dummy place-holders).
2020-01-01Added step README-hacking's checklist to avoid merging dummy filesMohammad Akhlaghi-0/+15
During the checklist we guide the user to delete the dummy `delete-me*' files from their custom branch. Later, if the dummy files are updated in the template's master branch, if the user merges with the template branch, these files will be written back into their project! This is very annoying! With this commit, a step was added in the `README-hacking.md' checklist, just after deleting the dummy files to avoid this problem using the `.gitattributes' file, telling Git to keep the changes as implemented in the merging branch (`ours').
2020-01-01Copyright statements updated to include 2020Mohammad Akhlaghi-3/+3
Now that its 2020, its necessary to include this year in the copyright statements.
2019-11-06Added 1911.01430 as most recent paper using this templateMohammad Akhlaghi-0/+7
Raul's paper (that uses this template) was just published on arXiv today (congratulations Raul!). So it has been added to the list of papers using this template.