aboutsummaryrefslogtreecommitdiff
path: root/reproduce/analysis
AgeCommit message (Collapse)AuthorLines
2021-01-02Supplement (containing appendices) optionally built separatelyMohammad Akhlaghi-47/+55
Until now, the build strategy of the paper was to have a single output PDF that either contains (1) the full paper with appendices in the same paper (2) only the main body of the paper with no appencies. But the editor in chief of CiSE recently recommended publishing the appendices as supplements that is a separate PDF (on its webpage). So with this commit, the project can make either (1) a single PDF (containing both the main body and the appendices) that will be published on arXiv and will be the default output (this is the same as before). (2) two PDFs: one that is only the main body of the paper and another that is only the appendices. Since the appendices will be printed as a PDF in any case now, the old '--no-appendix' option has been replaced by '--supplement'. Also, the internal shell/TeX variable 'noappendix' has been renamed to 'separatesupplement'.
2020-12-01Imported recent work in Maneage, minor conflicts fixedMohammad Akhlaghi-158/+32
Some minor conflicts that came up during the merge were fixed.
2020-12-01Default paper: macros available for date of commits citedMohammad Akhlaghi-1/+5
Until now, Maneage only provided the commit hashes (of the project and Maneage) as LaTeX macros to use in your paper. However, they are too cryptic and not really human friendly (unless you have access to the Git history on a computer). With this commit, to make things easier for the readers, the date of both commits are also available as LaTeX macros for use in the paper. The date of the Maneage commit is also included in the acknowledgements. Also, the paragraph above the acknowledgements has been updated with better explanation on why adding this acknowledgement in the science papers is good/necessary.
2020-12-01IMPORTANT: organizational improvements in Maneage TeX sourcesMohammad Akhlaghi-2/+15
This only concerns the TeX sources in the default branch. In case you don't use them, there should only be a clean conflict in 'paper.tex' (that is obvious and easy to fix). Conflicts may only happen in some of the 'tex/src/preamble-*.tex' files if you have actually changed them for your project. But generally any conflict that does arise by this commit with your project branch should be very clear and easy to fix and test. In short, from now on things will even be easier: any LaTeX configuration that you want to do for your project can be done in 'tex/src/preamble-project.tex', so you don't have to worry about any other LaTeX preamble file. They are either templates (like the ones for PGFPlots and BibLaTeX) or low-level things directly related to Maneage. Until now, this distinction wasn't too clear. Here is a summary of the improvements: - Two new options to './project make': with '--highlight-new' and '--highlight-notes' it is now possible to activate highlighting on the command-line. Until now, there was a LaTeX macro for this at the start of 'paper.tex' (\highlightchanges). But changing that line would change the Git commit hash, making it hard for the readers to trust that this is the same PDF. With these two new run-time options, the printed commit hash will not changed. - paper.tex: the sentences are formatted as one sentence per line (and one line per sentence). This helps in version controlling narrative and following the changes per sentence. A description of this format (and its advantages) is also included in the default text. - The internal Maneage preambles have been modified: - 'tex/src/preamble-header.tex' and 'tex/src/preamble-style.tex' have been merged into one preamble file called 'tex/src/preamble-maneage-default-style.tex'. This helps a lot in simply removing it when you use a journal style file for example. - Things like the options to highlight parts of the text are now put in a special 'tex/src/preamble-maneage.tex'. This helps highlight that these are Maneage-specific features that are independent of the style used in the paper. - There is a new 'tex/src/preamble-project.tex' that is the place you can add your project-specific customizations.
2020-11-27Fix paper.bbl flaw; reduce long author lists: save 70 wordsBoud Roukema-10/+21
This commit fixes the error of trying to run bibtex on appendix.tex when the --no-appendix option is selected. A hardwired hack, appropriate only for this specific paper, replaces the more-than-three-author parts of two long author lists by "et al." To test this without having to redownload the menke file, first do "rm -fv .build/tex/build/*.aux .build/tex/build/*.bbl" and then "./project make --no-appendix" a few times. This commit should reduce the word length by about 70 words.
2020-11-26All the referee points have been answeredMohammad Akhlaghi-1/+1
There is an answer for all the referee points now. I also did some minor edits in the paper. But we are still over the limit by around 250 words. The only remaining point that is not yet addressed (and has '####' around it) is the discussion on parallelization and its effect on reproducibility.
2020-11-25Reviewer points 1-15; appendix clickable linksBoud Roukema-0/+7
This commit updates "paper.tex" and "peer-review/1-answer.txt" for the first 15 (out of 59!) reviewer points, excluding points 2 (not yet done) and 9 (README-hacking.md needs tidying). A fix to "reproduce/analysis/make/paper.mk" for the links in the appendices is also done in this commit (the same algorithm as for paper.tex is added). The links in the appendices are not (yet) clickable.
2020-11-23Minor edits and correctionsMohammad Akhlaghi-1/+1
Raul's added point on the answer to the referee was very good, so I edited it a little to be more clear (and removed his name). Also, after looking in a few parts of the text, I fixed a few typos.
2020-11-23First draft of all the points addressed by the refereesMohammad Akhlaghi-2/+8
A new directory has been added at the top of the project's source called 'peer-review'. The raw reviews of the paper by the editors and referees has been added there as '1-review.txt'. All the main points raised by the referees have been listed in a numbered list and addressed (mostly) in '1-answers.txt'. The text of the paper now also includes all the implemented answers to the various points.
2020-11-20Highlighting changes can now be toggled at run-timeMohammad Akhlaghi-1/+11
Until now, the core Maneage 'paper.tex' had a '\highlightchanges' macro that defines two LaTeX macros: '\new' and '\tonote'. When '\highlightchanges' was defined, anything that was written within '\new' became dark green (highlighting new things that have been added). Also, anything that was written in '\tonote' was put within a '[]' and became dark red (to show that there is a note here that should be addressed later). When '\highlightchanges' wasn't defined, anything within the '\new' element would be black (like the rest of the text), and the things in '\tonote' would not be shown at all. Commenting the '\newcommand{\highlightchanges}{}' line within 'paper.tex' (to toggle the modes above) would create a different Git hash and has to be committed. But this different commit hash could create a false sense in the reader that other things have also been changed and the only way they could confirm was to actually go and look into the project history (which they will not usually have time to do, and thus won't be able to trust the two modes of the text). Also, the added highlights and the note highlights were bundeled together into one macro, so you couldn't only have one of them. With this commit, the choice of highlighting either one of the two is now done as two new run-time options to the './project' script (which are passed to the Makefiles, and written into the 'project.tex' file which is loaded into 'paper.tex'). In this way, we can generate two PDFs with the same Git commit (project's state): one with the selected highlights and another one without it. This issue actually came up for me while implementing the changes here: we need to submit one PDF to the journal/referees with highlights on the added features. But we also need to submit another PDF to arXiv and Zenodo without any highlights. If the PDFs have different commit hashes, the referees may associate it with other changes in any part of the work. For example https://oadoi.org/10.22541/au.159724632.29528907 that mentions "Another version of the manuscript was published on arXiv: 2006.03018", while the only difference was a few words in the abstract after the journal complained on the abstract word-count of our first submission (where the commit hashes matched with arXiv/Zenodo).
2020-11-15First edits on the newly added appendices in new formMohammad Akhlaghi-2/+3
With the optional appendices added recently to the paper, it was important to go through them and make them more fitting into the paper.
2020-11-15Building final PDF: pdf-build-final has to be given an explicit yesMohammad Akhlaghi-15/+15
Until now, when the 'pdf-build-final' configuration variable (defined in 'reproduce/analysis/config/pdf-build.conf') was given any string a PDF would be built. This was very confusing, because people could put a 'no' and the PDF would still be built! With this commit, only when this variable has a value of 'yes' will the PDF be built. If given any other string (or no string at all), it will not produce a PDF. This issue was reported by Zahra Sharbaf.
2020-11-04Appendix of long paper added, optionally we can disable itMohammad Akhlaghi-2/+15
Given the referee reports, after discussing with the editors of CiSE, we decided that it is important to include the complete appendix we had before that included a thorough review of existing tools and methods. However, the appendix will not be published in the paper (due to the strict word-count limit). It will only be used in the arXiv/Zenodo versions of the paper. This actually created a technical problem: we want the commit hash of the project source to remain the same when the paper is built with an appendix or without it. To fix this problem the choice of including an appendix has gone into the 'project' script as a run-time option called '--no-appendix'. So by default (when someone just runs './project make'), the PDF will have an appendix, but when we want to submit to the journal, or when the appendix isn't needed for a certain reason, we can use this new option. The appendix also has its own separate bibliography. Some other corrections made in this commit: 1. Some new references were added that had an '_' in their source, they were corrected in 'references.tex'. 2. I noticed that 'preamble-style.tex' is not actually used in this paper, so it has been deleted.
2020-10-18Recipes for final initialize and verify targets not on stdoutMohammad Akhlaghi-4/+11
The LaTeX macro files for these two subMakefiles are created on every run of './project make'. So their commands are also printed every time and hardly ever will a normal user want to modify or change these. So to avoid populating the standard output of a Maneaged project with all these extra lines every time (possibly getting mixed with the important analysis or LaTeX outputs), an '@' has been placed at the start of the recipes. With an '@' at the start of the recipe, Make is instructed to not print the commands it wants to run in the standard output.
2020-09-24Gnuastro's analysis configuration files removedMohammad Akhlaghi-133/+2
Until now, the core Maneage branch included some configuration files for Gnuastro's programs. This was actually a remnant of the distant past when Maneage didn't actually build its own software and we had to rely on the host's software versions. This file contained the configuration files specific to Gnuastro for this project and also had a feature to avoid checking the host's own configuration files. However, we now build all our software ourselves with fixed configuration files (for the version that is being installed and its version is stored). So those extra configuration files were just extra and caused confusion and problems in some scenarios. With this commit, those extra files are now removed. Also, two small issues are also addressed in parallel with this commit: - When running './project make clean', the 'hardware-parameters.tex' macro file (which is created by './project configure' is not deleted. - The project title is now written into the default output's PDF's properties (through 'hypersetup' in 'tex/src/preamble-header.tex') through the LaTeX macro. All these issues were found and fixed with the help of Samane Raji.
2020-09-03Imported recent work in Maneage, minor conflicts fixedMohammad Akhlaghi-5/+6
Only two small conflicts came up: * The addition of the hardware architecture macro in 'paper.tex' (which was removed for now, but will be added as the referee has requested within the text). * The usage of "" around directory variables in 'paper.mk'.
2020-08-27Machine architecture and byte-order available as LaTeX macroMohammadreza Khellat-4/+5
Until now, no machine-related specifications were being documented in the workflow. This information can become helpful when observing differences in the outcome of both software and analysis segments of the workflow by others (some software may behave differently based on host machine). With this commit, the host machine's 'hardware class' and 'byte-order' are collected and now available as LaTeX macros for the authors to use in the paper. Currently it is placed in the acknowledgments, right after mentioning the Maneage commit. Furthermore, the project and configuration scripts are now capable of dealing with input directory names that have SPACE (and other special characters) by putting them inside double-quotes. However, having spaces and metacharacters in the address of the build directory could cause build/install failure for some software source files which are beyond the control of Maneage. So we now check the user's given build directory string, and if the string has any '@', '#', '$', '%', '^', '&', '*', '(', ')', '+', ';', and ' ' (SPACE), it will ask the user to provide a different directory.
2020-08-20Imported recent updates in Maneage, minor conflicts fixedMohammad Akhlaghi-34/+82
Some very minor conflicts came up and were easily corrected. They were mostly in parts that are also shared with the demonstration in the core Maneage branch.
2020-08-17Minor typo correction in comments of paper.mkMohammad Akhlaghi-1/+1
The '.bbl' suffix in the comment of one call to LaTeX was incorrectly written as '.bb'.
2020-08-02initialize.mk: accounting for no maneage branchBoud Roukema-3/+8
One of the LaTeX macros reported by 'initialize.mk' is the git commit hash of the most recent 'maneage' branch that the project has been branched from. However, not all projects will retain the maneage reference. This can happen for example when people don't push the 'maneage' reference to their repository and then clone from their own repository to a second computer. Therefore, until now, in such situations, Maneage would break with an error. With this commit, in such scenarios, a place holder string is used instead, clearly highlighting that there is no 'maneage' reference.
2020-07-21Printing location when downloaded input data checksum is differentBoud Roukema-0/+1
There are many different directory trees involved in Maneage system: the top directory, the 'reproduce/' directory and its sub-directories, '.build/' (that point to a user-defined build area), and a possibly user-defined input directory. Until now, in the case of a download checksum failure, it was not immediately obvious [1] to the user *where* the file with a failed checksum is. To clarify to the user *where* the suspicious file is now located, this commit adds a line to 'reproduce/analysis/make/download.mk' to print out this full path location: '$$unchecked' along with the expected and calculated checksums. [1] Euphemism for me spending lots of time debugging and being confused.
2020-07-20README-hacking.md: clarify Zenodo usage in publication checklistBoud Roukema-2/+2
This commit clarifies the initial usage of Zenodo for reserving a Zenodo identifier and starting an 'unpublished' upload. Some other minor wording changes are done here.
2020-07-20make dist: only archive files that are under version controlBoud Roukema-17/+31
Until this commit, the '$(project-package-contents)' rules in 'reproduce/analysis/make/initialize.mk' included a line to provide all contents, recursively, of the directory 'reproduce/' in the package for further distribution. This could potentially lead to the distribution of private working files that are used during development and not intended for general distribution. With this commit, only those files in 'reproduce/' and 'tex/src' that are under version control are copied to the temporary directory (that is later used for creating an archive). With this change, the archiving commands actually became more clean (we don't have to manually remove 'LOCAL.conf' or other temporary files). Extensive comments have also been added above each step to clarify each step's purpose and method.
2020-07-07Project distribution tarball can account for no PDFs in tex/tikzMohammad Akhlaghi-1/+6
Until now the './project make dist' command implicitly assumed that the 'tex/tikz' directory always contains PDF files (because of the 'cp tex/tikz/*.pdf $$dir/tex/tikz' line). This was annoying for projects that don't use TiKZ or PGFPlots to generate their plots, and they had to manually comment this line. With this commit a check has been placed to see if any PDF files exist in there at all. If there aren't PDF files, the 'cp' command above is ignored.
2020-07-05Removing possibly existing paper.bbl before remaking itMohammad Akhlaghi-0/+6
Until now, when the bibliography file ('paper.bbl') had a LaTeX-related error (for example the journal name was a LaTeX macro that isn't defined), the first 'pdflatex' command that is run before 'biber' would crash, not allowing the project to reach 'biber'. So the user would have to manually remove 'paper.bbl' before running './project make'. With this commit, we remove any possibly existing 'paper.bbl' file before rebuilding it. Generally, this also helps in keeping things clean during the generation of the new bibliography. This bug was found by Mahdieh Nabavi.
2020-07-04Commit hash of Maneage branch used to build project as LaTeX macroMohammad Akhlaghi-0/+6
To help in the documentation, the Git hash of the Maneage branch commit that the project has most recently merged with (or branched from) is now also provided as a LaTeX macro ('\maneageversion'). It is calculated in 'reproduce/analysis/make/initialize.mk' (in the recipe to 'initialize.tex').
2020-07-04Better names and comments in INPUTS.confMohammad Akhlaghi-28/+32
Until now, the dataset's configuration names had a 'WFPC2' prefix. But this very alien to anyone that is not familiar with the history of the Hubble Space Telescope (the camera is no longer used! Its just used here since its one of the standard FITS files from the FITS standard webpage). With this commit the variable names have been modified to be more readable and clear (having a 'DEMO-' prefix). Also the comments of 'INPUTS.conf' (describing the purpose of each variable) were edited and made more clear.
2020-07-04Improved comments in paper.mk and README.mdMohammad Akhlaghi-1/+5
In 'README.md' I tried to explain a little better that TeXLive will only install its necessary packages, not the full TeXLive library! Also in paper.mk, I slightly improved the comments with very minor edits. Both these parts are slated to go into the core Maneage branch, so its important to maintain them here for now.
2020-06-30The distclean target accounts for non-existance of git hooksMohammad Akhlaghi-1/+1
Until now, when the user wanted to complete remove all built files (including software), the './project make distclean' command would fail if the git hooks weren't installed. They are present when the project's configuration has been successfully finished, but this bug can happen when trying to re-do an incomplete build. With this commit, this is fixed by adding an '-f' has been added before the 'rm' command for the Git hooks. This commit was also done in the core Maneage branch.
2020-06-30The distclean target accounts for non-existance of git hooksMohammad Akhlaghi-1/+1
Until now, when the user wanted to complete remove all built files (including software), the './project make distclean' command would fail if the git hooks weren't installed. They are present when the project's configuration has been successfully finished, but this bug can happen when trying to re-do an incomplete build. With this commit, this is fixed by adding an '-f' has been added before the 'rm' command for the Git hooks.
2020-06-28Zenodo identifier is extracted automatically from metadata.confMohammad Akhlaghi-1/+5
Until now, the Zenodo identifier was manually written in the paper. But now we have the Zenodo DOI in 'metadata.conf', so its much more robust to get it from there (in case updated versions of the paper is published).
2020-06-27Imported recent work in master, minor conflict fixed in paper.mkMohammad Akhlaghi-6/+14
Only two conflicts came up in the newly added comments of 'paper.mk' in the Maneage branch. It happened because in this project we don't use 'pdflatex', but 'latex' alone.
2020-06-19Removing preparation-done.mk when cleaning by ./project make cleanRaul Infante-Sainz-0/+1
Until this commit, the file `BDIR/software/preparation-done.mk' were not removed when cleaning the project with `./project make clean'. This file is generated in the preparation of the data during the analysis step. However, the cleaning is expected to remove anything generated in the analysis process! Step by step, with the commands: ./project make ---> Will make the preparation and analysis ./project make clean ---> Will remove all analysis outputs (but not `preparation-done.mk') ./project make ---> Won't do the preparation, only analysis! However, in the last step it should do the preparation again, because the input data could have change for any reason. With this commit, the file `BDIR/software/preparation-done.mk' is removed when cleaning the project, and consequently, in the analysis step the input data is prepared.
2020-06-17Security risk of LaTeX's -shell-escape option explained in commentBoud Roukema-0/+9
The 'pdflatex' program is used to build the default Maneage-branch paper. But since the default paper uses PGFPlots to build the figures within LaTeX as an external PDF, PGFPlots requires 'pdflatex' to be called with the '-shell-escape' option. Generally, this option can be considered as a security risk (in particular when 'pdflatex' is being run by an external LaTeX file: a malicious LaTeX writer may embed commands in the LaTeX source that will be executed on the host if this option is present). This is not too serious of an issue in Maneage, because when someone runs Maneage, they intentionally let it run many on their system. Hence if someone wants to exploit a host system, they can add the necessary commands long before 'pdflatex' is run. After all, all commands in Maneage are run with the calling user's permissions, hence they have access to many parts of the user's accounts. If someone is worried about security on a non-trusted Maneage project they should act the same as they do with any software: define a new user for it, and call it with that user (as a weak-level security), or run it in a virtual machine or container. However, since this option has been explicity mentioned as a security risk before, it helps if we have a comment explaining its usage in 'paper.mk'. With this commit, the concerned user will read a brief explanation and can read the brief discussion at [1] and possibly re-open the discussion or propose ways of mitigating the security risk(s). [1] https://savannah.nongnu.org/task/?15694
2020-06-17New target --dist-software to package all necessary software tarballsMohammad Akhlaghi-6/+2
When publishing a project, it is necessary to also publish the source code of all necessary software of the project. We had recently added a new './project make' target called 'dist-software' for this job, but had forgotten to add it in the output of './project --help'! There was also a small bug inside of it that didn't allow the successful copying of the created tarball to the top project directory. With this commit, an explanation for this target has been added in the output of './project --help' and that bug has been fixed.
2020-06-10Corrected bug in using local copy of input datasetMohammad Akhlaghi-13/+47
As described in Maneage's commit 2bd2e2f18 (which I found while testing this project), the existing download recipe had problems when using a local copy of the input dataset. It was first fixed here, then implemented there. Also, to clarify things for a new user, some long comments were added at the top of 'INPUTS.conf' to describe each of the variables, that comment has also been put here (and is also in commit 2bd2e2f18 of Maneage).
2020-06-10IMPORTANT: bug fix in default data download script of download.mkMohammad Akhlaghi-14/+54
Summary of possible semantic conflicts 1. The recipe to download input datasets has been modified. You have to re-set the old 'origname' variable to 'localname' (to avoid confusion) and the default dataset URL should now be complete (including the actual filename). See the newly added descriptions in 'INPUTS.conf' for more on this. Until now, when the dataset was already present on the host system, a link couldn't be made to it, causing the project to crash in the checksum phase. This has been fixed with properly naming the main variable as 'localname' to avoid the confusion that caused it. Some other problems have been fixed in this recipe in the meantime: - When the checksum is different, the expected and calculated checksums are printed. - In the default paper, we now print the full URL of the dataset, not just the server, so the checksum of the 'download.tex' step has been updated.
2020-06-09Minor edit printing arXiv URL in plain text metadataMohammad Akhlaghi-1/+1
Until now, in the 'print-copyright' function of 'initialize.mk' (that prints a fixed set of common meta necessary in plain-text files), we were simply printing this line: # Pre-print server: arXiv:1234.56789 But given that all the other elements are click-able URLs, it now prints: # Pre-print server: https://arxiv.org/abs/1234.56789
2020-06-09Two minor corrections to avoid warnings in make and make cleanMohammad Akhlaghi-10/+2
There were two small warnings that are removed with this commit: - In the end, when we print the number of words in the PDF, we hadn't accounted for the fact that 'paper.pdf' doesn't always exist (for example when './project make clean' is run). So a check was added to only print the number of words when a PDF exists. - I noticed that the '$(texdir)/to-publish' directory was being built both in 'initialize.mk' and in 'demo-plot.mk'. So the one in 'demo-plot.mk' has been removed.
2020-06-09Imported Maneage, minor conflicts fixed, a bug found and fixedMohammad Akhlaghi-54/+145
Some minor conflicts came up in 'initialize.mk' and 'verify.mk'. For the former, I chose the version on Maneage, for the latter, I kept the 'master' version on the checksums of this project, but kept the Maneage version for the rest of the improvements there (like printing the verified files as LaTeX comments in 'verify.tex'. While testing the conflicts, I noticed a bug (in the LaTeX macro for the number of years in the Menke+20 paper) in the previous build, thanks to the verification step :-)! Fortunately it wasn't actually printed in the PDF, so a normal reader won't recognize. The bug was caused by the recently added meta-data/commented lines in the 'tools-per-year.txt' file: when calculating the number of years studied in that paper, we were simply counting all the lines and we had forgot to correct this after adding comments. As a result, the un-used LaTeX macro file was saying that they have studied 47 years instead of the real 31 years! This element was actually used in the very first (+40 page!) draft of the paper that was summarized to fit into the journal limits.
2020-06-06IMPORTANT: Added publication checklist, improved relevant infrastructureMohammad Akhlaghi-67/+242
Possible semantic conflicts (that may not show up as Git conflicts but may cause a crash in your project after the merge): 1) The project title (and other basic metadata) should be set in 'reproduce/analysis/conf/metadata.conf'. Please include this file in your merge (if it is ignored because of '.gitattributes'!). 2) Consider importing the changes in 'initialize.mk' and 'verify.mk' (if you have added all analysis Makefiles to the '.gitattributes' file (thus not merging any change in them with your branch). For example with this command: git diff master...maneage -- reproduce/analysis/make/initialize.mk 3) The old 'verify-txt-no-comments-leading-space' function has been replaced by 'verify-txt-no-comments-no-space'. The new function will also remove all white-space characters between the columns (not just white space characters at the start of the line). Thus the resulting check won't involve spacing between columns. A common set of steps are always necessary to prepare a project for publication. Until now, we would simply look at previous submissions and try to follow them, but that was prone to errors and could cause confusion. The internal infrastructure also didn't have some useful features to make good publication possible. Now that the submission of a paper fully devoted to the founding criteria of Maneage is complete (arXiv:2006.03018), it was time to formalize the necessary steps for easier submission of a project using Maneage and implement some low-level features that can make things easier. With this commit a first draft of the publication checklist has been added to 'README-hacking.md', it was tested in the submission of arXiv:2006.03018 and zenodo.3872248. To help guide users on implementing the good practices for output datasets, the outputs of the default project shown in the paper now use the new features). After reading the checklist, please inspect these. Some other relevant changes in this commit: - The publication involves a copy of the necessary software tarballs. Hence a new target ('dist-software') was also added to package all the project's software tarballs in one tarball for easy distribution. - A new 'dist-lzip' target has been defined for those who want to distribute an Lzip-compressed tarball. - The '\includetikz' LaTeX macro now has a second argument to allow configuring the '\includegraphics' call when the plot should not be built, but just imported.
2020-06-04Minor improvements in the make dist command for this paperMohammad Akhlaghi-7/+12
This paper doesn't use pdflatex or biblatex, so it was necessary to make some small corrections in the make-dist rule of initialize.mk. Also, while testing the upload on arXiv, I noticed that it complains about an empty 'verify.tex' file, so that is also corrected.
2020-06-04Verification activated, README added, Proper metadata in plot dataMohammad Akhlaghi-18/+89
All the steps following the to-be-added (in 'README-hacking.md') publication checklist prior to the final check from new clone have been added: - 'README.md' file has been set. - "Reproducible supplement" was added just above the keywords, pointing to Zenodo. - A link to the to-be-uploaded data underlying the plot was added in the caption of the tools-per-year plot. - A new meta-data configuration file was added to store basic project metadata to be used throughout the project. This will later be taken into Maneage. For examle the project title is now stored here and written into the paper's LaTeX source and output datasets automatically. - Verification was activated and plot's data and LaTeX macro files are now automatically verified. - A complete metadata was added for the data underlying the plot. - A generic function was added in 'initialize.mk' that will automatically write project info and copyright in all plain-text outputs.
2020-06-03Imported recent updated in Maneage, minor conflict fixedMohammad Akhlaghi-88/+123
The minor conflict was with 'reproduce/software/make/high-level.mk', and in particular because we implemented the fix to Maneage's Task #15664 in this project first. After it was moved to the main Maneage branch some minor stylistic corrections were done to it, thus causing the conflict. To resolve the conflict, I simply imported the full Maneage version of the file with this command: git checkout maneage -- reproduce/software/make/high-level.mk The other conflicts were due to the deleted files (that were resolved as described in 'README-hacking.md') and the LaTeX files that I had told '.gitattributes' to ignore from the Maneage branch.
2020-06-02Core software build before using Make to build other softwareMohammad Akhlaghi-23/+34
Until now, Maneage would only build Flock before building everything else using Make (calling 'basic.mk') in parallel. Flock was necessary to avoid parallel downloads during the building of software (which could cause network problems). But after recently trying Maneage on FreeBSD (which is not yet complete, see bug #58465), we noticed that the BSD implemenation of Make couldn't parse 'basic.mk' (in particular, complaining with the 'ifeq' parts) and its shell also had some peculiarities. It was thus decided to also install our own minimalist shell, Make and compressor program before calling 'basic.mk'. In this way, 'basic.mk' can now assume the same GNU Make features that high-level.mk and python.mk assume. The pre-make building of software is now organized in 'reproduce/software/shell/pre-make-build.sh'. Another nice feature of this commit is for macOS users: until now the default macOS Make had problems for parallel building of software, so 'basic.mk' was built in one thread. But now that we can build the core tools with GNU Make on macOS too, it uses all threads. Furthermore, since we now run 'basic.mk' with GNU Make, we can use '.ONESHELL' and don't have to finish every line of a long rule with a backslash to keep variables and such. Generally, the pre-make software are now organized like this: first we build Lzip before anything else: it is downloaded as a simple '.tar' file that is not compressed (only ~400kb). Once Lzip is built, the pre-make phase continues with building GNU Make, Dash (a minimalist shell) and Flock. All of their tarballs are in '.tar.lz'. Maneage then enters 'basic.mk' and the first program it builds is GNU Gzip (itself packaged as '.tar.lz'). Once Gzip is built, we build all the other compression software (all downloaded as '.tar.gz'). Afterwards, any compression standard for other software is fine because we have it. In the process, a bug related to using backup servers was found in 'reproduce/analysis/bash/download-multi-try' for calling outside of 'basic.mk' and removed Bash-specific features. As a result of that bug-fix, because we now have multiple servers for software tarballs, the backup servers now have their own configuration file in 'reproduce/software/config/servers-backup.conf'. This makes it much easier to maintain the backup server list across the multiple places that we need it. Some other minor fixes: - In building Bzip2, we need to specify 'CC' so it doesn't use 'gcc'. - In building Zip, the 'generic_gcc' Make option caused a crash on FreeBSD (which doesn't have GCC). - We are now using 'uname -s' to specify if we are on a Linux kernel or not, if not, we are still using the old 'on_mac_os' variable. - While I was trying to build on FreeBSD, I noticed some further corrections that could help. For example the 'makelink' Make-function now takes a third argument which can be a different name compared to the actual program (used for examle to make a link to '/usr/bin/cc' from 'gcc'. - Until now we didn't know if the host's Make implementation supports placing a '@' at the start of the recipe (to avoid printing the actual commands to standard output). Especially in the tarball download phase, there are many lines that are printed for each download which was really annoying. We already used '@' in 'high-level.mk' and 'python.mk' before, but now that we also know that 'basic.mk' is called with our custom GNU Make, we can use it at the start for a cleaner stdout. - Until now, WCSLIB assumed a Fortran compiler, but when the user is on a system where we can't install GCC (or has activated the '--host-cc' option), it may not be present and the project shouldn't break because of this. So with this commit, when a Fortran compiler isn't present, WCSLIB will be built with the '--disable-fortran' configuration option. This commit (task #15667) was completed with help/checks by Raul Infante-Sainz and Boud Roukema.
2020-05-29Reproducible research based on open-access papersBoud Roukema-0/+17
Publishing a paper on reproducible research without making it easy for readers to read the references would defeat the point. Of course we have to make some compromises with some journals' reluctance to shift towards the free world, but to satisfy scientific ethics, we should at least provide clickable URLs to the references, preferably to the ArXiv version if available [1], and also to the DOI, again, preferably to an open-access version of the URL if available. I was not able to fully get this done in the .bst file, so there's an sed/tr hack done to the .bbl file in `reproduce/analysis/make/paper.mk` to tidy up commas and spaces. This commit also reverts some of the hacks in the Akhlaghi IAU Symposium `tex/src/references.tex` entry, to match the improved .bst file, `tex/src/IEEEtran_openaccess.bst`, provided here with a different name to the original, in order to satisfy the LaTeX licence. [1] https://cosmo.torun.pl/blog/arXiv_refs
2020-05-22Corrected copyright notices to fit GPL suggested formatMohammad Akhlaghi-75/+116
In time, some of the copyright license description had been mistakenly shortened to two paragraphs instead of the original three that is recommended in the GPL. With this commit, they are corrected to be exactly in the same three paragraph format suggested by GPL. The following files also didn't have a copyright notice, so one was added for them: reproduce/software/make/README.md reproduce/software/bibtex/healpix.tex reproduce/analysis/config/delete-me-num.conf reproduce/analysis/config/verify-outputs.conf
2020-05-22Re-write of the paper to fit in ~6000 words and IEEE formatMohammad Akhlaghi-6/+11
Following the fact that the DSJ editor decided that this paper doesn't fit into their scope, we decided to submit it to IEEE's Computing in Science and Engineering (CiSE). So with this commit the text was re-written to fit into their style and word-count limitations.
2020-05-02First implementation of style in IEEEtran styleMohammad Akhlaghi-7/+14
The paper is no longer using LuaLaTeX, but raw LaTeX (that saves a DVI), it is so much faster! Initially I had used LuaLaTeX to use special fonts to resemble the CODATA Data Science Journal, but all that overhead is no longer necessary. Therefore I also removed the MANY extra LaTeX packages we were importing. The paper builds and is able to construct one of its images (the git-branching figure) with only 7 packages beyond the minimal TeX/LaTeX installation. Also in terms of processing it is so much faster. The text is just temporary now, and mainly just a place holder. With the next commit, I'll fill it with proper text.
2020-05-01Imported recent changes in Maneage, minor conflicts fixedMohammad Akhlaghi-82/+67
A few small conflicts showed up here and there. They are fixed with this merge.