aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorLines
2019-12-04Added pdftexmcds to the TeXlive packages to installMohammad Akhlaghi-1/+1
Until now we weren't including this package, but Alberto Madrigal recently reported that his build of the template failed because it needed it. With this commit, it is now being installed with the template.
2019-12-04Perl built without -shared on macOSMohammad Akhlaghi-1/+11
Until now, while building Perl, the `-Dlddlflags="-shared $$LDFLAGS"' option was being used for all systems. However, this caused `symbol(s) not found for architecture x86_64' problems on macOS systems. With this commit, this configure option is only passed for GNU/Linux systems. With it, Perl can be built nicely on both GNU/Linux and macOS.
2019-12-04Updated verions of Ghostscript and GnuastroMohammad Akhlaghi-4/+4
Newer versions of these packages have recently come out with major improvements, so they have been updated in the template.
2019-12-04Checking for necessary X11 libraries before building GhostscriptMohammad Akhlaghi-1/+35
Until now, Ghostscript was using some host system's X11 libraries during its build (and later at run-time). We should ideally install all these necessary libraries within the template (Task #15481). But right now we are too busy. As a temporary work-around we try building a small dummy program that links with some of those libraries before attempting to built Ghostscript. If it fails, then a notice is printed with the cause and explaining a temporary solution is suggested: how to install those libraries on the system when you have root access.
2019-11-29Download links directly to actual file if it exists in INDIRMohammad Akhlaghi-2/+8
Until now, when an input dataset already exists in `INDIR', the template would just make a symbolic link to it in the build directory. However, in many cases, the files in INDIR will actually be links to other locations on the filesystem and some programs have problems following too many links. With this commit, the template is now using the `readlink' program (part of GNU Coreutils) to follow a possible link and point the link in the build directory directly to an actual non-link file.
2019-11-24Project's Makefiles run with no builtin rules or variablesMohammad Akhlaghi-1/+2
Until now, after removing all environment variables, we were just giving Make the top Makefile to execute. By default, for every target, Make will try many built-in rules (which is good when compiling programs, but redundant in other cases). All these checkings also populate the debugging output of Make (with `-d'). So its easier and slightly faster to just tell Make to ignore builtin rules and variables. With this commit, to address this issue, the `project' script runs `.local/bin/make' with `--no-builtin-rules' and `--no-builtin-variables'.
2019-11-08Matplotlib problem on macOS system with GNU GCC fixedMohammad Akhlaghi-3/+9
Until now when building Matplotlib on macOS systems, we were using the default C compiler. However, while Yahya Sefidbakht (previously) and Mahdieh Nabavi (now) were trying to build the template, on their macOS using the GNU Compiler Collection (GCC), we found that Matplotlib needs special macOS headers that GCC doesn't recognize. With this commit, when Matplotlib is being built on macOS systems, it uses `clang' and this fixed the problem (so far checked on Mahdieh's machine).
2019-11-08Minor corrections applied in software buildsMohammad Akhlaghi-8/+13
While working on a different branch to build the GNU C Library, I noticed a few places in the template that need corrections which are now applied: 1. A new-line character after the "C compiler works" notice at the start of the configure script. 2. Removing possible `::' in the `LD_LIBRARY_PATH' definition of `basic.mk'. Note that its not necessary in the other steps because we don't use any outside-defined `LD_LIBRARY_PATH'. 3. Building GMP for C++ and also with `--enable-fat'. 4. Removing the unpacked Perl tarball directory after its installation.
2019-11-06GNU Project Debugger (GDB) and Pexpect can now be builtMohammad Akhlaghi-0/+18
These two packages are necessary to build the GNU C Library.
2019-11-06Perl is now built as a basic dependency of the templateMohammad Akhlaghi-5/+52
Perl is necessary to build Texinfo and later to build LaTeX. Until now we were just using the host operating system's installation of Perl, but in some instances that Perl can be too old and not suppor the features necessary. With this commit, Perl is now built from source during the basic installation step of the template. This was reported by Idafen Santana PĂ©rez, after trying the pipeline on an Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
2019-11-06Added 1911.01430 as most recent paper using this templateMohammad Akhlaghi-0/+7
Raul's paper (that uses this template) was just published on arXiv today (congratulations Raul!). So it has been added to the list of papers using this template.
2019-11-04Acknowledged support from the RDA EU 4.0 grantMohammad Akhlaghi-1/+4
Since we got the RDA Adoption grant, it was necessary to add it in the acknowledgements.
2019-11-03Corrected build of log4cxx by re-building the tarball usedMohammad Akhlaghi-15/+31
When building the log4cxx tarball from its Git history, I noticed that files with very long names are not packaged by tar (because by default Automake uses the ancient v7 tar format that only supports file names less than 99 characters). So I build the tarball with the `tar-ustar' option to Automake (by modifying the log4cxx source) and the resulting tarball was able to compile and run successfully. This has been described above the rule to build log4cxx and I also sent an email to their developing mailing list to inform them of this problem. If they address it, I will remove the note on the necessary corrections.
2019-10-31Minor corrections in distribution and autoconf prerequisite of automakeMohammad Akhlaghi-6/+37
Some minor corrections were made in the template: - When making the distribution, `.swp' files (created by Vim) are also removed. - Autoconf is set as a prerequisite of Automake I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball needs some patches, so I have just left it half done until someone actually needs it and we apply the patch.
2019-10-30Added ./project prepare in the checklist of README-hacking.mdMohammad Akhlaghi-2/+3
Since adding this new step, I had forgot to mention it in the checklist of `README-hacking.md'. It is added with this commit.
2019-10-30Tarballs set as order-only prerequisites, not needed for build callsMohammad Akhlaghi-371/+427
Until now, the tarballs were the first normal prerequisite of the software. As a result if their date changed, the whole software would be re-built. However, for tarballs specifically, we actually check their contents with a checksum, so their date is irrelevant (if its newer than the built program, but has the same checksum, there is no need to re-build the software). Also, calling the tarball name as an argument to the building process (for example `gbuild') was redundant. It is now automatically found from the list of order-only prerequisites within `gbuild' and `cbuild' (similar to how it was previously found in the `pybuild' for Python building). A `README.md' file has also been placed in `reproduce/software/make' to help describe the shared properties of the software building Makefiles. This will hopefully grow much larger in the future.
2019-10-29lmfit and its dependencies added, Python tarballs as prerequisiteMohammad Akhlaghi-189/+260
Python's `lmfit' module and all its major dependencies (`asteval', `corner', `emcee' and `uncertainties') have been included in the template. While doing this I noticed that if the tarballs are the last prerequisite of each software building rule, then when building in parallel, the template will immediately start building packages as soon as the first one is downloaded. Not like the current way that it will attempt to download several, then start building. For now, this has been implemented in the Python build rules for all the modules and we'll later do the same for the other programs and libraries. This also motivated a simplification of the `pybuild' function: it now internally looks into the prerequisites and selects the tarball from the prerequisite that is in the tarballs directory. This isn't a problem for the build, but I just don't understand why Python can't recognize the version of `emcee', Python reads the version of `emcee' as `0.0.0'! But it doesn't cause any crash in the build, so for now its fine.
2019-10-29Minor edits to README-hacking.mdMohammad Akhlaghi-28/+31
The part on using shared memory was edited for a few things that weren't clear.
2019-10-29Further small edit in using shared memory of README-hacking.mdMohammad Akhlaghi-3/+2
Some typos were fixed.
2019-10-29Minor edits to suggestion on using shared memoryMohammad Akhlaghi-3/+9
The edits help it be more clear, and remind the reader to delete any remaining file in the RAM in the end.
2019-10-29Suggestion on good usage of /dev/shm in README-hacking.mdMohammad Akhlaghi-6/+57
When you are working with large files and there is some good RAM in the system (large/powerful computers), it is beneficial to work in the shared memory directory and not the actual persistent storage (like HDD or SSD). With this commit, a fully working demo has been added to `README-hacking.md' (under the tips of "Make programming") to show how to effectively work in situations like this.
2019-10-28HEALPix, Autoconf and Automake now in libraryMohammad Akhlaghi-1/+99
The tarball of HEALPix includes multiple languages and doesn't include the ready-to-run GNU Build System by default, we actually have to build the `./configure' script for the C/C++ libraries. So it was necessary to also include GNU Autoconf and GNU Automake as prerequisites of HEALPix. However, the official GNU Autoconf tarball (dating from 2012) doesn't build on modern systems, so I just cloned it from its source and bootstrapped it and built its modern tarball which we are using here.
2019-10-24Several more LSST pipeline dependencies are addedMohammad Akhlaghi-13/+144
The following software are added with this commit: eigency, esutil, flake8, future, galsim, lsstdesccoord, pybind11 and pyflakes.
2019-10-24APR, APR Utility, Boost and Eigen are now availableMohammad Akhlaghi-2/+70
As part of an effort to bring in all the dependencies of the LSST Science pipeline (which includes the last commit), these software are now available in the template.
2019-10-24GNU Bison, GNU Help2man and Flex are now availableMohammad Akhlaghi-3/+26
With this commit these three software packages are now installable with this template.
2019-10-19Minor improvments in packaging of project with make distMohammad Akhlaghi-12/+14
The steps to package the project have been made slightly more clear and also the temporary directory that is created for packaging is deleted after the tarball is made.
2019-10-11Properly working make clean when in group modeMohammad Akhlaghi-3/+4
Until now, when you ran `make clean', all the directories under `$(BDIR)/tex/' would be deleted except for `macros' and `build'. This was good for the single-user mode. But in group mode, this would delete the user-specific TeX build directory because its called `build-USER', not `build'. With this commit, to fix the problem, we define the new `texbtopdir' and based on the group condition, and use that to specify which directory to not delete.
2019-10-10Including trimspaces from TeXLiveMohammad Akhlaghi-1/+2
Carlos Morales Socorro reported that his LaTeX build was missing the `trimspaces' package, so it is now included in the pipeline.
2019-10-07Software citations now use modern ADS link for adsurlMohammad Akhlaghi-9/+9
Until now, some software were using the depreciated ADS URL (`http://adsabs.harvard.edu....') and some were using the modern one (`http://ui.adsabs.harvard.edu'). With this commit, to be consistent they all link to the modern URL.
2019-10-02Possibile to use download-multi-try script without locksMohammad Akhlaghi-4/+11
Until now, this script would always only work with a file-lock. But in some scenarios, we might want to download in parallel. For example when the system has multiple ports to the internet. With this commit, we have added this feature: when the lockfile name is `nolock', it won't lock and will download in parallel.
2019-10-01Minor corrections in configure and prepare phaseMohammad Akhlaghi-12/+11
Since ImageMagick can take long to build, we are now building it in parallel. Also, the part where we replace an `_' with `\_' in the software version at the end of the configure script was removed. It is more clear/readable that the actual rule that includes such a name deals with the underline (as is the case for `sip_tpv' which already dealt with it). Finally, I noticed that the checks at the start of `top-prepare' were missing new-lines. I had forgot that the Make single-shell variable isn't activated in this stage yet.
2019-10-01Infrastructure to keep preparation resultsMohammad Akhlaghi-18/+51
A special directory is now defined in `initialize.mk' that can be used in both the preparation and build phases. Also, the contents of prepared results can now be conditionally read during `./project make'.
2019-10-01Preparation phase added before final buildingMohammad Akhlaghi-74/+297
In many real-world scenarios, `./project make' can really benefit from having some basic information about the data before being run. For example when quering a server. If we know how many datasets were downloaded and their general properties, it can greatly optmize the process when we are designing the solution to be run in `./project make'. Therefore with this commit, a new phase has been added to the template's design: `./project prepare'. In the raw template this is empty, because the simple analysis done in the template doesn't warrant it. But everything is ready for projects using the template to add preparation phases prior to the analysis.
2019-10-01New versions for ImageMagick, Python, Numpy Scipy and MatplotlibMohammad Akhlaghi-19/+19
It was some time since these three software were not updated! With this commit the template now uses the most recent stable release of these packages. Also, the hosting server for ImageMagick was moved to my own webpage because unfortunately ImageMagick removes its tarballs from its own version.
2019-09-28Main project script sets executable flags at configure timeMohammad Akhlaghi-3/+23
Until now, when the project's source was downloaded from something like arXiv, in `README.md', we were instructing them to set the executable flags of all the files that need it. But except for `./project', the reader shouldn't have to worry about the project internals! Once its executable, `./project' can easily fix the executable flags of all the files that need it automatically. With this commit, in `README.md', we just instruct the reader to set the executable flag of `./project' and any other file that needs an executable flag is given one at the start of the set of commands for `./project configure'. In customized projects, if an author needs executable flags on any other files, they can easily add it there without involving the user.
2019-09-28Corrected name of SExtractor in its final targetMohammad Akhlaghi-1/+1
Until now we were calling it `Sextractor', but the official way of writing it is `SExtractor'. With this commit, this has been corrected.
2019-09-27Changing to the cloned directory added to README.mdMohammad Akhlaghi-0/+1
Konrad Hinsen pointed out that this part was missing from the instructions in `README.md' after cloning. So it is added.
2019-09-26Minor edits/clarifications in README.mdMohammad Akhlaghi-13/+7
The two modifications to the LaTeX source of an arXiv-downloaded source weren't rendered properly on Gitlab, so they are corrected to be in the same line and not have a separate code-block.
2019-09-26Working project when downloaded from arXivMohammad Akhlaghi-15/+113
Until now, we were assuming that the users would just clone the project in Git. But after submitting arXiv:1909.11230, and trying to build directly from the arXiv source, I noticed several problems that wouldn't allow users to build it automatically. So I tried the build step by step and was able to find a fix for the several issues that came up. The scripting parts of the fix were primarily related to the fact that the unpacked arXiv tarball isn't under version control, so some checks had to be put there. Also, we wanted to make it easy to remove the extra files, so an extra `--clean-texdit' option was added to `./project'. Finally, some manual corrections were necessary (prior to running `./project', which are now described in `README.md'. Most of the later steps can be automated and we should do it later, I just don't have enough time now.
2019-09-26Minor edit in README-hacking.mdMohammad Akhlaghi-3/+2
The description of arXiv:1909.11230 was slightly modified to be in the same format as the other papers.
2019-09-26arXiv:1909.11230 added as example paper using this templateMohammad Akhlaghi-4/+11
This paper was published on arXiv today and is a good example for people to see the application of this system in practice.
2019-09-25Won't copy previous distribution builds in new distributionMohammad Akhlaghi-1/+1
Until now, the pipeline was instructed to only ignore the current temporary project distribution directory. So if there were directories from previous builds, they were wrongly included in the current tarball. With this commit, we don't just ignore the directory of the current distribution, but generally, all directories starting with `paper-v*'.
2019-09-18README.md written to be more generic and easy to customizeMohammad Akhlaghi-37/+38
Until now customizing it was a little more detailed, for example the copyright statement wasn't generic and was about "this template". So the user would have to correct it. With this commit, the copyright statment just says "this project", so it can apply to the raw template and also any customization of it. Also, some minor edits were made in the various parts of the text to make it more clear.
2019-09-16Configure script won't allow build directory to be under sourceMohammad Akhlaghi-4/+28
Users that are not familiar with the file structure of the project may specify the current directory (to-level source directory) as their build-directory. This will cause a crash right after answering the questions, where `rm' will complain about `tex/build' not being deleted because it exists as a directory. To avoid such confusing situtations, the configure script now checks if the build directory is actually a sub-directory of the source. If it is, it will complain with a short message and abort. Also, a `CAUTION' statment has been put in the initial description, right ontop of the question. This bug was reported Carlos Allende Prieto and David Valls-Gabaud.
2019-09-16Git checksum printed even when on a tagMohammad Akhlaghi-2/+2
Until now, when the commit was tagged, `git describe' would just print the tag and no longer the commit checksum. This is bad because the checksum is a much more robust way to confirm the point in history. With this commit the `--long' option has been added to `git describe' to fix this issue. From now on, when we are on a tag, it will print the tag followed by a `-0-' and the first characters of the checksum.
2019-09-16Distribution tarball now builds in arXivMohammad Akhlaghi-19/+45
`./project make dist' will package all the LaTeX-specific files (and analysis source files) into one `tar.gz' file that is ready to upload to servers like arXiv. However, it wasn't updated for some time, so running it would complain about not having a `configure' script in the top of the project. With this commit, it now works with the new file-structure of the project and also copies all the BibLaTeX source files and `paper.bbl' into the top tarball directory, which allows arXiv to build the paper as intended. The output of `./project make dist' has been uploaded and tested on arXiv and it is built by arXiv perfectly. Also, a short description of all the special `make' targets was added to the output of `./project --help'.
2019-09-15Minor edits on the git bundle suggestion of README-hacking.mdMohammad Akhlaghi-8/+9
After a re-read on Gitlab, it has been slightly edited to be more clear.
2019-09-15Added tip on bundling Git history in one fileMohammad Akhlaghi-0/+24
When you want to publish your project, it is very convenient to have a single file that contains the whole history. So a tip is added to `README-hacking.md' that describes how to do this with `git bundle'.
2019-09-15BibLaTeX configured to only print initials of given namesMohammad Akhlaghi-0/+1
Until now, we weren't explicitly asking for BibLaTeX to build the citations with only initial of the author's given names. Therefore when one BibTeX entry had a full given name and another had only initials, this would also be reflected in the paper's bibliography. With this commit, the `giveninits=true' option is given to BibLaTeX to always only print the initial character of an author's given name.
2019-09-15SExtractor BibTex entry updated with ADS's exported formatMohammad Akhlaghi-10/+13
Until now the SExtractor BibTeX entry has the first-name and family name in the same field, while the family-name should be in curly braces. Also, it was missing an ADS URL. So the default ADS generated BibTeX entry for SExtractor is now used instead of the old one.