aboutsummaryrefslogtreecommitdiff
path: root/.file-metadata
AgeCommit message (Collapse)AuthorLines
2020-06-19Removing preparation-done.mk when cleaning by ./project make cleanRaul Infante-Sainz-0/+0
Until this commit, the file `BDIR/software/preparation-done.mk' were not removed when cleaning the project with `./project make clean'. This file is generated in the preparation of the data during the analysis step. However, the cleaning is expected to remove anything generated in the analysis process! Step by step, with the commands: ./project make ---> Will make the preparation and analysis ./project make clean ---> Will remove all analysis outputs (but not `preparation-done.mk') ./project make ---> Won't do the preparation, only analysis! However, in the last step it should do the preparation again, because the input data could have change for any reason. With this commit, the file `BDIR/software/preparation-done.mk' is removed when cleaning the project, and consequently, in the analysis step the input data is prepared.
2020-06-18Fixed small bug that was introduced four commits agoRaul Infante-Sainz-0/+0
In Commit 105467fe6402 (Software tarballs are downloaded even if not built), we introduced tests to download the tarballs of software even if they don't need to be built on the respective host. However some small typos in the checks existed that could cause a crash on macOS. In particular in the building of PatchELF and libbsd we had forgot to add the necessary 'x' before the 'yes' in the conditional to check if a we are on macOS or not. With this commit these two checks have been corrected. Also, in the building of 'isl' and 'mpc', we now check for 'host_cc' (signifying that the user wants to use their host C compiler for the high-level step) instead of 'on_mac_os'. The reason is that even on non-macOS systems, a user may not want to build the C compiler from scratch and use the '--host-cc' option. In such cases, they don't need to compile 'isl' and 'mpc'.
2020-05-23New software: Valgrind and PatchBoud Roukema-0/+0
With this commit, Maneage now includes instructions to build the memory tracing tool Valgrind and the program 'patch' (to apply corrections/patches in text files and in particular the sources of programs). For this version of Valgrind, some patches were necessary for an interface with OpenMPI 2.x (which is the case now). Also note that this version of Valgrind's checks can fail with GCC 10.1.x (when using '--host-cc'), and the failures aren't due to internal problems but due to how the tests are designed (https://bugs.gentoo.org/707598). So currently if any of Valgrind's checks fail, Maneage still assumes that Valgrind was built and installed successfully. While testing on macOS, we noticed that it needs the macOS-specific 'mig' program which we can't build in Maneage. DESCRIPTION: The mig command invokes the Mach Interface Generator to generate Remote Procedure Call (RPC) code for client-server style Mach IPC from specification files. So a symbolic link to the system's 'mig' is now added to the project's programs on macOS systems. This commit's build of Patch and Valgrind has been tested on two GNU/Linux distributions (Debian and ArchLinux) as well as macOS. Work on this commit started by Boud Roukema, but also involved tests and corrections by Mohammad Akhlaghi and Raul Infante-Sainz.
2020-05-08GNU Gettext built as a dependency of BashRaul Infante-Sainz-0/+0
Until now Maneage used the host's GNU Gettext if it was present. Gettext is a relatively low-level software that enables programs to print messages in different languages based on the host environment. Even though it has not direct effect on the running of the software for Maneage and the lanugage environment in Maneage is pre-determined, it is necessary to have it because if the basic programs see it in the host they will link with it and will have problems if/when the host's Gettext is updated. With this commit (which is actually a squashed rebase of 9 commits by Raul and Mohammad), Gettext and its two extra dependencies (libxml2 and libunistring) are now installed within Maneage as a basic software and built before GNU Bash. As a result, all programs built afterwards will successfully link with our own internal version of Gettext and libraries. To get this working, some of the basic software dependencies had to updated and re-ordered and it has been tested in both GNU/Linux and macoS. Some other minor issues that are fixed with this commit - Until this commit, when TeX was not installed, the warning message saying how to run the configure step in order to re-configure the project was not showing the option `-e'. However, the use of this option is more convenient than entering the top-build directory and etc every time. So with this commit, the warning message has been changed in order use the option `-e' in the re-configure of the project. - Until now, on macOS systems, Bash was not linking with our internally built `libncurses'. With this commit, this has been fixed by setting `--withcurses=yes' for Bash's configure script.
2020-04-06Astropy now depending on the Expat library to fix internal conflictRaul Infante-Sainz-0/+0
Until now, Astropy was instructed to build its own internal copy of the Expat library. However, with the recent commits before, Maneage now includes an installation of Expat and Astropy can't keep the two (its internal version and the project's version) separate, so they conflict and don't let Astropy get built. With this commit, the problem is fixed by setting the Expat library as an explicit dependency of Astropy and asking Astropy to ignore its internal copy. While doing this, I recognized that it is much easier and elegant to add steps in various stages of the `pybuild' function through hooks instead of variables. So the fifth argument of the `pybuild' function was removed and now it actually checks if hooks are defined as functions and if so, they will be called. The `pyhook_after' function was also implemented in the installation of `pybind11' (which needed it, given that the 5th argument of `pybuild' was removed) and after doing a test-build, I noticed that two lines were not ending with a `\' in `boost' (a dependency of `pybind11'). Commit written originally by Mohammad Akghlaghi
2020-04-05Commenting version numbers with an underscore for LaTeXRaul Infante-Sainz-0/+0
Until now we would simply return the version numbers as they were written into the separate files and situations can happen where the version numbers contain an underscore (`_'). However, this character is a methematical character in LaTeX, causing LaTeX to complain and abort. With this commit, a step has been added at the end of the configure script to convert any possible `_' to `\_'. Once it is commented (a backslash is put behind it), the underscore will be printed as it is in the final PDF. This commit was originally written by Mohammad Akhlaghi
2020-04-05The build of M4 and XLSX I/O on macOS has been fixedRaul Infante-Sainz-0/+0
Until now, the M4 that was built on macOS had internal problems (as discussed in #1): it would simply print `Abort trap: 6' in the output and abort. After looking at the build of Homebrew, I noticed that they apply a patch (correct one line) to fix this problem. To be able to apply that patch on macOS systems, I had fully open up the build recipe of M4 and atleast on the testing system, it was built successfully. Also, after successfully building M4, and thus Autoconf and thus Minizip, we were able to build XLSX I/O on a macOS and found out that the internal library's full address wasn't being put in the libraries and executables. With this commit, we now use macOS's `install_name_tool' to correct the positions of the two `libxlsxio_*' libraries in all its executables. This commit was originally written by Mohammad Akhlaghi
2020-04-05Including version number of Minizip in its installation targetRaul Infante-Sainz-0/+0
Until this commit, only the word `Minizip' was written into the Minizip installation target (without the version number of Minizip). With this commit, this minor bug has been fixed by using the appropiate Make variable: `$(minizip-version)'.
2020-04-05Building Minizip 1.x instead of Minizip 2.xRaul Infante-Sainz-0/+0
Minizip is a dependency of XLSX I/O and until now, I was just using the most recent version I found (2.9.2), but XLSX I/O is written for the Minizip 1.x series, not 2.x. Somehow it didn't cause a crash on my computer!!! I think XLSX I/O's CMake is instructed to look into system directories by default when it doesn't find the directories in the given places. And because I had installed Minizip on my operating system, it did't complain. Upon trying the build on their systems, Yahya, Raul and Zahra reported a failure in the build of XLSX I/O which was due the to the problem above (we were installing the wrong version of Minizip!). With this commit, this has been fixed by installing the 1.x series of Minizip (whish is actually installed within zlib!). This commit was original done by Mohammad Akhlaghi.
2020-04-03CMake updated to version 3.17.0Raul Infante-Sainz-0/+0
With this commit, CMake has been updated to its most recent version. This upgrade has been done because in the installation of XLSX I/O on macOS laptop, it crashes complaining about C compiler "not able to compile a simple test program". After a fast search, I found it could be possible to just use the most recent version of CMake to solve the problem. But it didn't work. In any case, it is good to have the most recent version of CMake included.
2020-04-01Corrected reference for Infante-Sainz+2020 in README-hacking.mdRaul Infante-Sainz-0/+0
Until this commit, the year of this paper was 2019 and the linking url was the temporal one. However, the final official publication year is 2020. With this commit, the year and the url have been changed to the final ones.
2020-04-01Removed multiple tabs in MissFITS tarball definitionRaul Infante-Sainz-0/+0
With this commit, multiples tabs in the definition of MissFITS tarball have been removed. Now they are white spaces.
2020-03-20Adding PyYAML, Html5lib, and Beautifulsoup4 as prerequsites of AstropyRaul Infante-Sainz-0/+0
Until this commit, PyYAML was not set as prerequisite of Astropy. This package is an optional dependency of Astropy for some particular functions. However, we have already included PyYAML into this project so it is available. With this commit, PyYAML has been set as a prerequisite of Astropy. In addition to this, Html5lib and Beautifulsoup4 have been also added as prerequsites of Astropy (and removed from Astroquery prerequisites). I noticed that both of them are optional dependencies of Astropy.
2020-03-17Astroquery updated to version 0.4Raul Infante-Sainz-0/+0
In the last update of Astropy to version 4.0 they removed some things that the previous version of Astroquery needs. As a consequence, it is also necessary to update the Astroquery version to be a ble to run with the Astropy 4.0. With this commit, the update of Astroquery to it most recent version (0.4) has been done.
2020-01-27Moving basic configuration of Git section in README-hacking.mdRaul Infante-Sainz-0/+0
Until this commit, the small section of `README-hacking.md' in which it is explained how to do the first configuration of Git was at the beginning of the section `First custom commit'. However, it is better to have it just before the item `Your first commit' in that section. With this commit, this change has been done. Now the reader has the necessary steps for configuring Git just before it is needed for making the first commit.
2020-01-22Adding Raul as contributor of README-hacking.mdRaul Infante-Sainz-0/+0
Since I (Raul) did some changes (and I hope to do more :-)) in the `README-hacking.md', I am adding my information at the beginning of this file.
2020-01-22Adding basic configuration of Git in README-hacking.mdRaul Infante-Sainz-0/+0
Until this commit, we were asuming that Git was already properly configured. However, in order to be as complete as possible, it would be good if the basic commands to configure Git were in the `README-hacking.md'. With this commit, a small paragraph has been added in order to have the basic Git configuration commands (i.e. to configure the name, email, and favorite text editor).
2020-01-13Added Infante-Sainz et al. 2019 as most recent paper using this templateRaul Infante-Sainz-0/+0
The "SDSS extended PSFs" paper was already included as an example of papers wich uses this template. However, the reference was the arXiv one. With this commit, since the paper has been finally published, it has been added the final reference to the journal.
2020-01-13Adding X11 libraries path to LDFLAGS in Ghostscript installationRaul Infante-Sainz-0/+0
Until this commit, the checking of X11 installation done to ensure that it is already available in the host system was crashing in macOS systems. The reason is that the place of the X11 libraries use to be `/opt/X11/lib' in macOS systems. With this commit, this issue has been fixed by adding this directory to the LDFLAGS.
2020-01-13Updated verion of Astropy to 4.0 the most recent oneRaul Infante-Sainz-0/+0
Newer versions of Astropy package has been released. With this commit, it has been updated. It has been increased from v3.2.1 to v4.0
2019-12-18Fixing Libxml2 checksum variable nameRaul Infante-Sainz-0/+0
Until this commit, the number `2' was missing in the checksum variable name of that library. It was `libxml-checksum' but it should be `libxml2-checksum'. With this commit, this issue has been fixed.
2019-12-18Removed extra backslash in Jeepney prerequisitesRaul Infante-Sainz-0/+0
An extra backslash in the prerequisites of Jeepney Python package was causing a crash in the installation of that software. With this commit, this problem has been fixed by removing the backslash.
2019-11-06GNU Project Debugger (GDB) and Pexpect can now be builtMohammad Akhlaghi-0/+0
These two packages are necessary to build the GNU C Library.
2019-11-06Perl is now built as a basic dependency of the templateMohammad Akhlaghi-0/+0
Perl is necessary to build Texinfo and later to build LaTeX. Until now we were just using the host operating system's installation of Perl, but in some instances that Perl can be too old and not suppor the features necessary. With this commit, Perl is now built from source during the basic installation step of the template. This was reported by Idafen Santana PĂ©rez, after trying the pipeline on an Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
2019-11-06Added 1911.01430 as most recent paper using this templateMohammad Akhlaghi-0/+0
Raul's paper (that uses this template) was just published on arXiv today (congratulations Raul!). So it has been added to the list of papers using this template.
2019-11-04Acknowledged support from the RDA EU 4.0 grantMohammad Akhlaghi-0/+0
Since we got the RDA Adoption grant, it was necessary to add it in the acknowledgements.
2019-11-03Corrected build of log4cxx by re-building the tarball usedMohammad Akhlaghi-0/+0
When building the log4cxx tarball from its Git history, I noticed that files with very long names are not packaged by tar (because by default Automake uses the ancient v7 tar format that only supports file names less than 99 characters). So I build the tarball with the `tar-ustar' option to Automake (by modifying the log4cxx source) and the resulting tarball was able to compile and run successfully. This has been described above the rule to build log4cxx and I also sent an email to their developing mailing list to inform them of this problem. If they address it, I will remove the note on the necessary corrections.
2019-10-31Minor corrections in distribution and autoconf prerequisite of automakeMohammad Akhlaghi-0/+0
Some minor corrections were made in the template: - When making the distribution, `.swp' files (created by Vim) are also removed. - Autoconf is set as a prerequisite of Automake I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball needs some patches, so I have just left it half done until someone actually needs it and we apply the patch.
2019-10-30Added ./project prepare in the checklist of README-hacking.mdMohammad Akhlaghi-0/+0
Since adding this new step, I had forgot to mention it in the checklist of `README-hacking.md'. It is added with this commit.
2019-10-30Tarballs set as order-only prerequisites, not needed for build callsMohammad Akhlaghi-0/+0
Until now, the tarballs were the first normal prerequisite of the software. As a result if their date changed, the whole software would be re-built. However, for tarballs specifically, we actually check their contents with a checksum, so their date is irrelevant (if its newer than the built program, but has the same checksum, there is no need to re-build the software). Also, calling the tarball name as an argument to the building process (for example `gbuild') was redundant. It is now automatically found from the list of order-only prerequisites within `gbuild' and `cbuild' (similar to how it was previously found in the `pybuild' for Python building). A `README.md' file has also been placed in `reproduce/software/make' to help describe the shared properties of the software building Makefiles. This will hopefully grow much larger in the future.
2019-10-29lmfit and its dependencies added, Python tarballs as prerequisiteMohammad Akhlaghi-0/+0
Python's `lmfit' module and all its major dependencies (`asteval', `corner', `emcee' and `uncertainties') have been included in the template. While doing this I noticed that if the tarballs are the last prerequisite of each software building rule, then when building in parallel, the template will immediately start building packages as soon as the first one is downloaded. Not like the current way that it will attempt to download several, then start building. For now, this has been implemented in the Python build rules for all the modules and we'll later do the same for the other programs and libraries. This also motivated a simplification of the `pybuild' function: it now internally looks into the prerequisites and selects the tarball from the prerequisite that is in the tarballs directory. This isn't a problem for the build, but I just don't understand why Python can't recognize the version of `emcee', Python reads the version of `emcee' as `0.0.0'! But it doesn't cause any crash in the build, so for now its fine.
2019-10-29Minor edits to README-hacking.mdMohammad Akhlaghi-0/+0
The part on using shared memory was edited for a few things that weren't clear.
2019-10-29Further small edit in using shared memory of README-hacking.mdMohammad Akhlaghi-0/+0
Some typos were fixed.
2019-10-29Minor edits to suggestion on using shared memoryMohammad Akhlaghi-0/+0
The edits help it be more clear, and remind the reader to delete any remaining file in the RAM in the end.
2019-10-29Suggestion on good usage of /dev/shm in README-hacking.mdMohammad Akhlaghi-0/+0
When you are working with large files and there is some good RAM in the system (large/powerful computers), it is beneficial to work in the shared memory directory and not the actual persistent storage (like HDD or SSD). With this commit, a fully working demo has been added to `README-hacking.md' (under the tips of "Make programming") to show how to effectively work in situations like this.
2019-10-28HEALPix, Autoconf and Automake now in libraryMohammad Akhlaghi-0/+0
The tarball of HEALPix includes multiple languages and doesn't include the ready-to-run GNU Build System by default, we actually have to build the `./configure' script for the C/C++ libraries. So it was necessary to also include GNU Autoconf and GNU Automake as prerequisites of HEALPix. However, the official GNU Autoconf tarball (dating from 2012) doesn't build on modern systems, so I just cloned it from its source and bootstrapped it and built its modern tarball which we are using here.
2019-10-24Several more LSST pipeline dependencies are addedMohammad Akhlaghi-0/+0
The following software are added with this commit: eigency, esutil, flake8, future, galsim, lsstdesccoord, pybind11 and pyflakes.
2019-10-24APR, APR Utility, Boost and Eigen are now availableMohammad Akhlaghi-0/+0
As part of an effort to bring in all the dependencies of the LSST Science pipeline (which includes the last commit), these software are now available in the template.
2019-10-24GNU Bison, GNU Help2man and Flex are now availableMohammad Akhlaghi-0/+0
With this commit these three software packages are now installable with this template.
2019-10-19Minor improvments in packaging of project with make distMohammad Akhlaghi-0/+0
The steps to package the project have been made slightly more clear and also the temporary directory that is created for packaging is deleted after the tarball is made.
2019-10-11Properly working make clean when in group modeMohammad Akhlaghi-0/+0
Until now, when you ran `make clean', all the directories under `$(BDIR)/tex/' would be deleted except for `macros' and `build'. This was good for the single-user mode. But in group mode, this would delete the user-specific TeX build directory because its called `build-USER', not `build'. With this commit, to fix the problem, we define the new `texbtopdir' and based on the group condition, and use that to specify which directory to not delete.
2019-10-10Including trimspaces from TeXLiveMohammad Akhlaghi-0/+0
Carlos Morales Socorro reported that his LaTeX build was missing the `trimspaces' package, so it is now included in the pipeline.
2019-10-07Software citations now use modern ADS link for adsurlMohammad Akhlaghi-0/+0
Until now, some software were using the depreciated ADS URL (`http://adsabs.harvard.edu....') and some were using the modern one (`http://ui.adsabs.harvard.edu'). With this commit, to be consistent they all link to the modern URL.
2019-10-02Possibile to use download-multi-try script without locksMohammad Akhlaghi-0/+0
Until now, this script would always only work with a file-lock. But in some scenarios, we might want to download in parallel. For example when the system has multiple ports to the internet. With this commit, we have added this feature: when the lockfile name is `nolock', it won't lock and will download in parallel.
2019-10-01Minor corrections in configure and prepare phaseMohammad Akhlaghi-0/+0
Since ImageMagick can take long to build, we are now building it in parallel. Also, the part where we replace an `_' with `\_' in the software version at the end of the configure script was removed. It is more clear/readable that the actual rule that includes such a name deals with the underline (as is the case for `sip_tpv' which already dealt with it). Finally, I noticed that the checks at the start of `top-prepare' were missing new-lines. I had forgot that the Make single-shell variable isn't activated in this stage yet.
2019-10-01Infrastructure to keep preparation resultsMohammad Akhlaghi-0/+0
A special directory is now defined in `initialize.mk' that can be used in both the preparation and build phases. Also, the contents of prepared results can now be conditionally read during `./project make'.
2019-10-01Preparation phase added before final buildingMohammad Akhlaghi-0/+0
In many real-world scenarios, `./project make' can really benefit from having some basic information about the data before being run. For example when quering a server. If we know how many datasets were downloaded and their general properties, it can greatly optmize the process when we are designing the solution to be run in `./project make'. Therefore with this commit, a new phase has been added to the template's design: `./project prepare'. In the raw template this is empty, because the simple analysis done in the template doesn't warrant it. But everything is ready for projects using the template to add preparation phases prior to the analysis.
2019-10-01New versions for ImageMagick, Python, Numpy Scipy and MatplotlibMohammad Akhlaghi-0/+0
It was some time since these three software were not updated! With this commit the template now uses the most recent stable release of these packages. Also, the hosting server for ImageMagick was moved to my own webpage because unfortunately ImageMagick removes its tarballs from its own version.
2019-09-28Main project script sets executable flags at configure timeMohammad Akhlaghi-0/+0
Until now, when the project's source was downloaded from something like arXiv, in `README.md', we were instructing them to set the executable flags of all the files that need it. But except for `./project', the reader shouldn't have to worry about the project internals! Once its executable, `./project' can easily fix the executable flags of all the files that need it automatically. With this commit, in `README.md', we just instruct the reader to set the executable flag of `./project' and any other file that needs an executable flag is given one at the start of the set of commands for `./project configure'. In customized projects, if an author needs executable flags on any other files, they can easily add it there without involving the user.
2019-09-28Corrected name of SExtractor in its final targetMohammad Akhlaghi-0/+0
Until now we were calling it `Sextractor', but the official way of writing it is `SExtractor'. With this commit, this has been corrected.