aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorLines
2020-03-23Analysis and configuration file sections completeMohammad Akhlaghi-89/+348
With this commit a description of these two important parts have been added to the project, along with several figures showing various parts of the files that are discussed. I also done some other restructuring of the figures and files to make things fit better into the the description of the paper.
2020-03-20Adding PyYAML, Html5lib, and Beautifulsoup4 as prerequsites of AstropyRaul Infante-Sainz-4/+4
Until this commit, PyYAML was not set as prerequisite of Astropy. This package is an optional dependency of Astropy for some particular functions. However, we have already included PyYAML into this project so it is available. With this commit, PyYAML has been set as a prerequisite of Astropy. In addition to this, Html5lib and Beautifulsoup4 have been also added as prerequsites of Astropy (and removed from Astroquery prerequisites). I noticed that both of them are optional dependencies of Astropy.
2020-03-17Astroquery updated to version 0.4Raul Infante-Sainz-3/+3
In the last update of Astropy to version 4.0 they removed some things that the previous version of Astroquery needs. As a consequence, it is also necessary to update the Astroquery version to be a ble to run with the Astropy 4.0. With this commit, the update of Astroquery to it most recent version (0.4) has been done.
2020-03-08Menke+20 example: properly count number of papers with softwareMohammad Akhlaghi-5/+17
Until now, I was mistakenly multiplying the fraction of papers in that journal. This is corrected with this commit.
2020-03-08Edited description of example subMakefile for analysis-1Mohammad Akhlaghi-27/+48
In order to make the description more clear and readable, the rules in the demonstrated Makefile (and their links to the data lineage plot) were made more clear.
2020-03-02Described the first analysis phase with a demo subMakefileMohammad Akhlaghi-98/+383
Until now, there was no explanation on an actual analysis phase, therefore with this commit an example scenario with a readable Makefile is included. The Data lineage graph was also simplified to both be more readable, and also to correspond to this new explanation and subMakefile. Some random edits/typos were also corrected and some references added for discussion.
2020-02-29IMPORTANT: re-preparation can only be done with --prepare-redoMohammad Akhlaghi-18/+21
Until now, the preparation phase was always executed before the final build phase when running `./project make'. But when it becomes necessary, project preparation can be slow and will un-necessarily slow down the project while the project is growing (focus is on the analysis that is done after preparation). With this commit, preparation will be done automatically the first time that the project is run (`.build/software/preparation-done.mk' doesn't exist). However, after preperation is complete once, future runs of `./project make' won't do preparation any more (by calling `top-prepare.mk'). They will directly call `top-make.mk' for the analysis. To manually invoke preparation after the first attempt, the `./project make' script should be run with the new `--prepare-redo' option. Also, since the preparation phase is now automatically done before the analysis phase, the long notice that describes running `./project make' at the end of the preparation phase has been removed in `top-prepare.mk'. It now just prints a short line, saying the preparation has been complete. Finally, when the project has not been run with the proper group configuration, it ends with an `exit 1' so the main `./project' script doesn't proceed any further.
2020-02-24MissFITS is now added to the templateSurena Fatemi-1/+28
MissFITS is package for manipulating FITS files. I added it as my first commit to the project for educational purposes.
2020-02-20Preparation phase: prepare.tex not needed to finish preparationMohammad Akhlaghi-1/+3
Until now, the final preparation target of the preparation phase depended on all the `$(makesrc)' files. This caused a problem because we were telling it to also depend on `prepare.tex' (which is the same file that is being built). With this commit, we are applying the same solution we have already done in `paper.mk' (for `paper.tex'): we are removing `prepare' from the list of prerequisites. This bug was found by Zahra Sharbaf.
2020-02-19Building of GCC now only done when /dev/shm has more than 10GB freeMohammad Akhlaghi-4/+21
Until now, like all software on GNU/Linux systems GCC would be built in RAM (to speed up the build slightly and also not put too much stress on the HDDs/SSDs). But some systems don't have enough RAM for building GCC and will complain and crash. With this commit, we have added a check on the amount of free space in the `build_tmp' directory (which will be `/dev/shm' on GNU/Linux systems). If the amount of free space isn't more than 10GB, then GCC won't be built there and a temporary directory will be built under the `$(BDIR)/software' directory for it. This bug was found by Zahra Sharbaf. This fixes bug #57853.
2020-02-18README-hacking.md: corrected typo in project commandMohammad Akhlaghi-1/+1
I had forgot to add a `./' before the call to `project' for the `--check-config'.
2020-02-16Two values from the input dataset are now written into the paperMohammad Akhlaghi-3/+17
This was done just to get going with describing the analysis process.
2020-02-16Menke+2020 data is now imported and ready for later steps in plain textMohammad Akhlaghi-19/+84
The main problems with this dataset was the names of the journals (which sometimes have single quotes or apostrophes in them that is really annoying for SED)! But ultimately, for the simple study we want to do here, the journal names are irrelevant, so in the end I just ignored the names. Later we can set an identifier for the journals if necessary. But now we have the basic information in a way that is usable in a plot to show in this paper.
2020-02-16Building XLSX I/O and its dependencies: expat and minizipMohammad Akhlaghi-2/+40
Until now, there was no easy way to read/write `.xlsx' files (Microsoft Excel spreadsheets) within the template. But XLSX I/O provides to simple programs and some libraries to easily convert `.xlsx' files to CSV that can easily be read by any tool. This has also been implemented in the core template branch.
2020-02-16XLSX I/O installed with its two dependencies: expat and minizipMohammad Akhlaghi-0/+38
XLSX I/O is a very simple and fast program and library for reading and writing `.xls' and `.xlsx' files (mainly used by Microsoft Excel) to CSV files. It has two separate executables that can be called for an Excel file and will output a CSV plain text file that can then be used within the pipeline with more standard tools.
2020-02-15Edits in text, added Menke+2020 as a referenceMohammad Akhlaghi-56/+106
The text was slightly improved/edited and I also recently came up to the Menke et al. 2020 (DOI:10.1101/2020.01.15.908111) which also has some good datasets we can use as a demonstration here.
2020-02-13Corrected version of Texinfo when reportingMohammad Akhlaghi-1/+1
Until now we were mistakenly reporting the version of SED instead of Texinfo. With this commit, we corrected it! This was reported by Raul Infante Sainz.
2020-02-13Adding a link to the *crt*.o files in the local install directoryMohammad Akhlaghi-1/+17
Until now, we defined `LIBRARY_PATH' to fix the problem of the `ld' linker of Binutils needing several `*crt*.o' files to run. However, some software (for example ImageMagick) over-write `LIBRARY_PATH', therefore there is no other way than to put a link to these necessary files in our local build directory. With this commit, we fixed the problem by putting a link to the system's relevant files in the local library directory. This fixed the problem with ImageMagick. Later, when we build the GNU C Library in the project, we should remove this step. This bug reported by Raul Castellanos Sanchez.
2020-02-11Configure script won't crash without Fortran compiler, only a warningMohammad Akhlaghi-27/+46
Until now, when a Fortran compiler didn't exist on the host operating system, the configure script would crash with a warning. But some projects may not need Fortran, so this is just an extra/annoying crash! With this commit, it will still print the warning, but instead of a crash, it will just sleep for some seconds, then continue. Later, when if a software needs Fortran, it's building will crash, but atleast the user was warned. In the future, we should add a step to check on the necessary software and see if Fortran is necessary for the project or not. The project configuration should indeed crash if Fortran is necessary, but we should tell the user that software XXXX needs Fortran so we can't continue without a Fortran compiler. Also, a small sentence ("Project's configuration will continue in XXXX seconds.") was added after all the warnings that won't cause a crash, so user's don't think its a crash.
2020-02-11Using backup server when original download server failsMohammad Akhlaghi-7/+40
Until now, the main download script could only check one server for the given URL. However, ultimately the actual server that a file is downloaded from is irrelevant for this project: we actually check its checksum. Especially in the case of software (which are distributed over many servers), this can usually be very annoying: the servers may not properly communicate with the running system and even the 10 trials won't be enough. With this commit, the download script `reproduce/analysis/bash/download-multi-try' can take a new optional argument (a 5th argument). It assumes this argument is a space-separated list of server(s) to use as backup for the original URL. When downloading from the original URL fails, it will look into this list and try downloading the same file from each given server.
2020-02-07Edited parts of the textMohammad Akhlaghi-127/+177
While reading over the already written parts (and hopefully complete the paper), they were edited/corrected to be more clear.
2020-02-06Minor edits to various partsMohammad Akhlaghi-73/+136
Some edits were made after rereading of some parts.
2020-02-05Imported Raul's additions to README-hacking.md, no conflictsMohammad Akhlaghi-2/+16
There were no conflicts in this merge.
2020-02-01Better message for analysis when configuration wasn't completeMohammad Akhlaghi-4/+13
Until now, when `./project make' was run after an insuccessful run of `./project configure', it would just say to run `./project configure'. But for a first time user, this could be confusing because when the configuration is done in parallel, the error message can be very high on the command-line outputs and not seen clearly. With this commit, the error message is more complete and describes the problem and what the users should do in which circumstance.
2020-02-01Make called with -k during software buildingMohammad Akhlaghi-2/+2
Until now, Make was just run ordinarily on the two Makefiles of the software building phase. Therefore when there was a problem with one software while building in parallel, Make would only complete the running rules and stop afterwards. But when other rules don't depened on the crashed rule, its a waste of time to stop the whole thing. With this commit, both calls to Make in the `configure.sh' script are done with the `-k' option (or `--keep-going' in GNU Make). With this option, if a rule crashes, the other rules that don't depend on it will also be run. Generally, anything that doesn't depend on the crashed rule will be done. The `-k' option is a POSIX definition in Make, so it is present in most implemenetations (for the call to `basic.mk').
2020-02-01IMPORTANT: reproduce/software/bash renamed to reproduce/software/shellMohammad Akhlaghi-12/+12
Until now the shell scripts in the software building phase were in the `reproduce/software/bash' directory. But given our recent change to a POSIX-only start, the `configure.sh' shell script (which is the main component of this directory) is no longer written with Bash. With this commit, to fix that problem, that directory's name has been changed to `reproduce/software/shell'.
2020-01-31Configure step: compiler checks done before basic settingsMohammad Akhlaghi-421/+413
Until now, the project would first ask for the basic directories, then it would start testing the compiler. But that was problematic because the build directory can come from a previous setting (with `./project configure -e'). Also, it could confuse users to first ask for details, then suddently tell them that you don't have a working C library! We also need to store the CPATH variable in the `LOCAL.conf' because in some cases, the compiler won't work without it. With this commit, the compiler checking has been moved at the start of the configure script. Instead of putting the test program in the build directory, we now make a temporary hidden directory in the source directory and delete that directory as soon as the tests are done. In the process, I also noticed that the copyright year of the two hidden files weren't updated and corrected them.
2020-01-31Architecture-specific C headers on Debian-based OSs now accountedMohammad Akhlaghi-6/+15
Rencely the building of GCC was allowed on Debian-based systems that have their basic C library in architecture-specific directories, like `/usr/lib/x86_64-linux-gnu'. However, these systems also have their headers in non-standard locations, for example `/usr/include/x86_64-linux-gnu' and this caused a crash on a new Ubuntu system. /usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No such file or directory 27 | #include <bits/libc-header-start.h> | ^~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. The reason it didn't cause problems on other Ubuntus that we tested before was historic: In the old days, we would ask Ubuntu systems to install multilib features to have GCC. Because they had installed those features, this problem didn't show up! But this wasn't mandatory! With this commit, the `CPATH' environment variable is set (similar to how `LIBRARY_PATH' was set) and this fixed the problem on a clean Debian virtual machine. This bug was reported by Sebastian Luna Valero.
2020-01-27Moving basic configuration of Git section in README-hacking.mdRaul Infante-Sainz-13/+13
Until this commit, the small section of `README-hacking.md' in which it is explained how to do the first configuration of Git was at the beginning of the section `First custom commit'. However, it is better to have it just before the item `Your first commit' in that section. With this commit, this change has been done. Now the reader has the necessary steps for configuring Git just before it is needed for making the first commit.
2020-01-27Initial scripts compatible with Dash (minimalistic POSIX)Mohammad Akhlaghi-40/+52
Until now, the initial project scripts were primarily tested with GNU Bash. But Bash is not generally available on all systems (it has many features beyond POSIX). Because of this, effectively we were imposing the requirement on the user that they must have Bash installed. We recently started this with setting the shebang of `project' and `reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul and Gaspar reported an error on their systems. To fix the problem, I installed Dash (a minimalist POSIX-compliant shell) on my computer and temporarily set the shebangs to `/bin/dash', ran the project configuration step and fixed all issues that came up. With this commit, it can go all the way to building GCC on my system's Dash. After this stage (when `high-level.mk' is called), there is no problem, because we have our own version of GNU Bash and that installed version is used. Probably some more issues still remain and will hopefully be found in the future. While doing this, I also noticed the following two minor issues: - The `./project configure' option `--input-dir' was not recognized because it was mistakenly checking `--inputdir'. It has been corrected. - The test C programs now use the `<<EOF' method instead of `echo'. - In `basic.mk', the extra space between `syspath' and `:=' was removed (it was an ancient relic!).
2020-01-27Analysis phase description started with Final paper and verificationMohammad Akhlaghi-54/+157
With this commit, the general outline of the analysis phase is given, as well as a description of the LaTeX macros and their relation to the paper and thier verification. Also, the data-lineage figure was updated to have references.tex also and some resizing of the folders in file-architecture to be more clear.
2020-01-26General project structure and configuration describedMohammad Akhlaghi-344/+669
In the last few days I have been writing these two sections in the middle of other work. But I am making this commit because it has already become a lot! I am now going onto the description of `./project make'.
2020-01-23Hashbangs of project and configure.sh set to /bin/shMohammad Akhlaghi-2/+2
Until now, the hashbang of these two shell scripts was set to `/bin/bash', hence assuming that GNU Bash exists on the host system! But this is an extra requirement on the host operating system and these two scripts should be written such that they operate on a POSIX shell (the generic `/bin/sh' which can point to any shell program). With this commit this has been implemented! We may confront some errors as the system is run on other systems, but we should fix such errors and work hard to make these two scripts as POSIX-compatible as possible (runnable on any shell, so as not to force users to install Bash before running the project). This completes Task #15525.
2020-01-23IMPORTANT: Project preparation is now also done with project makeMohammad Akhlaghi-98/+37
Until now, the main commands to run the project were these: `./project configure' (to build the software), `./project prepare' (to possibly arrange input datasets and build special configuration Makefiles) and finally `./project make' to run the project. The main logic behind the "prepare" phase `top-prepare.mk' is to build configuration files that can be fed into the "make" step and optimize its operation. For example when the total number of necessary inputs for the majority of the analysis is not as large as the total number of inputs. With "prepare" (when necessary), you go through the raw inputs, select the ones that are necessary for the rest of the project. The output of `top-prepare.mk' is a configuration file (a Make variable) that keeps the IDs (numbers, names, etc). That configuration file would then be used in the `top-make.mk' to identify the lower level targets and allow optimal project organization and management. But the last two are both part of the analysis, and while they indeed need different calls to Make to be executed, many projects don't actually need a preparation phase: ultimately, its an implementation choice by the project developers and doesn't concern the project users (or the developers when they are running it). To avoid confusing the users, or simply annoying them when a projet doesn't need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk' Makefiles are called with the single `./project make' command and `./project prepare' has been dropped. I noticed this while writing the paper on this system.
2020-01-22Adding Raul as contributor of README-hacking.mdRaul Infante-Sainz-1/+2
Since I (Raul) did some changes (and I hope to do more :-)) in the `README-hacking.md', I am adding my information at the beginning of this file.
2020-01-22Adding basic configuration of Git in README-hacking.mdRaul Infante-Sainz-1/+14
Until this commit, we were asuming that Git was already properly configured. However, in order to be as complete as possible, it would be good if the basic commands to configure Git were in the `README-hacking.md'. With this commit, a small paragraph has been added in order to have the basic Git configuration commands (i.e. to configure the name, email, and favorite text editor).
2020-01-22Better explanation for missing static C libraryMohammad Akhlaghi-12/+22
Until now, the explanation for a missing static C library didn't actually guide the users to look above and see the error message! So with this commit, I edited it a little to be more clear (and mention to look above). Also, I noticed that on Amazon AWS systems, the static C library is installed as a separate package, so to help the users, I added the necessary command and some better explanation.
2020-01-22Perl is now a dependency of Coreutils, LD_LIBRARY_PATH in basic.mkMohammad Akhlaghi-67/+80
Until now Perl was built after Coreutils, but I recently noticed that Coreutils actually uses Perl while creating its manpages. So it is now built before Coreutils. Also, while testing on an Amazon AWS EC2 server, we noticed that Coreutils can't build its man page for `md5sum'. The problem was found to be due to the fact that until now, we weren't actually setting LD_LIBRARY_PATH to our installed library path in `basic.mk'. Therefore, it would crash because the server had an older version of OpenSSL than the one that the template's Coreutils was built with. In the meantime (while addressing the issues above, because we only had one thread on the AWS server) I also noticed a few programs that were using a summarize compilation command (that just prints `CC xxx.c' instead of the whole command) so I fixed them by adding `V=1'. This bug was found by Idafen Santana PĂ©rez.
2020-01-20New rendering of the file architecture plotMohammad Akhlaghi-193/+109
Until now the file architecture plot at the directories ontop of the top-level files. This made it hard to visually identify the top-level files. They are not placed ontop of the sub-directories and some space is added to highlight the files in the top-level directory and those in the subdirectories. Two other changes were made: - The symbolic links created in the top source directory are also shown. - The coding of this figure was made much more elegant by defining a PGFPlots node class and just changing the things that are direrent between each directory.
2020-01-20Added figure showing project's file structureMohammad Akhlaghi-92/+364
It was a little hard to describe the file structure so instead of using a standard listing as most papers do, I thought of showing the file and directory structure as boxes within each other (modeled on the Gnome disk-utility). Some other polishing was done throughout the paper also.
2020-01-20IMPORTANT!!! Configuration Makefiles now have a .conf suffixMohammad Akhlaghi-87/+85
Until now, the configuration Makefiles (in `reproduce/software/config/installation' and `reproduce/analysis/config') had a `.mk' suffix, similar to the workhorse Makefiles. Although they are indeed Makefiles, but given their nature (to only keep configuration parameters), it is confusing (especially to early users) for them to also have a `.mk' (similar to the analysis or software building Makefiles). To address this issue, with this commit, all the configuration Makefiles (in those directories) are now given a `.conf' suffix. This is also assumed for all the files that are loaded. The configuration (software building) and running of the template have been checked with this change from scratch, but please report any error that may not have been noticed. THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
2020-01-19GNU Make updated to version 4.3Mohammad Akhlaghi-3/+3
GNU Make 4.3 was just announced, so I have updated it here is well. This was important because until now the installable version was in alpha-mode (4.2.90), now its a stable version.
2020-01-19New --check-config option to ./project to check software build statusMohammad Akhlaghi-31/+56
Until now, it was necessry to run a long `while true' loop to see what is currently being built at configure time. So with this commit, a new `--checkconfig' option has been added to `./project' that can be called to run that loop and make it easier to check.
2020-01-19Corrected typo in last commit (forgetting \ at end of line)Mohammad Akhlaghi-1/+1
In the previous commmit, I had forgot to add a `\' after the newly added `sys_library_path' variable to the `high-level.mk' call.
2020-01-19LIBRARY_PATH is set accordingly based on the hostMohammad Akhlaghi-20/+58
Until now, GCC wouldn't build properly on Debian-based operating systems because `ld' needed to link with several necessary C library features like `crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to add the directory containing them to `LIBRARY_PATH'. In the previous commit, I actually searched for these files, but while testing on another system, I noticed that it can be problematic (other architectures may exist). With this commit, we are actually finding the build architecture of the running GCC (which is the same as the `ld') and using that to fix a fixed directory to `LIBRARY_PATH'.
2020-01-19Better search for static C library at start of configurationMohammad Akhlaghi-108/+75
Until now, to see if a working static C library and `sys/cdefs.h' exist, we were checking absolute locations like `/usr/include/sys/cdefs.h' or `/usr/lib/libc.a' and `/usr/lib64/libc.a'. But this is not robust because on different systems, they can be in different locations. With this commit, we actually use `find' to find the location of `libc.a' and use that to add elements to CPPFLAGS and LDFLAGS. This should fix the problem on systems that have them on non-standard locations.
2020-01-18TeXLive version printing updated to include revisionMohammad Akhlaghi-7/+26
Until now, when find the versions of the TeXLive packages, we would assume that `cat-date' is always present (because some packages don't have a version!). However, apparently an update has been made in the TeXLive Manager (`tlmgr') and `cat-date' is no longer present! As a result, none of the TeXLive packages were being printed. With this commit, it now assumes that `revision' is always present for every package, but it also attempts to read `cat-date' (for backwards compatability). When `cat-version' isn't present, it will try printing `revision' and if that is also not present, it will print the date.
2020-01-18Raw draft (until now as a separate repository) importedMohammad Akhlaghi-411/+3663
Until now, I was writing the paper without the template. But we will soon be adding a tutorial to the template, and I thought it will be good to have an example demonstration here too. So I just brought the hole project into the template structure, allowing us to add the template analysis later when its ready, and also allowing us to easily reproduce this paper ofcourse (without having to worry about the host's TeXLive installation.
2020-01-18First set of customizations doneMohammad Akhlaghi-352/+11
The unnecessary parts were removed and the project now runs.
2020-01-18Added mweights as a TeXLive packageMohammad Akhlaghi-1/+2
After a new rebuild of the project, I noticed that we now need to also build the `mweights' package.