| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Since I (Raul) did some changes (and I hope to do more :-)) in the
`README-hacking.md', I am adding my information at the beginning of this
file.
|
|
Until this commit, we were asuming that Git was already properly
configured. However, in order to be as complete as possible, it would
be good if the basic commands to configure Git were in the
`README-hacking.md'. With this commit, a small paragraph has been added
in order to have the basic Git configuration commands (i.e. to configure
the name, email, and favorite text editor).
|
|
Until now, the configuration Makefiles (in
`reproduce/software/config/installation' and `reproduce/analysis/config')
had a `.mk' suffix, similar to the workhorse Makefiles. Although they are
indeed Makefiles, but given their nature (to only keep configuration
parameters), it is confusing (especially to early users) for them to also
have a `.mk' (similar to the analysis or software building Makefiles).
To address this issue, with this commit, all the configuration Makefiles
(in those directories) are now given a `.conf' suffix. This is also assumed
for all the files that are loaded.
The configuration (software building) and running of the template have been
checked with this change from scratch, but please report any error that may
not have been noticed.
THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS
FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE
SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES
ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
|
|
GNU Make 4.3 was just announced, so I have updated it here is well. This
was important because until now the installable version was in alpha-mode
(4.2.90), now its a stable version.
|
|
Until now, it was necessry to run a long `while true' loop to see what is
currently being built at configure time. So with this commit, a new
`--checkconfig' option has been added to `./project' that can be called to
run that loop and make it easier to check.
|
|
In the previous commmit, I had forgot to add a `\' after the newly added
`sys_library_path' variable to the `high-level.mk' call.
|
|
Until now, GCC wouldn't build properly on Debian-based operating systems
because `ld' needed to link with several necessary C library features like
`crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to
add the directory containing them to `LIBRARY_PATH'. In the previous
commit, I actually searched for these files, but while testing on another
system, I noticed that it can be problematic (other architectures may
exist).
With this commit, we are actually finding the build architecture of the
running GCC (which is the same as the `ld') and using that to fix a fixed
directory to `LIBRARY_PATH'.
|
|
Until now, to see if a working static C library and `sys/cdefs.h' exist, we
were checking absolute locations like `/usr/include/sys/cdefs.h' or
`/usr/lib/libc.a' and `/usr/lib64/libc.a'. But this is not robust because
on different systems, they can be in different locations.
With this commit, we actually use `find' to find the location of `libc.a'
and use that to add elements to CPPFLAGS and LDFLAGS. This should fix the
problem on systems that have them on non-standard locations.
|
|
Until now, when find the versions of the TeXLive packages, we would assume
that `cat-date' is always present (because some packages don't have a
version!). However, apparently an update has been made in the TeXLive
Manager (`tlmgr') and `cat-date' is no longer present! As a result, none of
the TeXLive packages were being printed.
With this commit, it now assumes that `revision' is always present for
every package, but it also attempts to read `cat-date' (for backwards
compatability). When `cat-version' isn't present, it will try printing
`revision' and if that is also not present, it will print the date.
|
|
After a new rebuild of the project, I noticed that we now need to also
build the `mweights' package.
|
|
The checklist descriptions were slightly edited to be more clear. Also,
while following them, I noticed that while removing the "delete-me" parts
on `verify.mk', would cause an error: the `if [ $$m == delete-me ];'
statement we were saying to delete cause an error because `elif' was the
first statement Bash would see. So with this commit, the `download'
conditional (which isn't instructed to be deleted) was set to be the top
(with an `if') and the `delete-me' conditional now has an `elif'.
|
|
Until now, the small one-line script that lists programs was introduced in
the checklist after running `./project configure'. But people would mostly
miss it because they would wait until the configuration is complete.
With this commit, that point has been put above the `./project configure'
step. Readers are instructed to open a new terminal and run that script,
then go to the next step so they see the directories get filled
actively. It will also help them understand what is going on.
|
|
Until now the actual journal webpage was used for Raul's paper. However,
the journal webpage needs authorized access for people to read it,
therefore its will be inaccessible for many people. A better and more well
known place for the paper (atleast in astronomy) is the ADS link. In the
ADS link, if someone has access to the journal, they will get the journal's
version and if not, they will get the arXiv version. It also has a common
BibTeX export tool for all journals. We had also done this for the other
papers in that list.
With this commit, I thus changed the URL for the paper, and also removed
the "issue" number (4 in this case), since that is mostly irrelevant, only
the volume and page numbers are usually used for the other papers too.
|
|
The "SDSS extended PSFs" paper was already included as an example of
papers wich uses this template. However, the reference was the arXiv
one. With this commit, since the paper has been finally published, it
has been added the final reference to the journal.
|
|
Until this commit, the checking of X11 installation done to ensure that
it is already available in the host system was crashing in macOS
systems. The reason is that the place of the X11 libraries use to be
`/opt/X11/lib' in macOS systems. With this commit, this issue has been
fixed by adding this directory to the LDFLAGS.
|
|
Newer versions of Astropy package has been released. With this commit,
it has been updated. It has been increased from v3.2.1 to v4.0
|
|
In markdown, to specify that part of the text should be shown as code we
need to put it between two `. But in `reproduce/software/make/README.md',
the code parts started with ` and finished in ' (like the file name above).
With this commit, this has been corrected and it will display properly on
web browsers.
|
|
Until now, if the file to be verified didn't exist, a different checksum
would be generated, and it would stop, but it wasn't immediately clear if
the differing checksum is because the file doesn't exist at all!
With this commit, before calculating the checksum, we first make sure if
the file exists. If it doesn't exist an explicit error is printed and thus
will help the project editor to find the cause of the problem.
|
|
In the previous commit, I had forgot to update a small part in the
checklist (when modifying `top-make.mk') which is now corrected.
I also added a few sentences in the description of how to customize the
verification to make it easier to understand.
|
|
Until now, the only verification that the template provided was the
published PDF. Users had to manually compare the published and generated
PDFs (numbers, plots, tables) and see if they obtained the same
result. However, this type of manual verification is not good and is prone
to frustration and missing important differences.
With this commit, a new Makefile has been added in the analysis steps:
`verify.mk'. It provides facilities to easily verify the results that go
into the paper. For example tables that go into making the paper's plots,
or the LaTeX macros that blend into the text. See the updated parts in
`README-hacking.md` for a more complete explanation.
This completes task #15497.
|
|
In the previous commit, we added the files to ignore from the template
branch, but only the files that had been deleted. With this commit,
`paper.tex' is also added to the files that must be ignored from the
template branch (the file remains in the project, but in the template
branch, its contents are just dummy place-holders).
|
|
During the checklist we guide the user to delete the dummy `delete-me*'
files from their custom branch. Later, if the dummy files are updated in
the template's master branch, if the user merges with the template branch,
these files will be written back into their project! This is very annoying!
With this commit, a step was added in the `README-hacking.md' checklist,
just after deleting the dummy files to avoid this problem using the
`.gitattributes' file, telling Git to keep the changes as implemented in
the merging branch (`ours').
|
|
Now that its 2020, its necessary to include this year in the copyright
statements.
|
|
Until this commit, the number `2' was missing in the checksum variable
name of that library. It was `libxml-checksum' but it should be
`libxml2-checksum'. With this commit, this issue has been fixed.
|
|
An extra backslash in the prerequisites of Jeepney Python package was
causing a crash in the installation of that software. With this commit,
this problem has been fixed by removing the backslash.
|
|
OpenMP takes a LONG TIME to build, so to keep things reproducible we are
explicitly disabling OpenMP, if a user needs OpenMP, its trivial to just
add it as a prerequisite of R. The problem is that in some scenarios (based
on other dependencies and when they were built in the build directory),
OpenMP may be present when R is being installed and in other it may not. We
don't want the result to be different between the two builds.
|
|
With this commit, we now have the core R interpretter within the
template. We should later include instructions to install R packages
(possibly in a separate top-level Makefile like Python).
|
|
Recent builds of the template need these three packages to build the PDF.
This was reported by Hamed Altafi.
|
|
Until now we weren't including this package, but Alberto Madrigal recently
reported that his build of the template failed because it needed it.
With this commit, it is now being installed with the template.
|
|
Until now, while building Perl, the `-Dlddlflags="-shared $$LDFLAGS"'
option was being used for all systems. However, this caused `symbol(s) not
found for architecture x86_64' problems on macOS systems.
With this commit, this configure option is only passed for GNU/Linux
systems. With it, Perl can be built nicely on both GNU/Linux and macOS.
|
|
Newer versions of these packages have recently come out with major
improvements, so they have been updated in the template.
|
|
Until now, Ghostscript was using some host system's X11 libraries during
its build (and later at run-time). We should ideally install all these
necessary libraries within the template (Task #15481). But right now we are
too busy.
As a temporary work-around we try building a small dummy program that links
with some of those libraries before attempting to built Ghostscript. If it
fails, then a notice is printed with the cause and explaining a temporary
solution is suggested: how to install those libraries on the system when
you have root access.
|
|
Until now, when an input dataset already exists in `INDIR', the template
would just make a symbolic link to it in the build directory. However, in
many cases, the files in INDIR will actually be links to other locations on
the filesystem and some programs have problems following too many links.
With this commit, the template is now using the `readlink' program (part of
GNU Coreutils) to follow a possible link and point the link in the build
directory directly to an actual non-link file.
|
|
Until now, after removing all environment variables, we were just giving
Make the top Makefile to execute. By default, for every target, Make will
try many built-in rules (which is good when compiling programs, but
redundant in other cases). All these checkings also populate the debugging
output of Make (with `-d'). So its easier and slightly faster to just tell
Make to ignore builtin rules and variables.
With this commit, to address this issue, the `project' script runs
`.local/bin/make' with `--no-builtin-rules' and `--no-builtin-variables'.
|
|
Until now when building Matplotlib on macOS systems, we were using the
default C compiler. However, while Yahya Sefidbakht (previously) and
Mahdieh Nabavi (now) were trying to build the template, on their macOS
using the GNU Compiler Collection (GCC), we found that Matplotlib needs
special macOS headers that GCC doesn't recognize.
With this commit, when Matplotlib is being built on macOS systems, it uses
`clang' and this fixed the problem (so far checked on Mahdieh's machine).
|
|
While working on a different branch to build the GNU C Library, I noticed a
few places in the template that need corrections which are now applied:
1. A new-line character after the "C compiler works" notice at the start
of the configure script.
2. Removing possible `::' in the `LD_LIBRARY_PATH' definition of
`basic.mk'. Note that its not necessary in the other steps because we
don't use any outside-defined `LD_LIBRARY_PATH'.
3. Building GMP for C++ and also with `--enable-fat'.
4. Removing the unpacked Perl tarball directory after its installation.
|
|
These two packages are necessary to build the GNU C Library.
|
|
Perl is necessary to build Texinfo and later to build LaTeX. Until now we
were just using the host operating system's installation of Perl, but in
some instances that Perl can be too old and not suppor the features
necessary. With this commit, Perl is now built from source during the basic
installation step of the template.
This was reported by Idafen Santana Pérez, after trying the pipeline on an
Amazon AWS EC2 system (a Linux distro by Amazon for its cloud services).
|
|
Raul's paper (that uses this template) was just published on arXiv today
(congratulations Raul!). So it has been added to the list of papers using
this template.
|
|
Since we got the RDA Adoption grant, it was necessary to add it in the
acknowledgements.
|
|
When building the log4cxx tarball from its Git history, I noticed that
files with very long names are not packaged by tar (because by default
Automake uses the ancient v7 tar format that only supports file names less
than 99 characters).
So I build the tarball with the `tar-ustar' option to Automake (by
modifying the log4cxx source) and the resulting tarball was able to compile
and run successfully. This has been described above the rule to build
log4cxx and I also sent an email to their developing mailing list to inform
them of this problem. If they address it, I will remove the note on the
necessary corrections.
|
|
Some minor corrections were made in the template:
- When making the distribution, `.swp' files (created by Vim) are also
removed.
- Autoconf is set as a prerequisite of Automake
I was also trying to add the Apache log4cxx, but its default 0.10.0 tarball
needs some patches, so I have just left it half done until someone actually
needs it and we apply the patch.
|
|
Since adding this new step, I had forgot to mention it in the checklist of
`README-hacking.md'. It is added with this commit.
|
|
Until now, the tarballs were the first normal prerequisite of the
software. As a result if their date changed, the whole software would be
re-built. However, for tarballs specifically, we actually check their
contents with a checksum, so their date is irrelevant (if its newer than
the built program, but has the same checksum, there is no need to re-build
the software).
Also, calling the tarball name as an argument to the building process (for
example `gbuild') was redundant. It is now automatically found from the
list of order-only prerequisites within `gbuild' and `cbuild' (similar to
how it was previously found in the `pybuild' for Python building).
A `README.md' file has also been placed in `reproduce/software/make' to
help describe the shared properties of the software building
Makefiles. This will hopefully grow much larger in the future.
|
|
Python's `lmfit' module and all its major dependencies (`asteval',
`corner', `emcee' and `uncertainties') have been included in the template.
While doing this I noticed that if the tarballs are the last prerequisite
of each software building rule, then when building in parallel, the
template will immediately start building packages as soon as the first one
is downloaded. Not like the current way that it will attempt to download
several, then start building. For now, this has been implemented in the
Python build rules for all the modules and we'll later do the same for the
other programs and libraries. This also motivated a simplification of the
`pybuild' function: it now internally looks into the prerequisites and
selects the tarball from the prerequisite that is in the tarballs
directory.
This isn't a problem for the build, but I just don't understand why Python
can't recognize the version of `emcee', Python reads the version of `emcee'
as `0.0.0'! But it doesn't cause any crash in the build, so for now its
fine.
|
|
The part on using shared memory was edited for a few things that weren't
clear.
|
|
Some typos were fixed.
|
|
The edits help it be more clear, and remind the reader to delete any
remaining file in the RAM in the end.
|
|
When you are working with large files and there is some good RAM in the
system (large/powerful computers), it is beneficial to work in the shared
memory directory and not the actual persistent storage (like HDD or
SSD).
With this commit, a fully working demo has been added to
`README-hacking.md' (under the tips of "Make programming") to show how to
effectively work in situations like this.
|
|
The tarball of HEALPix includes multiple languages and doesn't include the
ready-to-run GNU Build System by default, we actually have to build the
`./configure' script for the C/C++ libraries. So it was necessary to also
include GNU Autoconf and GNU Automake as prerequisites of HEALPix.
However, the official GNU Autoconf tarball (dating from 2012) doesn't build
on modern systems, so I just cloned it from its source and bootstrapped it
and built its modern tarball which we are using here.
|
