| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Summary: this commit will not affect your analysis: only the version of GCC
has been updated for portability and jq (a JSON parser) has been added.
Until now, several portability issues existed in Maneage, causing problems
mostl on macOS, but some also some on GNU/Linux. They are listed below
along with the fix.
With this commit, the problems mentioned above have been fixed:
- 'pkg-config' on macOS needed some manual steps for a successful
installation. The cause was its improper checks on the macOS "carbon"
library. So we now manually fix this check within the code to fix the
problem. This was found and fixed by Martin Wiesmann.
- Apple chip variants like M1 Pro, M1 Max, M1 Ultra, M2 Pro, etc could not
be properly recognized as a macOS system since we expected a single
word. This was solved by using pattern matching instead. Found and fixed
by Martin Wiesmann.
- macOS's sha512sum wraps the 128-character hex hash across two output
lines (80 + 48 chars) and the command that we used to read the output
would only captured the first 80 characters, so checksums never
matched. The solution was to merge the lines before the check. Found and
fixed by Martin Wiesmann.
- When a checksum failed, the script exited with an error but left the bad
'.unchecked' file on disk. On the next run, the download script saw the
file already existed and skipped downloading entirely. As a result, the
corrupt file was passed to the checksum check again, failing
forever. The fix was to remove the file when we confirmed a bad
download. Found and fixed by Martin Wiesmann.
- When the server of a file returns a HTML output instead of the file we
want, the download script could not detect and would assume that the
download was successful. Therefore, it would always crash with a
checksum error. The fix was to check the first few bytes of the output
and consider it failed if we detect '<html' in them. Found and fixed by
Martin Wiesmann.
- Unzip could not be built due to linking to old C library constructs. The
fix was to edit that part of the code before starting the build. Found
and fixed by Faezeh Bidjarchian.
- top-prepare.mk: in a few cases, we had incorrectly written './project
prepare' (which does not exist!); by Mohammad Akhlaghi.
- Tar's 'acl' features were causing crashes with GCC 16.1.1 on an Arch
GNU/Linux. Since this feature is not relevant in Maneage, the fix was to
disable it; by Mohammad Akhlaghi.
- To disable CFITSIO's fortran library (which could be problematic in
macOS), we were using the wrong command! Found and fixed by Martin
Wiesmann.
- GCC 15.2.0 (the previous version on the 'maneage' branch) needed more
patches to build with deprecated Linux headers (that are no longer
present in Arch GNU/Linux, and will be removed from other distros in due
time). The fix was to upgrade GCC to the latest version of 16.1.0 by
Mohammad Akhlaghi.
- Zenodo is no longer checked at the start of the Maneage
configuration. This is because from early 2025, Zenodo does not allow
having so many files in a single project, as described in this task:
https://savannah.nongnu.org/task/?16621. Therefore many of the new
software (since then) are no longer on Zenodo. Implemented by Mohammad
Akhlaghi.
- initialize.mk: when there was no 'maneage' branch in the project, a
warning was printed when writing the LaTeX macros. However, this is only
relevant when compiling the PDF. So when the user hasn't activated the
PDF creation variable, this warning was not relevant and was just
annoying. The fix was to put a condition and only print it when it can
be useful by Mohammad Akhlaghi.
- README-hacking.md:
- There was no checklist for maintainers; by Mohammad Akhlaghi
- Incorrect arXiv URL to Eskandarlou+2026; found by Boud Roukema.
Furthermore, the jq program has been added to Maneage with this commit; it
is useful when it is necessary to parse/edit JSON files.
|
|
Summary: this commit has no effect on your project's analysis steps, just
check the list of changed Python software versions (if you used them).
Until now, the Python packages of Maneage had not been maintained for some
time, there were several portability issues: updates to the GNU C Library
needed an update to some of the basic software, and the latest macOS
features caused some crashes.
With this commit, along with the software version updates of [1], the
removed software of [2] and newly added software of [3], the following
issues in Maneage have been addressed:
- README-hacking:
- Published paper list: Added links to two published papers.
- Publication checklist: added tips on how to squash the full LaTeX
source of a Maneage'd projet (which includes many LaTeX files) into a
single LaTeX file for some journals.
- A new "Maneage Development Workflow" section has been added; see it
for details.
- paper.mk: empty macro files are not included in the built 'project.tex'
since they have no effect and because they can cause problems with
uploading to arXiv (which removes empty files).
- verify.mk: when verification is activated:
- When a macro file does not exist (can happen by mistake), an error is
printed to remind the author.
- Better dealing with empty macro files (no need to add their hash).
- versions.conf: 26 Python packages that cannot currently be built with
the new Python build system of Maneage have been separated and commented
to cause an early crash if they are required; and encourage volunteers
to help update them.
This commit is a git "squash" of contributions from the following people
- Boud Roukema: updated/added Python packages and fixed many portability
issues.
- Faezeh Bidjarchian: fixed some portability issues on macOS.
- Giacomo Lorenzetti: tested and fixed issues in newly added software.
- Khalid Masum: fixed issue with order of LD_LIBRARY_PATH in 'basic.mk'.
- Mohammad Akhlaghi: fixed basic software issues and reviewed edits for
the merge.
- Raul Infante-Sainz: fixed portability issues on macOS.
- Sepideh Eskandarlou: helped in finding the verification fixes.
[1] Updated software
Name Group Version (old -> new)
---- ----- --------------------
GNU Coreutils basic 9.9 -> 9.11
GNU Gettext basic 0.26 -> 1.0
GNU libunistring basic 1.4.1 -> 1.4.2
GNU M4 basic 1.4.20 -> 1.4.21
GNU Nano basic 8.7 -> 9.0
Ninja high-level 1.12.1 -> 1.13.2
OpenMPI high-level 4.1.1 -> 5.0.10
SWIG high-level 4.0.2 -> 4.3.1
Zip high-level 3.0 -> 3.0-15
Astropy python 7.0.1 -> 7.1.0
mpi4py python 3.1.3 -> 4.0.3
setuptools python 75.8.0 -> 80.8.0
[2] Removed software: 'podlators' has been removed from basic software
because the necessary scripts that it provided are included in Perl's
default installation.
[3] New software:
Name Group Version
---- ----- -------
framel high-level 8.47.3
GNU Parallel high-level 20260322
Healpix high-level 3.83-2024Nov13
Metaio high-level 8.5.0-a5da19d
PCRE2 high-level 10.45
SQLite high-level 3.53.0
TCL high-level 9.1a1
TK high-level 9.1a1
tzdb high-level 2026a
lalsuite high-level v7.26-15-g1e4c5e5080
astropy-healpix python 1.1.2
colorama python 0.4.6
cosmodesi-mpytools python 1.0.0-04369fc9
exceptiongroup python 1.3.0
filt-scm python 1.7.0
hatchling python 1.27.0
hatch-vcs python 0.5.0
healpy python 1.18.1
joblib python 1.5.0
igwn-ligolw python 2.1.0
igwn-segments python 2.1.0
iniconfig python 2.1.0
ligo-skymap python 2.4.0
lscsoft-glue python 4.1.0
networkx python 3.5
pandas python 2.2.3
pathspec python 0.12.1
pluggy python 1.6.0
pyfftw python 0.15.1
pygments python 2.19.2
pytest python 8.4.1
python3-fitsio python 1.3.0
pytz python 2025.2
scikit-base python 0.12.2
scikit-learn python 1.6.1
threadpoolctl python 3.6.0
tqdm python 4.67.1
trove-classifiers python 2025.5.9.12
versioneer python 0.29
|
|
Summary: this commit will not affect your project's software or analysis.
Until now, the ending copyright years of all files in Maneage were
2025. But we have already entered 2026 so it is important to update them.
With this commit, the ending copyright year of all files is changed to
2026.
|
|
Summary: The Apptainer script's option names have been renamed: '--sif' and
'--base-sif' are new names for the old '--project-name' and '--base-name'.
Until now the options that took SIF files in the Apptainer script were not
too clear (in their name): ending in '-name' (not clearly mentioning that
it needs a file name). Also, in the same script, we had forgot to add an
option to allow the user to choose a different base operating system!
With this commit, both the issues above have been fixed: the options end in
'-sif' (clearly informing the user that a SIF file is expected) and we now
have a new '--base-os' option. In the process, a few other parts of Maneage
were also polished:
- README-hacking.md: the newly published J-PLUS PSF paper was added to
the list of Maneage'd papers and the final journal information for the
NASIM paper has been added. Some typos here and there were also fixed.
- The curretnly used VIM needed the the '-std=gnu17': necessary now that
the default Maneage GCC 15.2.0 for the high-level software which
defaults to C23.
|
|
Summary: this commit involves a major version update of many basic software
and Gnuastro's major dependencies. So it may affect your configuration and
analysis.
Until now, the most recent software updates weren't properly tested on
MacOS, causing the software configuration step to fail. In particular, the
'prep-source.sh' (that corrects any hard-coded '/bin/sh' within the
unpacked tarballs of all software) as well as some issues in the 'tar',
'util-linux' and 'ghostscript' programs. These portability issues were
found and fixed by Raul Infante-Sainz and Giacomo Lorenzetti.
With this commit, those portability issues have been addressed while
updating the following software. Also see notes below the list.
High-level software:
CFITSIO 4.5.0 --> 4.6.3
Ghostscript 10.04.0 --> 10.06.0
Gnuastro 0.23 --> 0.24
Util-linux 2.40.4 --> 2.41.3
WCSLIB 8.4 --> 8.5
Basic software:
cURL 8.11.1 --> 8.17.0
GNU Bash 5.2.37 --> 5.3.9
GNU Binutils 2.43.1 --> 2.45.1
GNU Coreutils 9.6 --> 9.9
GNU Diffutils 3.10 --> 3.12
GNU AWK 5.3.1 --> 5.3.2
GNU GCC 14.2.0 --> 15.2.0
GNU Gettext 0.23.1 --> 0.26
GNU Grep 3.11 --> 3.12
GNU Gzip 1.13 --> 1.14
GNU Libunistring 1.3 --> 1.4.1
GNU M4 1.4.19 --> 1.4.20
GNU MPFR 4.2.1 --> 4.2.2
GNU Nano 8.3 --> 8.7
GNU Readline 8.2.13 --> 8.3.3
Git 2.48.1 --> 2.52.0
Less 668 --> 685
Libxml2 2.13.1 --> 2.15.1
OpenSSL 3.4.0 --> 3.6.0
Perl 5.40.1 --> 5.42.0
The latest versions of Bash and M4 were prepared for the new default C
standard of GCC 15.*, so the '-std=gnu17' was removed for them. Also, while
a new version of Dash was available, it would cause an issue when building
in Apptainer, so it has been described in 'versions.conf' to be aware in
the future updates.
Libgit2 has been removed as a dependency of Gnuastro. The reason is fully
described in the comment on top of Gnuastro's build rule in
'high-level.mk'.
|
|
Summary: after merging this commit into your project, it should be
re-configured since the location of software installation files like
'LOCAL.conf' or the LaTeX macros of the software environment have
changed. But it should not affect the analysis phase of your project.
Until this commit, it was not possible to run a pre-built Maneage'd project
(in a container) on a newly cloned Maneage'd project source. This was
because the containers should be read-only, but during the various checks
of the configuration (to verify that we are using the same software
environment in the container and the source), we were writing/testing many
things in the build directory, and 'LOCAL.conf' which was actually in the
source directory!
Furthermore, the '.local' and '.build' were built at configure time, making
it hard to run the same container from a newly cloned Maneage'd project. To
make things harder for the scenario above, the 'configure.sh' script would
pause on every message and didn't have a quiet mode (making it practically
impossible to run './project configure' before './project make' on every
container run).
With this commit, all these issues have been addressed and it is now
possible to simply get a built container, clone a Maneage'd project and run
the analysis (using the built environment of the container that is verified
on every run). The respective changes/additions are described below:
- The high-level container scripts ('apptainer.sh' and 'docker.sh', along
with their READMEs) have been moved to the 'reproduce/software/shell'
directory and the old 'reproduce/software/containers' directory has been
deleted. This is because we have classified the software files by their
language/format and the container scripts are scripts in the end.
- The './project' script:
- Now has two extra options: '--quiet' and '--no-pause'. Both are
directly passed to the 'configure.sh' script. They will respectively
disable any informative printed message or any pause after that
message (if it is printed).
- The '--build-dir' option is now also relevant for './project make':
when it is given, it will re-create the two '.build' and '.local'
symbolic links at the top source directory in all scenarios
('configure', 'make' or 'shell'). This will allow both the
configuration, analysis and shell phases to safely assume they exist
and match the user's desire at run-time.
- The build/analysis directory's sub-directories that need to be built
before 'top-make.mk' are now built in a separate function to help in
readability.
- The 'configure.sh' script:
- For developers: a new 'check_elapsed' variable has been defined that
will enable the newly added 'elapsed_time_from_prev_step'
function. This function should be used from now on at the end of
every major step to help find bottlenecks.
- The targets of the software in 'pre-make-build.sh' now also have the
version of the software in their file name. Until now, they didn't have
the version, so there was no way to detect if the software has been
updated or not in the source. For Lzip and Make (that also get built
after GCC), the ones in this script have a '-pre-make' suffix also.
- 'Local.conf.in' now has descriptions for every variable.
- The '-std=gnu17' option is now used instead of '-std=c17' for basic
software that cannot be built without specifying the C standard in GCC
15.1 (described in previous commit: 2881fc0a6205). See [1] for more
details; in summary: '-std=gnu17' is also supported on macOS's Clang and
has some features that 'pkg-config' needs
- Generally: some longer code lines have been broken or indentation
decreased to fit the 75 character line length. This has not reduced
readability however. For example the long 'echo' commands are now
replaced by multiple 'printf's, or the indentation is still clearly
visible.
The seeds of the work on this commit started by a branch containing three
commits by Giacomo Lorenzetti (133 insertions, 100 deletions). Upon merging
with the main 'maneage' branch, they were generalized and re-organized to
become this commit.
The following issues have also been addressed with this commit:
- The LaTeX calls (during the building of 'paper.pdf') do not contain
Maneage'd dynamic libraries. This is because we don't build the LaTeX
binaries from source, an TeXLive manager uses the host environment.
- The 'docker.sh' script:
- Adds the '--project-name' option: its internal variable existed, but
the option for the user to define it at run-time was not.
- Ported to macOS: it does not check being a member of the 'docker'
group, and finds the number of threads using macOS-specific tools.
- The 'apptainer.sh' script:
- Now installs 'wget' in the base container also (necessary when the
user doesn't have the tarballs).
[1] https://savannah.nongnu.org/bugs/?67068#comment2
|
