<feed xmlns='http://www.w3.org/2005/Atom'>
<title>project.git/reproduce/software/bash, branch maneage</title>
<subtitle>Core Maneage branch (where all projects derive from)</subtitle>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/'/>
<entry>
<title>IMPORTANT: reproduce/software/bash renamed to reproduce/software/shell</title>
<updated>2020-02-01T20:24:00+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-02-01T20:24:00+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=35ed6cf0df743175688b49a4559793cb7f6e9d66'/>
<id>35ed6cf0df743175688b49a4559793cb7f6e9d66</id>
<content type='text'>
Until now the shell scripts in the software building phase were in the
`reproduce/software/bash' directory. But given our recent change to a
POSIX-only start, the `configure.sh' shell script (which is the main
component of this directory) is no longer written with Bash.

With this commit, to fix that problem, that directory's name has been
changed to `reproduce/software/shell'.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now the shell scripts in the software building phase were in the
`reproduce/software/bash' directory. But given our recent change to a
POSIX-only start, the `configure.sh' shell script (which is the main
component of this directory) is no longer written with Bash.

With this commit, to fix that problem, that directory's name has been
changed to `reproduce/software/shell'.
</pre>
</div>
</content>
</entry>
<entry>
<title>Configure step: compiler checks done before basic settings</title>
<updated>2020-01-31T21:43:11+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-31T21:43:11+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=334af0e29a950873a19f958b0e47d847808e8910'/>
<id>334af0e29a950873a19f958b0e47d847808e8910</id>
<content type='text'>
Until now, the project would first ask for the basic directories, then it
would start testing the compiler. But that was problematic because the
build directory can come from a previous setting (with `./project configure
-e'). Also, it could confuse users to first ask for details, then suddently
tell them that you don't have a working C library! We also need to store
the CPATH variable in the `LOCAL.conf' because in some cases, the compiler
won't work without it.

With this commit, the compiler checking has been moved at the start of the
configure script. Instead of putting the test program in the build
directory, we now make a temporary hidden directory in the source directory
and delete that directory as soon as the tests are done.

In the process, I also noticed that the copyright year of the two hidden
files weren't updated and corrected them.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the project would first ask for the basic directories, then it
would start testing the compiler. But that was problematic because the
build directory can come from a previous setting (with `./project configure
-e'). Also, it could confuse users to first ask for details, then suddently
tell them that you don't have a working C library! We also need to store
the CPATH variable in the `LOCAL.conf' because in some cases, the compiler
won't work without it.

With this commit, the compiler checking has been moved at the start of the
configure script. Instead of putting the test program in the build
directory, we now make a temporary hidden directory in the source directory
and delete that directory as soon as the tests are done.

In the process, I also noticed that the copyright year of the two hidden
files weren't updated and corrected them.
</pre>
</div>
</content>
</entry>
<entry>
<title>Architecture-specific C headers on Debian-based OSs now accounted</title>
<updated>2020-01-31T19:29:02+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-31T19:29:02+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=50832921ab5eac62350855955d3af3f6f0766bbf'/>
<id>50832921ab5eac62350855955d3af3f6f0766bbf</id>
<content type='text'>
Rencely the building of GCC was allowed on Debian-based systems that have
their basic C library in architecture-specific directories, like
`/usr/lib/x86_64-linux-gnu'. However, these systems also have their headers
in non-standard locations, for example `/usr/include/x86_64-linux-gnu' and
this caused a crash on a new Ubuntu system.

/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No
such file or directory
   27 | #include &lt;bits/libc-header-start.h&gt;
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

The reason it didn't cause problems on other Ubuntus that we tested before
was historic: In the old days, we would ask Ubuntu systems to install
multilib features to have GCC. Because they had installed those features,
this problem didn't show up! But this wasn't mandatory!

With this commit, the `CPATH' environment variable is set (similar to how
`LIBRARY_PATH' was set) and this fixed the problem on a clean Debian
virtual machine.

This bug was reported by Sebastian Luna Valero.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Rencely the building of GCC was allowed on Debian-based systems that have
their basic C library in architecture-specific directories, like
`/usr/lib/x86_64-linux-gnu'. However, these systems also have their headers
in non-standard locations, for example `/usr/include/x86_64-linux-gnu' and
this caused a crash on a new Ubuntu system.

/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No
such file or directory
   27 | #include &lt;bits/libc-header-start.h&gt;
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

The reason it didn't cause problems on other Ubuntus that we tested before
was historic: In the old days, we would ask Ubuntu systems to install
multilib features to have GCC. Because they had installed those features,
this problem didn't show up! But this wasn't mandatory!

With this commit, the `CPATH' environment variable is set (similar to how
`LIBRARY_PATH' was set) and this fixed the problem on a clean Debian
virtual machine.

This bug was reported by Sebastian Luna Valero.
</pre>
</div>
</content>
</entry>
<entry>
<title>Initial scripts compatible with Dash (minimalistic POSIX)</title>
<updated>2020-01-27T15:28:50+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-27T15:15:40+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=f37005b729065f0e4ff6bfa99e5410ebb210cd60'/>
<id>f37005b729065f0e4ff6bfa99e5410ebb210cd60</id>
<content type='text'>
Until now, the initial project scripts were primarily tested with GNU
Bash. But Bash is not generally available on all systems (it has many
features beyond POSIX). Because of this, effectively we were imposing the
requirement on the user that they must have Bash installed. We recently
started this with setting the shebang of `project' and
`reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul
and Gaspar reported an error on their systems.

To fix the problem, I installed Dash (a minimalist POSIX-compliant shell)
on my computer and temporarily set the shebangs to `/bin/dash', ran the
project configuration step and fixed all issues that came up. With this
commit, it can go all the way to building GCC on my system's Dash. After
this stage (when `high-level.mk' is called), there is no problem, because
we have our own version of GNU Bash and that installed version is used.
Probably some more issues still remain and will hopefully be found in the
future.

While doing this, I also noticed the following two minor issues:

 - The `./project configure' option `--input-dir' was not recognized
   because it was mistakenly checking `--inputdir'. It has been corrected.

 - The test C programs now use the `&lt;&lt;EOF' method instead of `echo'.

 - In `basic.mk', the extra space between `syspath' and `:=' was removed
   (it was an ancient relic!).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the initial project scripts were primarily tested with GNU
Bash. But Bash is not generally available on all systems (it has many
features beyond POSIX). Because of this, effectively we were imposing the
requirement on the user that they must have Bash installed. We recently
started this with setting the shebang of `project' and
`reproduce/software/bash/configure.sh' to `/bin/sh'. After doing so, Raul
and Gaspar reported an error on their systems.

To fix the problem, I installed Dash (a minimalist POSIX-compliant shell)
on my computer and temporarily set the shebangs to `/bin/dash', ran the
project configuration step and fixed all issues that came up. With this
commit, it can go all the way to building GCC on my system's Dash. After
this stage (when `high-level.mk' is called), there is no problem, because
we have our own version of GNU Bash and that installed version is used.
Probably some more issues still remain and will hopefully be found in the
future.

While doing this, I also noticed the following two minor issues:

 - The `./project configure' option `--input-dir' was not recognized
   because it was mistakenly checking `--inputdir'. It has been corrected.

 - The test C programs now use the `&lt;&lt;EOF' method instead of `echo'.

 - In `basic.mk', the extra space between `syspath' and `:=' was removed
   (it was an ancient relic!).
</pre>
</div>
</content>
</entry>
<entry>
<title>Hashbangs of project and configure.sh set to /bin/sh</title>
<updated>2020-01-23T19:21:15+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-23T19:17:16+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=cccf8782014beb85b8c8cb23d6ece56c849773f3'/>
<id>cccf8782014beb85b8c8cb23d6ece56c849773f3</id>
<content type='text'>
Until now, the hashbang of these two shell scripts was set to `/bin/bash',
hence assuming that GNU Bash exists on the host system! But this is an
extra requirement on the host operating system and these two scripts should
be written such that they operate on a POSIX shell (the generic `/bin/sh'
which can point to any shell program).

With this commit this has been implemented! We may confront some errors as
the system is run on other systems, but we should fix such errors and work
hard to make these two scripts as POSIX-compatible as possible (runnable on
any shell, so as not to force users to install Bash before running the
project).

This completes Task #15525.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the hashbang of these two shell scripts was set to `/bin/bash',
hence assuming that GNU Bash exists on the host system! But this is an
extra requirement on the host operating system and these two scripts should
be written such that they operate on a POSIX shell (the generic `/bin/sh'
which can point to any shell program).

With this commit this has been implemented! We may confront some errors as
the system is run on other systems, but we should fix such errors and work
hard to make these two scripts as POSIX-compatible as possible (runnable on
any shell, so as not to force users to install Bash before running the
project).

This completes Task #15525.
</pre>
</div>
</content>
</entry>
<entry>
<title>IMPORTANT: Project preparation is now also done with project make</title>
<updated>2020-01-23T19:16:16+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-23T18:54:29+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=2264c0fea5668cd8f827cdbd89004fd390b0b14f'/>
<id>2264c0fea5668cd8f827cdbd89004fd390b0b14f</id>
<content type='text'>
Until now, the main commands to run the project were these: `./project
configure' (to build the software), `./project prepare' (to possibly
arrange input datasets and build special configuration Makefiles) and
finally `./project make' to run the project.

The main logic behind the "prepare" phase `top-prepare.mk' is to build
configuration files that can be fed into the "make" step and optimize its
operation. For example when the total number of necessary inputs for the
majority of the analysis is not as large as the total number of
inputs. With "prepare" (when necessary), you go through the raw inputs,
select the ones that are necessary for the rest of the project. The output
of `top-prepare.mk' is a configuration file (a Make variable) that keeps
the IDs (numbers, names, etc). That configuration file would then be used
in the `top-make.mk' to identify the lower level targets and allow optimal
project organization and management.

But the last two are both part of the analysis, and while they indeed need
different calls to Make to be executed, many projects don't actually need a
preparation phase: ultimately, its an implementation choice by the project
developers and doesn't concern the project users (or the developers when
they are running it).

To avoid confusing the users, or simply annoying them when a projet doesn't
need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk'
Makefiles are called with the single `./project make' command and
`./project prepare' has been dropped. I noticed this while writing the
paper on this system.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the main commands to run the project were these: `./project
configure' (to build the software), `./project prepare' (to possibly
arrange input datasets and build special configuration Makefiles) and
finally `./project make' to run the project.

The main logic behind the "prepare" phase `top-prepare.mk' is to build
configuration files that can be fed into the "make" step and optimize its
operation. For example when the total number of necessary inputs for the
majority of the analysis is not as large as the total number of
inputs. With "prepare" (when necessary), you go through the raw inputs,
select the ones that are necessary for the rest of the project. The output
of `top-prepare.mk' is a configuration file (a Make variable) that keeps
the IDs (numbers, names, etc). That configuration file would then be used
in the `top-make.mk' to identify the lower level targets and allow optimal
project organization and management.

But the last two are both part of the analysis, and while they indeed need
different calls to Make to be executed, many projects don't actually need a
preparation phase: ultimately, its an implementation choice by the project
developers and doesn't concern the project users (or the developers when
they are running it).

To avoid confusing the users, or simply annoying them when a projet doesn't
need it, with this commit, the top-level `top-prepare.mk' and `top-make.mk'
Makefiles are called with the single `./project make' command and
`./project prepare' has been dropped. I noticed this while writing the
paper on this system.
</pre>
</div>
</content>
</entry>
<entry>
<title>Better explanation for missing static C library</title>
<updated>2020-01-22T15:19:44+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-22T15:14:03+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=cf0427009decbcad33a269898a428341e9c0ceff'/>
<id>cf0427009decbcad33a269898a428341e9c0ceff</id>
<content type='text'>
Until now, the explanation for a missing static C library didn't actually
guide the users to look above and see the error message! So with this
commit, I edited it a little to be more clear (and mention to look
above). Also, I noticed that on Amazon AWS systems, the static C library is
installed as a separate package, so to help the users, I added the
necessary command and some better explanation.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the explanation for a missing static C library didn't actually
guide the users to look above and see the error message! So with this
commit, I edited it a little to be more clear (and mention to look
above). Also, I noticed that on Amazon AWS systems, the static C library is
installed as a separate package, so to help the users, I added the
necessary command and some better explanation.
</pre>
</div>
</content>
</entry>
<entry>
<title>IMPORTANT!!! Configuration Makefiles now have a .conf suffix</title>
<updated>2020-01-20T01:27:03+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-20T00:05:41+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=7ac86df891798fd1f0ef4d7a40aff43ec9621854'/>
<id>7ac86df891798fd1f0ef4d7a40aff43ec9621854</id>
<content type='text'>
Until now, the configuration Makefiles (in
`reproduce/software/config/installation' and `reproduce/analysis/config')
had a `.mk' suffix, similar to the workhorse Makefiles. Although they are
indeed Makefiles, but given their nature (to only keep configuration
parameters), it is confusing (especially to early users) for them to also
have a `.mk' (similar to the analysis or software building Makefiles).

To address this issue, with this commit, all the configuration Makefiles
(in those directories) are now given a `.conf' suffix. This is also assumed
for all the files that are loaded.

The configuration (software building) and running of the template have been
checked with this change from scratch, but please report any error that may
not have been noticed.

THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS
FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE
SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES
ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the configuration Makefiles (in
`reproduce/software/config/installation' and `reproduce/analysis/config')
had a `.mk' suffix, similar to the workhorse Makefiles. Although they are
indeed Makefiles, but given their nature (to only keep configuration
parameters), it is confusing (especially to early users) for them to also
have a `.mk' (similar to the analysis or software building Makefiles).

To address this issue, with this commit, all the configuration Makefiles
(in those directories) are now given a `.conf' suffix. This is also assumed
for all the files that are loaded.

The configuration (software building) and running of the template have been
checked with this change from scratch, but please report any error that may
not have been noticed.

THIS IS AN IMPORTANT CHANGE AND WILL CAUSE CRASHES OR UNEXPECTED BEHAVIORS
FOR PROJECTS THAT HAVE BRANCHED FROM THIS TEMPLATE. PLEASE CORRECT THE
SUFFIX OF ALL YOUR PROJECT'S CONFIGURATION MAKEFILES (IN THE DIRECTORIES
ABOVE), OTHERWISE THEY AREN'T AUTOMATICALLY LOADED ANYMORE.
</pre>
</div>
</content>
</entry>
<entry>
<title>Corrected typo in last commit (forgetting \ at end of line)</title>
<updated>2020-01-19T22:21:34+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-19T22:21:34+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=0ccea404bf994525a6f42ccea8ed7e325660627a'/>
<id>0ccea404bf994525a6f42ccea8ed7e325660627a</id>
<content type='text'>
In the previous commmit, I had forgot to add a `\' after the newly added
`sys_library_path' variable to the `high-level.mk' call.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
In the previous commmit, I had forgot to add a `\' after the newly added
`sys_library_path' variable to the `high-level.mk' call.
</pre>
</div>
</content>
</entry>
<entry>
<title>LIBRARY_PATH is set accordingly based on the host</title>
<updated>2020-01-19T21:15:37+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2020-01-19T21:15:37+00:00</published>
<link rel='alternate' type='text/html' href='http://git.maneage.org/project.git/commit/?id=be73ea496657099603cce7e20413b9c6ee993c94'/>
<id>be73ea496657099603cce7e20413b9c6ee993c94</id>
<content type='text'>
Until now, GCC wouldn't build properly on Debian-based operating systems
because `ld' needed to link with several necessary C library features like
`crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to
add the directory containing them to `LIBRARY_PATH'. In the previous
commit, I actually searched for these files, but while testing on another
system, I noticed that it can be problematic (other architectures may
exist).

With this commit, we are actually finding the build architecture of the
running GCC (which is the same as the `ld') and using that to fix a fixed
directory to `LIBRARY_PATH'.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, GCC wouldn't build properly on Debian-based operating systems
because `ld' needed to link with several necessary C library features like
`crti.o' and `crtn.o' (this is an `ld' issue, not GCC). The solution is to
add the directory containing them to `LIBRARY_PATH'. In the previous
commit, I actually searched for these files, but while testing on another
system, I noticed that it can be problematic (other architectures may
exist).

With this commit, we are actually finding the build architecture of the
running GCC (which is the same as the `ld') and using that to fix a fixed
directory to `LIBRARY_PATH'.
</pre>
</div>
</content>
</entry>
</feed>
