<feed xmlns='http://www.w3.org/2005/Atom'>
<title>project.git/for-group, branch maneage</title>
<subtitle>Core Maneage branch (where all projects derive from)</subtitle>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/'/>
<entry>
<title>Single wrapper instead of old ./configure, Makefile and ./for-group</title>
<updated>2019-07-28T01:53:34+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-07-28T01:32:20+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=6ef4cc854d1df46b719de5d66b45537b0aa11f92'/>
<id>6ef4cc854d1df46b719de5d66b45537b0aa11f92</id>
<content type='text'>
Until now, to work on a project, it was necessary to `./configure' it and
build the software. Then we had to run `.local/bin/make' to run the project
and do the analysis every time. If the project was a shared project between
many users on a large server, it was necessary to call the `./for-group'
script.

This way of managing the project had a major problem: since the user
directly called the lower-level `./configure' or `.local/bin/make' it was
not possible to provide high-level control (for example limiting the
environment variables). This was especially noticed recently with a bug
that was related to environment variables (bug #56682).

With this commit, this problem is solved using a single script called
`project' in the top directory. To configure and build the project, users
can now run these commands:

   $ ./project configure
   $ ./project make

To work on the project with other users in a group these commands can be
used:

  $ ./project configure --group=GROUPNAME
  $ ./project make      --group=GROUPNAME

The old options to both configure and make the project are still valid. Run
`./project --help' to see a list. For example:

  $ ./project configure -e --host-cc
  $ ./project make -j8

The old `configure' script has been moved to
`reproduce/software/bash/configure.sh' and is called by the new `./project'
script. The `./project' script now just manages the options, then passes
control to the `configure.sh' script. For the "make" step, it also reads
the options, then calls Make. So in the lower-level nothing has
changed. Only the `./project' script is now the single/direct user
interface of the project.

On a parallel note: as part of bug #56682, we also found out that on some
macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set
to blank. This is no problem because RPATH is automatically set in macOS
and the executables and libraries contain the absolute address of the
libraries they should link with. But having `DYLD_LIBRARY_PATH' can
conflict with some low-level system libraries and cause very hard to debug
linking errors (like that reported in the bug report).

This fixes bug #56682.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, to work on a project, it was necessary to `./configure' it and
build the software. Then we had to run `.local/bin/make' to run the project
and do the analysis every time. If the project was a shared project between
many users on a large server, it was necessary to call the `./for-group'
script.

This way of managing the project had a major problem: since the user
directly called the lower-level `./configure' or `.local/bin/make' it was
not possible to provide high-level control (for example limiting the
environment variables). This was especially noticed recently with a bug
that was related to environment variables (bug #56682).

With this commit, this problem is solved using a single script called
`project' in the top directory. To configure and build the project, users
can now run these commands:

   $ ./project configure
   $ ./project make

To work on the project with other users in a group these commands can be
used:

  $ ./project configure --group=GROUPNAME
  $ ./project make      --group=GROUPNAME

The old options to both configure and make the project are still valid. Run
`./project --help' to see a list. For example:

  $ ./project configure -e --host-cc
  $ ./project make -j8

The old `configure' script has been moved to
`reproduce/software/bash/configure.sh' and is called by the new `./project'
script. The `./project' script now just manages the options, then passes
control to the `configure.sh' script. For the "make" step, it also reads
the options, then calls Make. So in the lower-level nothing has
changed. Only the `./project' script is now the single/direct user
interface of the project.

On a parallel note: as part of bug #56682, we also found out that on some
macOS systems, the `DYLD_LIBRARY_PATH' environment variable has to be set
to blank. This is no problem because RPATH is automatically set in macOS
and the executables and libraries contain the absolute address of the
libraries they should link with. But having `DYLD_LIBRARY_PATH' can
conflict with some low-level system libraries and cause very hard to debug
linking errors (like that reported in the bug report).

This fixes bug #56682.
</pre>
</div>
</content>
</entry>
<entry>
<title>for-group script using internal chmod</title>
<updated>2019-05-15T15:52:57+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-05-15T15:51:55+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=146122a0ecc8b35263e9d4bfd968535e06456ad7'/>
<id>146122a0ecc8b35263e9d4bfd968535e06456ad7</id>
<content type='text'>
Until now, the `for-group' script was using the system's `chmod' at the end
of its operation. But when `for-group' reaches this level, we already have
our own custom `chmod'.

With this commit, for a consistent operation, we are now using our custom
`chmod' program for this final step.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the `for-group' script was using the system's `chmod' at the end
of its operation. But when `for-group' reaches this level, we already have
our own custom `chmod'.

With this commit, for a consistent operation, we are now using our custom
`chmod' program for this final step.
</pre>
</div>
</content>
</entry>
<entry>
<title>Replaced all occurances of pipeline in text</title>
<updated>2019-04-14T16:49:55+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-04-14T16:48:40+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=4722ea598edd6b630227404c48c1c09ac527e9b8'/>
<id>4722ea598edd6b630227404c48c1c09ac527e9b8</id>
<content type='text'>
All occurances of "pipeline" have been chanaged to "project" or "template"
withint the text (comments, READMEs, and comments) of the template. The
main template branch is now also named `template'.

This was all because `pipeline' is too generic and couldn't be
distinguished from the base, and customized project.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
All occurances of "pipeline" have been chanaged to "project" or "template"
withint the text (comments, READMEs, and comments) of the template. The
main template branch is now also named `template'.

This was all because `pipeline' is too generic and couldn't be
distinguished from the base, and customized project.
</pre>
</div>
</content>
</entry>
<entry>
<title>Corrected copyright notices and info about adding copyright info</title>
<updated>2019-04-13T22:43:07+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-04-13T22:43:07+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=0cbd2243458611caa2a3564b577987531bcd6934'/>
<id>0cbd2243458611caa2a3564b577987531bcd6934</id>
<content type='text'>
Until now, the files where the people were meant to change didn't have a
proper copyright notice (for example `Copyright (C) YOUR NAME.'). This was
wrong because the license does not convey copyright ownership. So the name
of the file's original author must always be included and when people
modify it (and add their own copyright-able modifications).

With this commit, the file's original author (and email) are added to the
copyright notice and when more than one person modified a file, both names
have their individual copyright notice.

Based on this, the description for adding a copyright notice in
`README-hacking.md' has also been modified.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the files where the people were meant to change didn't have a
proper copyright notice (for example `Copyright (C) YOUR NAME.'). This was
wrong because the license does not convey copyright ownership. So the name
of the file's original author must always be included and when people
modify it (and add their own copyright-able modifications).

With this commit, the file's original author (and email) are added to the
copyright notice and when more than one person modified a file, both names
have their individual copyright notice.

Based on this, the description for adding a copyright notice in
`README-hacking.md' has also been modified.
</pre>
</div>
</content>
</entry>
<entry>
<title>for-group: Options passed to configure or make</title>
<updated>2019-04-08T20:10:50+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-04-08T20:10:50+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=7b369c861aff6eefb5b0c0530eeb9ab2e3053891'/>
<id>7b369c861aff6eefb5b0c0530eeb9ab2e3053891</id>
<content type='text'>
Until now, the `for-group' script could only two options to Make and didn't
pass anything to `./configure'. With this commit, any thing given after the
first two tokens to `for-group' is passed directly to `./configure' or
Make.

Also, the test for the existance of the group has been modified to avoid
making a temporary file.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the `for-group' script could only two options to Make and didn't
pass anything to `./configure'. With this commit, any thing given after the
first two tokens to `for-group' is passed directly to `./configure' or
Make.

Also, the test for the existance of the group has been modified to avoid
making a temporary file.
</pre>
</div>
</content>
</entry>
<entry>
<title>for-group gives write permission to all built software in the end</title>
<updated>2019-03-11T21:09:04+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-03-11T21:09:04+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=7ea5e28b74535797277c2f796dcaddf7425a7d61'/>
<id>7ea5e28b74535797277c2f796dcaddf7425a7d61</id>
<content type='text'>
Since the `install' script also sets permissions manually, the permissions
that we define in `for-group' don't usually affect the installed
files. Therefore the installed files of one user can't be modified/deleted
by another. With this commit, after for-group finishes configuration, it
also adds the write flag for all group members in the whole installation
directory.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Since the `install' script also sets permissions manually, the permissions
that we define in `for-group' don't usually affect the installed
files. Therefore the installed files of one user can't be modified/deleted
by another. With this commit, after for-group finishes configuration, it
also adds the write flag for all group members in the whole installation
directory.
</pre>
</div>
</content>
</entry>
<entry>
<title>For-group script can allow to arguments to Make call</title>
<updated>2019-03-08T15:24:44+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-03-08T15:24:44+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=37b4551dc7cb394a76be1534a3c4de90bdb569c3'/>
<id>37b4551dc7cb394a76be1534a3c4de90bdb569c3</id>
<content type='text'>
Until now the `./for-group' script would only add one argument to the Make
call, but in some situations, you need a second argument is well. With this
option, any possible fourth argument to `./for-group' is passed to Make.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now the `./for-group' script would only add one argument to the Make
call, but in some situations, you need a second argument is well. With this
option, any possible fourth argument to `./for-group' is passed to Make.
</pre>
</div>
</content>
</entry>
<entry>
<title>for-group: better check of group name and fixed make argument</title>
<updated>2019-02-05T18:19:46+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-02-05T18:19:46+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=438e1970093621d3c67acc900b17afdeaa60329f'/>
<id>438e1970093621d3c67acc900b17afdeaa60329f</id>
<content type='text'>
A few issues came up while testing the `for-group' script in one of the
projects based on this pipeline that are being fixed with this commit:

  1) We are ultimately using the `sg' command to use the specified group,
     not `chgrp'. So in cases where `chgrp' has problems, this would cause
     a wrong error. So for the test of the given group's existance, we are
     now directly calling `sg'.

  2) In the call to `make' we were mistakenly giving make the `$2' (which
     is `make' on the command-line) argument. Since `./for-group' now takes
     the group name as its first argument, this should have been `$3'.

  3) To help in readability, and also allow for group names with a space,
     `reproducible_paper_group_name' is now defined and exported before the
     final call to `sg'.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A few issues came up while testing the `for-group' script in one of the
projects based on this pipeline that are being fixed with this commit:

  1) We are ultimately using the `sg' command to use the specified group,
     not `chgrp'. So in cases where `chgrp' has problems, this would cause
     a wrong error. So for the test of the given group's existance, we are
     now directly calling `sg'.

  2) In the call to `make' we were mistakenly giving make the `$2' (which
     is `make' on the command-line) argument. Since `./for-group' now takes
     the group name as its first argument, this should have been `$3'.

  3) To help in readability, and also allow for group names with a space,
     `reproducible_paper_group_name' is now defined and exported before the
     final call to `sg'.
</pre>
</div>
</content>
</entry>
<entry>
<title>Group name is now part of the local configuration</title>
<updated>2019-02-01T18:38:44+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-02-01T18:30:02+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=f41be41e376dfd521881ad2e6137360e549777de'/>
<id>f41be41e376dfd521881ad2e6137360e549777de</id>
<content type='text'>
Until now, the group name to build the project actually went into the Git
source of the project! This doesn't allow exact reproducibility on
different machines (where the group name may be different).

With this commit, the `for-group' script has been modified to accept the
group name as its first argument and pass that onto `configure' and
Make. This is much better now, because not only the existance of a group
installation is checked, but also the name of the group. It also made
things simpler (in particular in `LOCAL.mk.in').
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Until now, the group name to build the project actually went into the Git
source of the project! This doesn't allow exact reproducibility on
different machines (where the group name may be different).

With this commit, the `for-group' script has been modified to accept the
group name as its first argument and pass that onto `configure' and
Make. This is much better now, because not only the existance of a group
installation is checked, but also the name of the group. It also made
things simpler (in particular in `LOCAL.mk.in').
</pre>
</div>
</content>
</entry>
<entry>
<title>Sanity check to run the Make with proper group permissions</title>
<updated>2019-01-18T21:17:01+00:00</updated>
<author>
<name>Mohammad Akhlaghi</name>
<email>mohammad@akhlaghi.org</email>
</author>
<published>2019-01-18T20:36:11+00:00</published>
<link rel='alternate' type='text/html' href='https://git.maneage.org/project.git/commit/?id=42d3cef11bd9a84d11eb48a4ff9686d2e0ce5436'/>
<id>42d3cef11bd9a84d11eb48a4ff9686d2e0ce5436</id>
<content type='text'>
If the `./for-group' script is not used properly, it can lead to the whole
pipeline being re-run. Therefore it is important to do a sanity check
immediately at the start of Make's processing and inform the user if there
is a problem.

With this commit, `./for-group' exports the `reproducible_paper_for_group'
variable which is used by both the initial `./configure' script, and later
in each call to Make. The `./configure' script will use it to write a value
in `reproduce/config/pipeline/LOCAL.mk' and Make will use it to compare
with the value in `reproduce/config/pipeline/LOCAL.mk'.

If there is an inconsistency, Make will not even attempt to build anything
and will just print a message and abort.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
If the `./for-group' script is not used properly, it can lead to the whole
pipeline being re-run. Therefore it is important to do a sanity check
immediately at the start of Make's processing and inform the user if there
is a problem.

With this commit, `./for-group' exports the `reproducible_paper_for_group'
variable which is used by both the initial `./configure' script, and later
in each call to Make. The `./configure' script will use it to write a value
in `reproduce/config/pipeline/LOCAL.mk' and Make will use it to compare
with the value in `reproduce/config/pipeline/LOCAL.mk'.

If there is an inconsistency, Make will not even attempt to build anything
and will just print a message and abort.
</pre>
</div>
</content>
</entry>
</feed>
