From 40da7c8cc19098f21d93e8f1720071a64759f9ee Mon Sep 17 00:00:00 2001
From: Pedram Ashofteh Ardakani
```
* to this:
```html
some code
```
And the extra space is gone.
---
about.html | 112 +++++++++++++++---------------------------------
index.html | 22 +++-------
tutorial.html | 135 ++++++++++++++++++----------------------------------------
3 files changed, 84 insertions(+), 185 deletions(-)
diff --git a/about.html b/about.html
index cc233f1..248142a 100644
--- a/about.html
+++ b/about.html
@@ -212,9 +212,7 @@
command-line with the following command (you can come out of the "Info"
environment by pressing some code
q
).
-info make
-
+ info make
If you aren't familiar with the Info documentation format, we strongly
recommend running $ info info
and reading along. In less than an hour,
@@ -549,14 +547,12 @@ info make
create and go into the conventional master
branch to start
committing in your project later.
-git clone https://git.maneage.org/project.git # Clone/copy the project and its history.
+ git clone https://git.maneage.org/project.git # Clone/copy the project and its history.
mv project my-project # Change the name to your project's name.
cd my-project # Go into the cloned directory.
git remote rename origin origin-maneage # Rename current/only remote to "origin-maneage".
git checkout -b master # Create and enter your own "master" branch.
-pwd # Just to confirm where you are.
-
+pwd # Just to confirm where you are.
Prepare to build project: The ./project configure
command of the
next step will build the different software packages within the
"build" directory (that you will specify). Nothing else on your system
@@ -572,10 +568,8 @@ pwd # Just to
installed in the project build directory it will be removed. Again,
don't worry, nothing will be installed outside the build directory.
-# On another terminal (go to top project source directory, last command above)
-./project --check-config
-
# On another terminal (go to top project source directory, last command above)
+./project --check-config
Test Maneage: Before making any changes, it is important to test it
and see if everything works properly with the commands below. If there
is any problem in the ./project configure
or ./project make
steps,
@@ -587,11 +581,9 @@ pwd # Just to
Maneage for your project. But before that, clean all the extra Maneage
outputs with make clean
as shown below.
-./project configure # Build the project's software environment (can take an hour or so).
+ ./project configure # Build the project's software environment (can take an hour or so).
./project make # Do the processing and build paper (just a simple demo).
-# Open 'paper.pdf' and see if everything is ok.
-
Setup the remote: You can use any hosting facility that supports Git to keep an online copy of your project's version @@ -614,11 +606,9 @@ pwd # Just to command, you can actually check this (which local and remote branches are tracking each other).
-
-git remote add origin XXXXXXXXXX # Newly created repo is now called 'origin'.
+ git remote add origin XXXXXXXXXX # Newly created repo is now called 'origin'.
git push --set-upstream origin master # Push 'master' branch to 'origin' (with tracking).
-git push origin maneage # Push 'maneage' branch to 'origin' (no tracking).
-
Title, short description and author: The title and basic
information of your project's output PDF paper should be added in
paper.tex
. You should see the relevant place in the preamble (prior
@@ -657,11 +647,9 @@ git push origin maneage # Push 'maneage' b
download
to XXXXX
(any temporary string, you'll fix it in the
end of your project, when its complete).
Delete all delete-me*
files in the following directories:
-rm tex/src/delete-me*
+ rm tex/src/delete-me*
rm reproduce/analysis/make/delete-me*
-rm reproduce/analysis/config/delete-me*
-
Disable verification of outputs by removing the yes
from
reproduce/analysis/config/verify-outputs.conf
. Later, when you are
ready to submit your paper, or publish the dataset, activate
@@ -672,10 +660,8 @@ rm reproduce/analysis/config/delete-me*
Re-make the project (after a cleaning) to see if you haven't introduced any errors.
-
-./project make clean
-./project make
-
./project make clean
+./project make
Don't merge some files in future updates: As described below, you can later update your infra-structure (for example to fix bugs) by @@ -694,15 +680,13 @@ rm reproduce/analysis/config/delete-me* other change in that file. Also, don't add core Maneage Makefiles, otherwise Maneage can break on the next run.
-
-echo "paper.tex merge=ours" > .gitattributes
+ echo "paper.tex merge=ours" > .gitattributes
echo "tex/src/delete-me.mk merge=ours" >> .gitattributes
echo "tex/src/delete-me-demo.mk merge=ours" >> .gitattributes
echo "reproduce/analysis/make/delete-me.mk merge=ours" >> .gitattributes
echo "reproduce/software/config/TARGETS.conf merge=ours" >> .gitattributes
echo "reproduce/analysis/config/delete-me-num.conf merge=ours" >> .gitattributes
-git add .gitattributes
-
Copyright and License notice: It is necessary that all the
"copyright-able" files in your project (those larger than 10 lines)
have a copyright and license notice. Please take a moment to look at
@@ -720,10 +704,8 @@ git add .gitattributes
tex/src/preamble-header.tex
, reproduce/analysis/make/top-make.mk
,
and generally, all the files you modified in the previous step.
-Copyright (C) 2018-2020 Existing Name <existing@email.address>
-Copyright (C) 2020 YOUR NAME <YOUR@EMAIL.ADDRESS>
-
Copyright (C) 2018-2020 Existing Name <existing@email.address>
+Copyright (C) 2020 YOUR NAME <YOUR@EMAIL.ADDRESS>
Configure Git for fist time: If this is the first time you are
running Git on this system, then you have to configure it with some
basic information in order to have essential information in the commit
@@ -732,11 +714,9 @@ Copyright (C) 2020 YOUR NAME <YOUR@EMAIL.ADDRESS>
can also specify your favorite text editor for making the commit
(emacs
, vim
, nano
, and etc.).
-git config --global user.name "YourName YourSurname"
+ git config --global user.name "YourName YourSurname"
git config --global user.email your-email@example.com
-git config --global core.editor nano
-
Your first commit: You have already made some small and basic
changes in the steps above and you are in your project's master
branch. So, you can officially make your first commit in your
@@ -745,16 +725,14 @@ git config --global core.editor nano
always re-build the system before a commit to be sure it works as
expected.
-git status # See which files you have changed.
+ git status # See which files you have changed.
git diff # Check the lines you have added/changed.
./project make # Make sure everything builds successfully.
git add -u # Put all tracked changes in staging area.
git status # Make sure everything is fine.
git diff --cached # Confirm all the changes that will be committed.
git commit # Your first commit: put a good description!
-git push # Push your commit to your remote.
-
Start your exciting research: You are now ready to add flesh and blood to this raw skeleton by further modifying and adding your exciting research steps. You can use the "published works" section in @@ -798,9 +776,7 @@ git push # Push your commit to your remo dataset with the command below and replace it with your input's dataset.
-
-grep -ir wfpc2 ./*
-
grep -ir wfpc2 ./*
README.md
: Correct all the XXXXX
place holders (name of your
project, your own name, address of your project's online/remote
repository, link to download dependencies and etc). Generally, read
@@ -861,9 +837,7 @@ grep -ir wfpc2 ./*
attempt, the ./project make
script should be run with the
--prepare-redo
option, or you can delete the special file above.
-./project make --prepare-redo
-
./project make --prepare-redo
Pre-publication: add notice on reproducibility**: Add a notice somewhere prominent in the first page within your paper, informing the reader that your research is fully reproducible. For example in the @@ -981,9 +955,7 @@ grep -ir wfpc2 ./* here. If you use GNU Make, you can also see this page on your command-line:
-
-info make "automatic variables"
-
info make "automatic variables"
Debug: Since Make doesn't follow the common top-down paradigm, it
can be a little hard to get accustomed to why you get an error or
un-expected behavior. In such cases, run Make with the -d
@@ -1004,15 +976,13 @@ info make "automatic variables"
large2.fits
and delete large1.fits
in the same rule (when its no
longer needed). We can later do the same with large2.fits
when it
is no longer needed and so on.
-
-large1.fits.txt: input.fits
+ large1.fits.txt: input.fits
astarithmetic $< 2 + --output=$(subst .txt,,$@)
echo "done" > $@
large2.fits.txt: large1.fits.txt
astarithmetic $(subst .txt,,$<) 2 - --output=$(subst .txt,,$@)
rm $(subst .txt,,$<)
-echo "done" > $@
-
+echo "done" > $@
A more advanced Make programmer will use Make's call function
to define a wrapper in reproduce/analysis/make/initialize.mk
. This
wrapper will replace $(subst .txt,,XXXXX)
. Therefore, it will be
@@ -1031,8 +1001,7 @@ echo "done" > $@
deletion. For example, see the minimal working example Makefile below
(which you can actually put in a Makefile
and run if you have an
input.fits
in the same directory, and Gnuastro is installed).
-
-.ONESHELL:
+ .ONESHELL:
.SHELLFLAGS = -ec
all: mean-std.txt
shm-maneage := /dev/shm/$(shell whoami)-maneage-XXXXXXXXXX
@@ -1049,8 +1018,7 @@ echo "$$out" > $@
mean-std.txt: large2.txt
input=$$(cat $<)
aststatistics $$input.fits --mean --std > $@
-rm $$input.fits $$input
-
+rm $$input.fits $$input
The important point here is that the temporary name template
(shm-maneage
) has no suffix. So you can add the suffix
corresponding to your desired format afterwards (for example
@@ -1117,14 +1085,12 @@ rm $$input.fits $$input
conflicts which might happen in the merge (updated settings that
you have customized in Maneage).
-
-git checkout maneage
+ git checkout maneage
git pull # Get recent work in Maneage
git log XXXXXX..XXXXXX --reverse # Inspect new work (replace XXXXXXs with hashs mentioned in output of previous command).
git log --oneline --graph --decorate --all # General view of branches.
git checkout master # Go to your top working branch.
-git merge maneage # Import all the work into master.
-
Adding Maneage to a fork of your project: As you and your colleagues
continue your project, it will be necessary to have separate
forks/clones of it. But when you clone your own project on a
@@ -1135,11 +1101,9 @@ git merge maneage # Import all the work
below will setup the origin-maneage
remote, and a local maneage
branch to track it, on the new clone.
-git remote add origin-maneage https://git.maneage.org/project.git
+ git remote add origin-maneage https://git.maneage.org/project.git
git fetch origin-maneage
-git checkout -b maneage --track origin-maneage/maneage
-
Commit message: The commit message is a very important and useful aspect of version control. To make the commit message useful for others (or yourself, one year later), it is good to follow a @@ -1197,18 +1161,14 @@ git checkout -b maneage --track origin-maneage/maneage project):
-git bundle create my-project-git.bundle --all
-
+ git bundle create my-project-git.bundle --all
my-project-git.bundle
anywhere. Later, if
you need to un-bundle it, you can use the following command.
-git clone my-project-git.bundle
-
+ git clone my-project-git.bundle
@@ -1241,9 +1201,7 @@ git clone my-project-git.bundle
architecture) and the standard program name. For example (from the Nix
webpage):
-
-/nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1/
-
+ /nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1/
The important thing is that the "store" is not in the project's search
path. After the complete installation of the software, symbolic links are
diff --git a/index.html b/index.html
index ff5c557..59e661f 100755
--- a/index.html
+++ b/index.html
@@ -82,19 +82,15 @@
To start a new project, simply run these commands to clone it from its Git repository.
- Start building your project in Maneage
-git clone https://git.maneage.org/project.git # Clone Maneage, default branch `maneage'.
+
+git checkout -b master # Make new `master' branch, start customizing.git clone https://git.maneage.org/project.git # Clone Maneage, default branch `maneage'.
mv project my-project && cd my-project # Set custom name and enter directory.
git remote rename origin origin-maneage # Rename remote server to use `origin' later.
-git checkout -b master # Make new `master' branch, start customizing.
-
You are now ready to configure and make the raw template with the commands below. If they are successful, you can start customizing it. -
-./project configure # Build all necessary software from source.
-./project make # Do the analysis (download data, run software on data, build PDF).
-
+ ./project configure # Build all necessary software from source.
+./project make # Do the analysis (download data, run software on data, build PDF).
See the Customization Checklist in the cloned README-hacking.md
file for the next steps to start customizing Maneage for your project.
maneage
branch and create a new branch from there like below:
-
-git checkout maneage
-git branch -b my-fix
-
+ git checkout maneage
+git branch -b my-fix
-git push my-remote my-fix
-
+ git push my-remote my-fix
reproduce/software/config/installation/TARGETS.mk
and add to the
top-level-python
line, the word matplotlib
.
-
-# Python libraries/modules.
- top-level-python = astropy matplotlib
-
+ # Python libraries/modules.
+ top-level-python = astropy matplotlib
After that, run the configure step again with the option -e
to continue
using the same configuration options given before (input and build
directories). Also, run the prepare and make steps:
-./project configure -e
+ ./project configure -e
./project prepare
-./project make
-
+./project make
Open 'paper.pdf' and see if everything is fine. Note that now, Matplotlib
is appearing in the software appendix at the end of the document.
-git status # See which files have been changed.
+ git status # See which files have been changed.
git diff # See the lines you have modified.
git add -u # Put all tracked changes in staging area.
git status # Make sure everything is fine.
git commit # Your first commit, add a nice description.
-git tag -a v0.0 # Tag this as the zero-th version of your project.
-
+git tag -a v0.0 # Tag this as the zero-th version of your project.
Now, have a look at the Git
history of the project. Note that the local
master branch is one commit above than the remote origin/master branch.
@@ -166,11 +160,9 @@ git tag -a v0.0 # Tag this as the zero-th version of your project.
with the next commands. Since you had setup your master
branch to follow
origin/master
, you can just use git push
.
-git log --oneline --decorate --all --graph # Have a look at the Git history.
+ git log --oneline --decorate --all --graph # Have a look at the Git history.
git push # Push the commit to the remote/origin.
-git push --tags # Push all tags to the remote/origin.
-
+git push --tags # Push all tags to the remote/origin.
Now it is time to start including your own scripts to download and make the
analysis of the data. It is important to bear in mind that the goal of this
@@ -205,9 +197,7 @@ git push --tags # Push all tags to the remote/origi
into the bash
directory. Since there is any python
directory, create it
with the following command.
-mkdir reproduce/analysis/python
-
+ mkdir reproduce/analysis/python
After that, you need the Python script itself. The code is very simple: it
will take an input file containing two columns (year and population), the
@@ -217,8 +207,7 @@ mkdir reproduce/analysis/python
into the directory generated in the above step
(reproduce/analysis/python
).
-# Make a linear fit of an input data set
+ # Make a linear fit of an input data set
# This Python script makes a linear fitting of a data consisting in time and
# population. It generates a figure in which the original data and the
# fitted curve is plotted. Finally, it saves the fitting parameters.
@@ -299,9 +288,7 @@ np.savetxt(ofile, params, fmt='%.3f')
As it can be seen, this Python script (linear-fit.py
) is designed to be
invoked from the command line in the following way.
-
-python /path/to/linear-fit.py /path/to/input.dat /path/to/output.dat /path/to/figure.pdf
-
+ python /path/to/linear-fit.py /path/to/input.dat /path/to/output.dat /path/to/figure.pdf
/path/to/input.dat
is the input data file, /path/to/output.dat
is the
output data file (with the fitted parameters), and /path/to/figure.pdf
is
@@ -315,21 +302,17 @@ python /path/to/linear-fit.py /path/to/input.dat /path/to/output.dat /path/to/fi
information, comments or clarify any step. After that, add the files and
commit the work. Finally, push the commit to the remote/origin.
-
-git status # See which files you have changed.
+ git status # See which files you have changed.
git diff # See the lines you have added/changed.
git add reproduce/analysis/python/linear-fit.py # Put all tracked changes in staging area.
git commit # Commit, add a nice descriptions.
-git push # Push the commit to the remote/origin.
-
+git push # Push the commit to the remote/origin.
Check that everything is fine having a look at the Git
history of the
project. Note that the master
branch has been increased in one commit,
while the template
branch is behind.
-
-git log --oneline --decorate --all --graph # See the `Git` history.
-
+ git log --oneline --decorate --all --graph # See the `Git` history.
In short: in this section you have included a Python
script that will
be used for making the linear fitting.
@@ -355,8 +338,7 @@ git log --oneline --decorate --all --graph # See the `Git` history.
analysis. Save this Makefile in the dedicated directory
(reproduce/analysis/make
) with the name getdata-analysis.mk
. In that
Makefile, paste the following code.
-
-# Download data for the tutorial
+ # Download data for the tutorial
#
# In this Makefile, data for the tutorial is downloaded.
#
@@ -384,8 +366,7 @@ wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@
# It is very important to mention the address where the data were
# downloaded in the final report.
$(mtexdir)/getdata-analysis.tex: $(pop-data) | $(mtexdir)
-echo "\newcommand{\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
-
+echo "\newcommand{\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
Have a look at this Makefile and see the different parts. The first line is
a descriptive title. Below, include your name, contact email, and finally,
the copyright. Please, take your time in order to add all relevant
@@ -396,10 +377,8 @@ echo "\newcommand{\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
different parts. Then, you have the Make rule to download the data. Remember
the general structure of a Make rule:
-
-TARGETS: PREREQUISITES
-RECIPE
-
+ TARGETS: PREREQUISITES
+RECIPE
In a rule, it is said how to construct the TARGETS
from the
PREREQUISITES
, following the RECIPE
. Note that the white space at the
@@ -408,10 +387,8 @@ RECIPE
Now you can see this structure in our particular case:
-
-(pop-data): | $(indir)
-wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@
-
+ (pop-data): | $(indir)
+wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@
Here we have:
@@ -439,10 +416,8 @@ wget http://akhlaghi.org/data/template-tutorial/ESP.dat -O $@
consequence, they will be TeX
macros in which relevant information to be
included into the final paper are saved . Here, you are saving the URL
.
-
-(mtexdir)/getdata-analysis.tex: $(pop-data) | $(mtexdir)
-echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
-
+ (mtexdir)/getdata-analysis.tex: $(pop-data) | $(mtexdir)
+echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
In this final rule we have:
@@ -469,13 +444,11 @@ echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $
reproduce/analysis/make/top-make.mk
Makefile. There it is defined which
Makefiles have to be executed. You have to end up having:
-
-makesrc = initialize \
+ makesrc = initialize \
download \
getdata-analyse \
delete-me \
-paper
-
+paper
As allways, read carefully all comments and information in order to know
what is going ong. Also, add your own comments and information in order to
@@ -483,9 +456,7 @@ paper
fine, now the project is ready to download the data in the make step. Try
it!
-
-./project make
-
+ ./project make
Hopefully, it will download and save the file into the folder called
inputs
under the build-directory
. Check that it is there, and also have
@@ -528,34 +499,26 @@ paper
generated for downloading the data. But, before this, define the directory
in which the target is going to be saved.
-
-odir = $(BDIR)/fit-parameters
-
+ odir = $(BDIR)/fit-parameters
This is a folder under the build-directory
called fit-parameters
. After
that, define the target: a plain text file in which the linear fit
parameters are saved (by the Python script). Put it into the previously
defined directory. As the data is from Spain, name it ESP.txt
.
-
-param-file = $(odir)/ESP.txt
-
+ param-file = $(odir)/ESP.txt
Now, include a rule to construct the output directory odir
. This is
necessary because this directory is needed for saving the file ESP.txt
.
-
-(odir):
-mkdir $@
-
+ (odir):
+mkdir $@
With all the previous definitions, now it is possible to set the rule for
making the analysis:
-
-(param-file): $(indir)/ESP.dat | $(odir)
-python reproduce/analysis/python/linear-fit.py $< $@ $(odir)/ESP.pdf
-
+ (param-file): $(indir)/ESP.dat | $(odir)
+python reproduce/analysis/python/linear-fit.py $< $@ $(odir)/ESP.pdf
In this rule you have:
@@ -585,37 +548,29 @@ python reproduce/analysis/python/linear-fit.py $< $@ $(odir)/ESP.pdf
rule, define a couple of bash variables (a
and b
) that are the fitted
parameters extracted from the prerequisite. For a
:
-
-a=$$(cat $< | awk 'NR==1{print $1}')
-
+ a=$$(cat $< | awk 'NR==1{print $1}')
Similarly, for obtaining the parameter b
(which is in the second row):
-
-b=$$(cat $< | awk 'NR==2{print $1}')
-
+ b=$$(cat $< | awk 'NR==2{print $1}')
Then you have to specify the new TeX
commands for these two parameters,
just write them as it was done before for the URL
:
-
-echo "\newcommand{\afitparam}{$$a}" >> $@
-echo "\newcommand{\bfitparam}{$$b}" >> $@
-
+ echo "\newcommand{\afitparam}{$$a}" >> $@
+echo "\newcommand{\bfitparam}{$$b}" >> $@
So, at the end you will have the final rule like this:
(mtexdir)/getdata-analysis.tex: $(param-file) | $(mtexdir)
-
-echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
+ echo "\\newcommand{\\popurl}{http://akhlaghi.org/data/template-tutorial}" > $@
a=$$(cat $< | awk 'NR==1{print $1}')
b=$$(cat $< | awk 'NR==2{print $1}')
echo "\newcommand{\afitparam}{$$a}" >> $@
-echo "\newcommand{\bfitparam}{$$b}" >> $@
-
+echo "\newcommand{\bfitparam}{$$b}" >> $@
Important notes: you have to use two $
in order to use the bash $
character inside of a Make rule. Also, note that you have to put >>
in
@@ -626,8 +581,7 @@ echo "\newcommand{\bfitparam}{$$b}" >> $@
With all the above modifications, you are ready to obtain the fitting
parameters. If you add the necessary comments and information, the final
Makefile would look similar to:
-
-# Download data and linear fitting for the tutorial
+# Download data and linear fitting for the tutorial
# In this Makefile, data for the tutorial is downloaded. Then, a Python
# script is used to make a linear fitting. Finally, fitted parameters as
# well as the URL is saved into a TeX macro.
@@ -677,17 +631,14 @@ a=$$(cat $< | awk 'NR==1{print $1}')
b=$$(cat $< | awk 'NR==2{print $1}')
echo "\newcommand{\afitparam}{$$a}" >> $@
-echo "\newcommand{\bfitparam}{$$b}" >> $@
-
+echo "\newcommand{\bfitparam}{$$b}" >> $@
Have look at this Makefile and note that it is what it has been described
above. Take your time for making useful comments and modifying whatever you
think it is necessary. If everything is fine, now the project is ready to
download the data and make the linear fitting. Try it!
-
-./project make
-
+ ./project make
Hopefully, now you will have the fitted parameters into the
build-directory/fit-parameters/ESP.txt
file, and the figure in the same
@@ -721,11 +672,9 @@ echo "\newcommand{\bfitparam}{$$b}" >> $@
and add the following paragraph just at the beginning of the abstract
section.
-
-By following the steps described in the tutorial, I have been able to obtain this reproducible paper!
+ By following the steps described in the tutorial, I have been able to obtain this reproducible paper!
The project is very simple and it consists in download a file (from \popurl), and make an easy linear fit using a Python script.
-The linear fitting is $y=a*x+b$, with the following parameters: $a=\afitparam$ and $b=\bfitparam$
-
+The linear fitting is $y=a*x+b$, with the following parameters: $a=\afitparam$ and $b=\bfitparam$
As you can see, the TeX
definitions done before in the Makefiles, are now
included into the paper: \popurl
, \afitparam
, and \bfitparam
. If you
--
cgit v1.2.1