Added server authentication and FITS DATASUM for verficiation

SUMMARY: Nothing special is necessary for your existing projects. This
commit just addds two new features (read the commit description for more):
  1. To provide a user and password to servers that need authentication
     before they allow downloading of proprietary data,
  2. To use the FITS Standard's DATASUM for file verification (for cases
     where the file is not static on the server, and is generated upon
     receiving your download request).

Until now, Maneage didn't have any infrastructure for databases that
require authentication (through a user or password, when calling
'wget'). Furthermore, when the downloaded file is automatically generated
by the server upon request, the server usually adds metadata (like file
date, or query number and etc) in the header. Therefore the simple SHA256
checksum of the file would differ on every download! This made it very hard
to verify if the data (not headers) are unchanged.

With this commit, both these problems have been addressed:

 - Server authentication: the 'reproduce/software/config/LOCAL.conf' now
   contains three new variables for this purpose. With them, you can give
   your username and password, along with the authentication method of the
   server. The comments on top of these three variables give a full
   description of their usage.

 - Verifying only the data in a file (ignoring the headers): The
   'reproduce/analysis/config/INPUTS.conf' now accepts two new optional
   variables for each input file using the FITS standard's DATASUM
   convention: 'INPUT-%-fitsdatasum' and 'INPUT-%-fitshdu'. If the SHA256
   isn't specified for a file, Maneage will use these to verify the
   file. With the latter, you specify the HDU of the data you want to
   verify and with the former you give the DATASUM value for that HDU. As
   the name suggests, this is only valid for FITS files. If we find other
   formats that support a similar behavior, we can add this feature for
   those formats also. This is also thoroughly discussed in the comments of
   'reproduce/analysis/config/INPUTS.conf'.

This commit was done with the help of Pedram Ashofte Ardakani, Sepideh
Eskandarlou and Mohammadreza Khellat.

1 files changed, 43 insertions, 0 deletions

diff --git a/reproduce/software/config/LOCAL.conf.in b/reproduce/software/config/LOCAL.conf.in
index a7434ea..b9d6842 100644
--- a/reproduce/software/config/LOCAL.conf.in
+++ b/reproduce/software/config/LOCAL.conf.in
@@ -15,3 +15,46 @@ DEPENDENCIES-DIR = @ddir@
 SYS_CPATH        = @sys_cpath@
 DOWNLOADER       = @downloader@
 GROUP-NAME       = @groupname@
+
+
+
+
+
+# Server authentication/identification
+# ------------------------------------
+#
+# If you need to identify yourself to the database server (when downloading
+# input files), you can write your user name and password in the two
+# variables below. When these two variables are defined, the download rule
+# (in 'reproduce/analysis/make/initialize.mk') will pass their values to
+# the '--user' and '--password' options of WGET.\
+#
+# The 'DATABASEAUTHTYPE' specifies which type of authentication is
+# recognized by the database with the 'wget' command. It can take the
+# following values:
+#   'userpass': Assumes --user='XXXX' --password='YYYY'
+#   'postdata': Assumes --post-data 'username=XXXX&password=YYYY'
+#
+# SPECIAL CHARACTERS IN PASSWORD: if your password has special characters
+# like '#' or '$' (that can have special meaning for Make), then comment
+# them with a back-slash. For example if your password is 'ab#cd', write it
+# below as 'ab\#cd'. Within 'initialize.mk', the user name and password are
+# placed inside single quotes before being used, so special characters
+# won't be problematic. However, if your password includes characters like
+# the single quote itself, this can be problematic. In this case, you can
+# temporarily (without committing!) directly enter the password in the
+# respective rule of 'initialize.mk' and after downloading, undo the change
+# (which is easy with 'git restore'). If you have any ideas on how to
+# account for such characters in a generic way, please let us know and
+# we'll suggest them here for future users.
+#
+# SECURITY WARNING: only set the values of these two variables in
+# 'LOCAL.conf' (which is _not_ under version control), _not_
+# 'LOCAL.conf.in' (which is under version control). It is also recommended
+# to remove the ID and password values from 'LOCAL.conf' immediately after
+# your downloads finish: even though 'LOCAL.conf' is not under version
+# control, it will still be on your computer, which may be accessed by
+# others or (mistakenly) shared with others.
+DATABASEUSER =
+DATABASEPASS =
+DATABASEAUTHTYPE =