From 43186705c89e99fd4dbf5549ad031d27d77dfe6f Mon Sep 17 00:00:00 2001 From: Mohammad Akhlaghi Date: Thu, 25 Aug 2022 17:02:29 +0200 Subject: Added server authentication and FITS DATASUM for verficiation SUMMARY: Nothing special is necessary for your existing projects. This commit just addds two new features (read the commit description for more): 1. To provide a user and password to servers that need authentication before they allow downloading of proprietary data, 2. To use the FITS Standard's DATASUM for file verification (for cases where the file is not static on the server, and is generated upon receiving your download request). Until now, Maneage didn't have any infrastructure for databases that require authentication (through a user or password, when calling 'wget'). Furthermore, when the downloaded file is automatically generated by the server upon request, the server usually adds metadata (like file date, or query number and etc) in the header. Therefore the simple SHA256 checksum of the file would differ on every download! This made it very hard to verify if the data (not headers) are unchanged. With this commit, both these problems have been addressed: - Server authentication: the 'reproduce/software/config/LOCAL.conf' now contains three new variables for this purpose. With them, you can give your username and password, along with the authentication method of the server. The comments on top of these three variables give a full description of their usage. - Verifying only the data in a file (ignoring the headers): The 'reproduce/analysis/config/INPUTS.conf' now accepts two new optional variables for each input file using the FITS standard's DATASUM convention: 'INPUT-%-fitsdatasum' and 'INPUT-%-fitshdu'. If the SHA256 isn't specified for a file, Maneage will use these to verify the file. With the latter, you specify the HDU of the data you want to verify and with the former you give the DATASUM value for that HDU. As the name suggests, this is only valid for FITS files. If we find other formats that support a similar behavior, we can add this feature for those formats also. This is also thoroughly discussed in the comments of 'reproduce/analysis/config/INPUTS.conf'. This commit was done with the help of Pedram Ashofte Ardakani, Sepideh Eskandarlou and Mohammadreza Khellat. --- reproduce/software/config/LOCAL.conf.in | 43 +++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) (limited to 'reproduce/software') diff --git a/reproduce/software/config/LOCAL.conf.in b/reproduce/software/config/LOCAL.conf.in index a7434ea..b9d6842 100644 --- a/reproduce/software/config/LOCAL.conf.in +++ b/reproduce/software/config/LOCAL.conf.in @@ -15,3 +15,46 @@ DEPENDENCIES-DIR = @ddir@ SYS_CPATH = @sys_cpath@ DOWNLOADER = @downloader@ GROUP-NAME = @groupname@ + + + + + +# Server authentication/identification +# ------------------------------------ +# +# If you need to identify yourself to the database server (when downloading +# input files), you can write your user name and password in the two +# variables below. When these two variables are defined, the download rule +# (in 'reproduce/analysis/make/initialize.mk') will pass their values to +# the '--user' and '--password' options of WGET.\ +# +# The 'DATABASEAUTHTYPE' specifies which type of authentication is +# recognized by the database with the 'wget' command. It can take the +# following values: +# 'userpass': Assumes --user='XXXX' --password='YYYY' +# 'postdata': Assumes --post-data 'username=XXXX&password=YYYY' +# +# SPECIAL CHARACTERS IN PASSWORD: if your password has special characters +# like '#' or '$' (that can have special meaning for Make), then comment +# them with a back-slash. For example if your password is 'ab#cd', write it +# below as 'ab\#cd'. Within 'initialize.mk', the user name and password are +# placed inside single quotes before being used, so special characters +# won't be problematic. However, if your password includes characters like +# the single quote itself, this can be problematic. In this case, you can +# temporarily (without committing!) directly enter the password in the +# respective rule of 'initialize.mk' and after downloading, undo the change +# (which is easy with 'git restore'). If you have any ideas on how to +# account for such characters in a generic way, please let us know and +# we'll suggest them here for future users. +# +# SECURITY WARNING: only set the values of these two variables in +# 'LOCAL.conf' (which is _not_ under version control), _not_ +# 'LOCAL.conf.in' (which is under version control). It is also recommended +# to remove the ID and password values from 'LOCAL.conf' immediately after +# your downloads finish: even though 'LOCAL.conf' is not under version +# control, it will still be on your computer, which may be accessed by +# others or (mistakenly) shared with others. +DATABASEUSER = +DATABASEPASS = +DATABASEAUTHTYPE = -- cgit v1.2.1