Difference between revisions of "DAP Clients - Authentication"

From OPeNDAP Documentation
(ToolsUI)
(Panoply)
Line 325: Line 325:
 
When I committed the edit (aka hit Enter) Panoply popped up a dialog box that indicated that the ''uat.urs.earthdata.nasa.gov'' server wanted my credentials.
 
When I committed the edit (aka hit Enter) Panoply popped up a dialog box that indicated that the ''uat.urs.earthdata.nasa.gov'' server wanted my credentials.
  
[[File:PanoplyAuthDialog.png |broder|frame|left|Panoply Authentication Dialog]]
+
[[File:PanoplyAuthDialog.png |broder|frame|left|Panoply URS Authentication Dialog]]
 +
<div style="clear: both"></div>
  
<div style="clear: both"></div>
 
 
I entered them, clicked the save password check box, and clicked the ''OK'' button. ToolsUI was then able to access the requested resource.
 
I entered them, clicked the save password check box, and clicked the ''OK'' button. ToolsUI was then able to access the requested resource.
  
 
=== LDAP ===
 
=== LDAP ===
 +
 +
[[File:Panoply-LDAP.png |broder|frame|left|Panoply LDAP Authentication Dialog]]
 +
<div style="clear: both"></div>
 +
 
=== Shibboleth ===
 
=== Shibboleth ===
 +
 +
[[File:Panoply-Shibboleth.png |broder|frame|left|Panoply Shibboleth Authentication Failure]]
 +
<div style="clear: both"></div>

Revision as of 15:26, 24 November 2014

1 Overview

Many users access DAP servers using a browser as their primary software interface. However there is also a growing group of users that utilize either:

  • A "smart" tool. Where "smart" means that the tool understands how to interact with a DAP service and construct DAP queries for data and use that in higher level client side activities like GUI based graphs, image display, selection, navigation etc.
  • A command line tool such as "wget" or "curl" that can be used to extract data from a DAP service, but the URL construction is left to other software.

In both these examples we want these client software applications to be able to manage authentication without user intervention, otherwise the automation benefits of these tools is lost.

2 curl (a.k.a. lib_curl)

2.1 URS

I was able to use command line curl to retrieve URS authenticated resources using the following technique.

First in my home directory I created a .netrc file and set its file permissions to read only for owner:

[spooky:~] ndp% touch .netrc
[spooky:~] ndp% chmod 600 .netrc
[spooky:~] ndp% ls -l .netrc
-rw-------@ 1 ndp  staff  92 Nov 13 06:08 .netrc

Then I edited the .netrc file and associated my URS credentials with the URS IdP instance utilized by my target DAP server:

machine uat.urs.earthdata.nasa.gov
	login my_urs_uid
	password my_urs_password

I could then access the top level directory of the URS authentication enabled Hyrax server with the following curl command:

curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap

What is happening here?

-k
This tells curl to accept self-signed certificates. This is ok for working with trusted (as in your own) "test" services but should be removed for working with production systems. Because: Security, Chain-Of-Trust, etc.
-n
This tells curl to use that ~/.netrc file I created.
-c ursCookies
This tells curl to stash cookies in the file ursCookies
-b ursCookies
This tells curl to read cookies from the file ursCookies
-L
This tells curl to follow redirects, which is a must for any OAuth2 flow.
--url https://54.172.97.47/opendap
The desired URL, protected by the URS OAuth2 authentication flow.

In order to retrieve multiple URLs with out reauthenticating you can use multiple instances of the --url parameter:

curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap --url https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds --url https://54.172.97.47/opendap/data/nc/coads_climatology.nc.dds

Or, since curl is actually pretty smart about using cookies and such you can also make multiple curl requests with the same cookies and it won't have to reauthenticate with URS once it's authenticated the first time:

curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap
curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap/data/nc/coads_climatology.nc.dds

2.2 LDAP

I was able to use command line curl to retrieve LDAP authenticated resources using the following technique.

First in my home directory I created a .netrc file and set its file permissions to read only for owner:

[spooky:~] ndp% touch .netrc
[spooky:~] ndp% chmod 600 .netrc
[spooky:~] ndp% ls -l .netrc
-rw-------@ 1 ndp  staff  92 Nov 13 06:08 .netrc

Then I edited the .netrc file and associated my LDAP credentials with the LDAP authenticated DAP server:

machine 130.56.244.153
	login tesla
	password password

I could then access the top level directory of the LDAP authentication enabled Hyrax server with the following curl command:

curl -k -n -c ldapCookies -b ldapCookies  --url https://130.56.244.153/opendap

What is happening here?

-k
This tells curl to accept self-signed certificates. This is ok for working with trusted (as in your own) "test" services but should be removed for working with production systems. Because: Security, Chain-Of-Trust, etc.
-n
This tells curl to use that ~/.netrc file I created.
-c ursCookies
This tells curl to stash cookies in the file ldapCookies
-b ursCookies
This tells curl to read cookies from the file ldapCookies
--url https://130.56.244.153/opendap
The desired URL, protected LDAP authentication.

Note that the credentials are sent with every request so secure transport is a must if user account are to be protected.

2.3 Shibboleth

2.3.1 .netrc

I was not able to use command line curl to retrieve Shibboleth authentication resources using the .netrc technique described in the LDAP and URS sections.

Analysis of the HTTP conversation between the idp.testshib.org server and curl shows that curl correctly follows the the series of 302 redirects issued to it, first by the Apache service bound to the Hyrax server and then from the idp.testshib.org server. In every request to the idp.testshib.org server the curl client correctly offers the credentials via the HTTP Authorization header:

0000: GET /idp/Authn/UserPassword HTTP/1.1
0026: Authorization: Basic bXlzZWxmOm15c2VsZg==
0051: User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.2
0091: 1.4 OpenSSL/0.9.8z zlib/1.2.5
00b0: Host: idp.testshib.org
00c8: Accept: */*
00d5: Cookie: _idp_authn_lc_key=efbb6e2a9d893b47fb802ed575329ce69c101b
0115: 3ea8beb6744fab64fc406c358f; JSESSIONID=5A1731EDE00613B13803968CF
0155: AF06284
015e:

But the Shibboleth system doesn't respond to them. This may be a simple configuration issue on the Shibboleth end, or it could be that the Shibboleth protocol specifically forbids accepting credentials via HTTP Authorization headers.

2.3.2 certificates

3 wget

3.1 URS

The wget documentation indicates that wget understands to use the .netrc file that we created for curl, and happily it appears to work, as long as other things are in place. Consider this wget command:

wget  --load-cookies cookies --save-cookies cookies --keep-session-cookie --no-check-certificate https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds

What's happening here?

--load-cookies cookies
Load cookies from the file "cookies"
--save-cookies cookies
Save cookies to the file "cookies"
--keep-session-cookie
Save session cookies.
--no-check-certificate
Do not check the authenticity of the (self signed) certificates. This is good for testing against your own servers running with self-signed certificates in that this switch will allow you to experience success when interacting with such servers. However, this switch breaks the chain of trust and may allow bad things to happen if used on the open internets. THus, for regular use, do not include this switch!
https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
The URL to retrieve.


Here's the output of said wget request:

[spooky:olfs/testsuite/urs] ndp% wget  --load-cookies cookies --save-cookies cookies --keep-session-cookie --no-check-certificate https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
--2014-11-14 11:22:18--  https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
Connecting to 54.172.97.47:443... connected.
WARNING: cannot verify 54.172.97.47's certificate, issued by `/C=US/ST=RI/L=Narragansett/O=OPeNDAP Inc./OU=Engineering/CN=54.172.97.47/emailAddress=support@opendap.org':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 302 Found
Location: https://uat.urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=04xHKVaNdYNzCBG6KB7-Ig&response_type=code&redirect_uri=https%3A%2F%2F54.172.97.47%2Fopendap%2Flogin&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw [following]
--2014-11-14 11:22:19--  https://uat.urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=04xHKVaNdYNzCBG6KB7-Ig&response_type=code&redirect_uri=https%3A%2F%2F54.172.97.47%2Fopendap%2Flogin&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw
Resolving uat.urs.earthdata.nasa.gov... 198.118.243.34, 2001:4d0:241a:4089::91
Connecting to uat.urs.earthdata.nasa.gov|198.118.243.34|:443... connected.
WARNING: certificate common name `uat.earthdata.nasa.gov' doesn't match requested host name `uat.urs.earthdata.nasa.gov'.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to uat.urs.earthdata.nasa.gov|198.118.243.34|:443... connected.
WARNING: certificate common name `uat.earthdata.nasa.gov' doesn't match requested host name `uat.urs.earthdata.nasa.gov'.
HTTP request sent, awaiting response... 302 Found
Location: https://54.172.97.47/opendap/login?code=a590cfc189783e29a7b8ab3ce1e0357618cbab3f590e7268a26e7ad1f7cf899d&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw [following]
--2014-11-14 11:22:20--  https://54.172.97.47/opendap/login?code=a590cfc189783e29a7b8ab3ce1e0357618cbab3f590e7268a26e7ad1f7cf899d&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw
Connecting to 54.172.97.47:443... connected.
WARNING: cannot verify 54.172.97.47's certificate, issued by `/C=US/ST=RI/L=Narragansett/O=OPeNDAP Inc./OU=Engineering/CN=54.172.97.47/emailAddress=support@opendap.org':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 302 Found
Location: https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds [following]
--2014-11-14 11:22:21--  https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
Connecting to 54.172.97.47:443... connected.
WARNING: cannot verify 54.172.97.47's certificate, issued by `/C=US/ST=RI/L=Narragansett/O=OPeNDAP Inc./OU=Engineering/CN=54.172.97.47/emailAddress=support@opendap.org':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: `fnoc1.nc.dds'

    [ <=>                                                                                                                                                                                                            ] 197         --.-K/s   in 0s     

2014-11-14 11:22:22 (7.23 MB/s) - `fnoc1.nc.dds' saved [197]

[spooky:olfs/testsuite/urs] ndp% more fnoc1.nc.dds
Dataset {
    Int16 u[time_a = 16][lat = 17][lon = 21];
    Int16 v[time_a = 16][lat = 17][lon = 21];
    Float32 lat[lat = 17];
    Float32 lon[lon = 21];
    Float32 time[time = 16];
} fnoc1.nc;

It appears that wget followed the first redirect to uat.urs.earthdata.nasa.gov, where the URS server responded with a "401 Unauthorized" (thanks to the the app_type=401 query parameter in the redirect URL provided by mod_auth_urs). After getting the 401 wget resubmits the request with the authentication credentials and the URS server accepts them and redirects wget back to the mod_auth_urs server to complete the request.

3.2 LDAP

3.3 Shibboleth

4 ncdump

ncdump utilizes the NetCDF-C library (which uses the the "oc" library, which uses lib_curl under the hood) to access DAP resources so in theory if it works for ncdump then it should work for other clients that use the NetCDF-C library.

4.1 URS

The following works with the ncdump (and oc client) code bundled with NetCDF-4.3.2 Previous versions including 4.3.1 will not work.

Edit (create as needed) the file .dodsrc in your current working directory and set its file permissions to read only for owner:

[spooky:~] ndp% touch .dodsrc
[spooky:~] ndp% chmod 600  .dodsrc
[spooky:~] ndp% ls -l . dodsrc
-rw-------@ 1 ndp  staff  92 Nov 13 06:08 . dodsrc

Add your URS credentials and associate them with the URS authenticated DAP service that you wish to access. This will not work if you (correctly) associate the URS credentials with the URS server (go figure):

[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************


Here is a typical .dodsrc file.

# OPeNDAP client configuation file. See the OPeNDAP
# users guide for information.
USE_CACHE=0
# Cache and object size are given in megabytes (20 ==> 20Mb).
MAX_CACHE_SIZE=20
MAX_CACHED_OBJ=5
IGNORE_EXPIRES=0
CACHE_ROOT=/Users/ndp/.dods_cache/
DEFAULT_EXPIRES=1
ALWAYS_VALIDATE=1
# Request servers compress responses if possible?
# 1 (yes) or 0 (false).
DEFLATE=0
# Proxy configuration:
# PROXY_SERVER=<protocol>,<[username:password@]host[:port]>
# NO_PROXY_FOR=<protocol>,<host|domain>
# AIS_DATABASE=<file or url>
[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************

This version of ncdump (the oc client) appears to ignore the .dodsrc HTTP.COOKIEJAR and HTTP.COOKIEFILE parameters. This means that ncdump will authenticate each time it is invoked.

4.2 LDAP

4.3 Shibboleth

5 Integrated Data Viewer (IDV)

The Integrated Data Viewer is GUI driven data client that is based around the CDM/NetCDF data model and utilizes that NetCDF-Java (and thus the Java DAP implementation) to access remote DAP datasets. Because it has a GUI it can retrieve (and cache for later) users credentials directly from the user. Since IDV utilizes the Java-NetCDF library to access DAP resources then in theory if it works for IDV then it should work for all the other clients that use the Java-NetCDF library.

5.1 URS

For testing I utilized my AWS test service, configured to require URS authentication for all access of Hyrax.

I downloaded the latest version of IDV (5.0u2 on 11/19/14) and installed it on my local system. I launched IDV and then attempted to choose a new dataset by starting with the "Data" menu: Data > Choose Data > From A Web Server

In the resulting pane I entered the AWS test service URL for our friend coads_climatology.nc:

https://54.172.97.47/opendap/data/nc/coads_climatology.nc

When I committed the edit (aka hit Enter) IDV popped up a dialog box that indicated that the uat.urs.earthdata.nasa.gov server wanted my credentials:

IDV URS Authentication Dialog

I entered them, clicked the save password check box, and clicked the OK button. IDV was then able to access the requested resource. After the first successful access other resources at the AWS server were also available, but without an additional authentication challenge being presented to the user.

5.2 LDAP

IDV LDAP Authentication Dialog

5.3 Shibboleth

IDV Shibboleth Authentication Failure Dialog

6 ToolsUI

The ToolsUI application is a simple is GUI driven data client that is based around the CDM/NetCDF data model and utilizes that NetCDF-Java (and thus the Java DAP implementation) to access remote DAP datasets. Because it has a GUI it can retrieve (and cache for later) users credentials directly from the user.

6.1 URS

For testing I utilized my AWS test service, configured to require URS authentication for all access of Hyrax.

I downloaded the latest version of ToolsUI (4.5 on 11/19/14) and installed it on my local system. I launched ToolsUI using the command line:

java -Xmx1g -jar toolsUI-4.5.jar

I selected the Viewer tab, and entered the AWS test service URL for our friend coads_climatology.nc:

https://54.172.97.47/opendap/data/nc/coads_climatology.nc

When I committed the edit (aka hit Enter) ToolsUI popped up a dialog box that indicated that the uat.urs.earthdata.nasa.gov server wanted my credentials.

ToolsUI URS Authentication Dialog

I entered them, clicked the save password check box, and clicked the OK button. ToolsUI was then able to access the requested resource.

6.2 LDAP

ToolsUI LDAP Authentication Dialog


6.3 Shibboleth

ToolsUI Shibboleth Authentication Failure

7 Panoply

The Panoply application is a sophisticated GUI driven data client that is based around the CDM/NetCDF data model and utilizes that NetCDF-Java (and thus the Java DAP implementation) to access remote DAP datasets. Because it has a GUI it can retrieve (and cache for later) users credentials directly from the user.

7.1 URS

For testing I utilized my AWS test service, configured to require URS authentication for all access of Hyrax.

I downloaded the latest version of Panoply (4.0.5 on 11/20/14) and installed it on my local system. I launched Panoply (clicking it's icon in my Applications folder)

From the File menu, I selected "Open Remote Dataset.." and in the pop dialog I entered the URL for our friend coads_climatology.nc:

https://54.172.97.47/opendap/data/nc/coads_climatology.nc

When I committed the edit (aka hit Enter) Panoply popped up a dialog box that indicated that the uat.urs.earthdata.nasa.gov server wanted my credentials.

Panoply URS Authentication Dialog

I entered them, clicked the save password check box, and clicked the OK button. ToolsUI was then able to access the requested resource.

7.2 LDAP

Panoply LDAP Authentication Dialog

7.3 Shibboleth

Panoply Shibboleth Authentication Failure