DAP Clients - Authentication: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
No edit summary
Line 131: Line 131:
=== LDAP ===
=== LDAP ===
=== Shibboleth ===
=== Shibboleth ===
== ''ncdump'' ==
ncdump utilizes the NetCDF-C library (which uses the the "oc" library, which uses ''lib_curl'' under the hood) to access DAP resources so in theory if it works for ncdump then it should work for other clients that use the NetCDF-C library.
=== URS ===
The following works with the ncdump (and oc client) code bundled with NetCDF-4.3.2 Previous versions including 4.3.1 will not work.
Edit (create as needed) the file ''.dodsrc'' in your '''current working directory''' and set its file permissions to read only for owner:
<source lang="bash">
[spooky:~] ndp% touch .dodsrc
[spooky:~] ndp% chmod 600  .dodsrc
[spooky:~] ndp% ls -l . dodsrc
-rw-------@ 1 ndp  staff  92 Nov 13 06:08 . dodsrc
</source>
Add your URS credentials and '''associate them with the URS authenticated DAP service that you wish to access'''. This will not work if you (correctly) associate the URS credentials with the URS server (go figure):
<source lang="apache">
[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************
</source>
Here is a typical ''.dodsrc'' file.
<source lang="apache">
# OPeNDAP client configuation file. See the OPeNDAP
# users guide for information.
USE_CACHE=0
# Cache and object size are given in megabytes (20 ==> 20Mb).
MAX_CACHE_SIZE=20
MAX_CACHED_OBJ=5
IGNORE_EXPIRES=0
CACHE_ROOT=/Users/ndp/.dods_cache/
DEFAULT_EXPIRES=1
ALWAYS_VALIDATE=1
# Request servers compress responses if possible?
# 1 (yes) or 0 (false).
DEFLATE=0
# Proxy configuration:
# PROXY_SERVER=<protocol>,<[username:password@]host[:port]>
# NO_PROXY_FOR=<protocol>,<host|domain>
# AIS_DATABASE=<file or url>
[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************
</source>
This version of ncdump (the oc client) appears to ignore the .dodsrc HTTP.COOKIEJAR and HTTP.COOKIEFILE parameters. This means that ncdump will authenticate each time it is invoked.
=== LDAP ===
=== Shibboleth ===


== ''Integrated Data Viewer (IDV)'' ==
== ''Integrated Data Viewer (IDV)'' ==
Line 202: Line 255:
<div style="clear: both"></div>
<div style="clear: both"></div>
I entered them, clicked the save password check box, and clicked the ''OK'' button. ToolsUI was then able to access the requested resource.
I entered them, clicked the save password check box, and clicked the ''OK'' button. ToolsUI was then able to access the requested resource.
=== LDAP ===
=== Shibboleth ===
== ''ncdump'' ==
ncdump utilizes the NetCDF-C library (which uses the the "oc" library, which uses ''lib_curl'' under the hood) to access DAP resources so in theory if it works for ncdump then it should work for other clients that use the NetCDF-C library.
=== URS ===
The following works with the ncdump (and oc client) code bundled with NetCDF-4.3.2 Previous versions including 4.3.1 will not work.
Edit (create as needed) the file ''.dodsrc'' in your '''current working directory''' and set its file permissions to read only for owner:
<source lang="bash">
[spooky:~] ndp% touch .dodsrc
[spooky:~] ndp% chmod 600  .dodsrc
[spooky:~] ndp% ls -l . dodsrc
-rw-------@ 1 ndp  staff  92 Nov 13 06:08 . dodsrc
</source>
Add your URS credentials and '''associate them with the URS authenticated DAP service that you wish to access'''. This will not work if you (correctly) associate the URS credentials with the URS server (go figure):
<source lang="apache">
[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************
</source>
Here is a typical ''.dodsrc'' file.
<source lang="apache">
# OPeNDAP client configuation file. See the OPeNDAP
# users guide for information.
USE_CACHE=0
# Cache and object size are given in megabytes (20 ==> 20Mb).
MAX_CACHE_SIZE=20
MAX_CACHED_OBJ=5
IGNORE_EXPIRES=0
CACHE_ROOT=/Users/ndp/.dods_cache/
DEFAULT_EXPIRES=1
ALWAYS_VALIDATE=1
# Request servers compress responses if possible?
# 1 (yes) or 0 (false).
DEFLATE=0
# Proxy configuration:
# PROXY_SERVER=<protocol>,<[username:password@]host[:port]>
# NO_PROXY_FOR=<protocol>,<host|domain>
# AIS_DATABASE=<file or url>
[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************
</source>
This version of ncdump (the oc client) appears to ignore the .dodsrc HTTP.COOKIEJAR and HTTP.COOKIEFILE parameters. This means that ncdump will authenticate each time it is invoked.


=== LDAP ===
=== LDAP ===
=== Shibboleth ===
=== Shibboleth ===

Revision as of 15:51, 23 November 2014

Overview

Many users access DAP servers using a browser as their primary software interface. However there is also a growing group of users that utilize either:

  • A "smart" tool. Where "smart" means that the tool understands how to interact with a DAP service and construct DAP queries for data and use that in higher level client side activities like GUI based graphs, image display, selection, navigation etc.
  • A command line tool such as "wget" or "curl" that can be used to extract data from a DAP service, but the URL construction is left to other software.

In both these examples we want these client software applications to be able to manage authentication without user intervention, otherwise the automation benefits of these tools is lost.

curl (a.k.a. lib_curl)

URS

I was able to use command line curl to retrieve URS authentication protected resources using the following technique.

First in my home directory I created a .netrc file and set its file permissions to read only for owner:

[spooky:~] ndp% touch .netrc
[spooky:~] ndp% chmod 600 .netrc
[spooky:~] ndp% ls -l .netrc
-rw-------@ 1 ndp  staff  92 Nov 13 06:08 .netrc

Then I edited the .netrc file and associated my URS credentials with the URS IdP instance utilized by my target DAP server:

machine uat.urs.earthdata.nasa.gov
	login my_urs_uid
	password my_urs_password

I could then access the top level directory of the URS authentication enabled Hyrax server with the following curl command:

curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap

What is happening here?

-k
This tells curl to accept self-signed certificates. This is ok for working with trusted (as in your own) "test" services but should be removed for working with production systems. Because: Security, Chain-Of-Trust, etc.
-n
This tells curl to use that ~/.netrc file I created.
-c ursCookies
This tells curl to stash cookies in the file ursCookies
-b ursCookies
This tells curl to read cookies from the file ursCookies
-L
This tells curl to follow redirects, which is a must for any OAuth2 flow.
--url https://54.172.97.47/opendap
The desired URL, protected by the URS OAuth2 authentication flow.

In order to retrieve multiple URLs with out reauthenticating you can use multiple instances of the --url parameter:

curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap --url https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds --url https://54.172.97.47/opendap/data/nc/coads_climatology.nc.dds

Or, since curl is actually pretty smart about using cookies and such you can also make multiple curl requests with the same cookies and it won't have to reauthenticate with URS once it's authenticated the first time:

curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap
curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
curl -k -n -c ursCookies -b ursCookies -L --url https://54.172.97.47/opendap/data/nc/coads_climatology.nc.dds

LDAP

Shibboleth

wget

URS

The wget documentation indicates that wget understands to use the .netrc file that we created for curl, and happily it appears to work, as long as other things are in place. Consider this wget command:

wget  --load-cookies cookies --save-cookies cookies --keep-session-cookie --no-check-certificate https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds

What's happening here?

--load-cookies cookies
Load cookies from the file "cookies"
--save-cookies cookies
Save cookies to the file "cookies"
--keep-session-cookie
Save session cookies.
--no-check-certificate
Do not check the authenticity of the (self signed) certificates. This is good for testing against your own servers running with self-signed certificates in that this switch will allow you to experience success when interacting with such servers. However, this switch breaks the chain of trust and may allow bad things to happen if used on the open internets. THus, for regular use, do not include this switch!
https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
The URL to retrieve.


Here's the output of said wget request:

[spooky:olfs/testsuite/urs] ndp% wget  --load-cookies cookies --save-cookies cookies --keep-session-cookie --no-check-certificate https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
--2014-11-14 11:22:18--  https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
Connecting to 54.172.97.47:443... connected.
WARNING: cannot verify 54.172.97.47's certificate, issued by `/C=US/ST=RI/L=Narragansett/O=OPeNDAP Inc./OU=Engineering/CN=54.172.97.47/emailAddress=support@opendap.org':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 302 Found
Location: https://uat.urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=04xHKVaNdYNzCBG6KB7-Ig&response_type=code&redirect_uri=https%3A%2F%2F54.172.97.47%2Fopendap%2Flogin&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw [following]
--2014-11-14 11:22:19--  https://uat.urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=04xHKVaNdYNzCBG6KB7-Ig&response_type=code&redirect_uri=https%3A%2F%2F54.172.97.47%2Fopendap%2Flogin&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw
Resolving uat.urs.earthdata.nasa.gov... 198.118.243.34, 2001:4d0:241a:4089::91
Connecting to uat.urs.earthdata.nasa.gov|198.118.243.34|:443... connected.
WARNING: certificate common name `uat.earthdata.nasa.gov' doesn't match requested host name `uat.urs.earthdata.nasa.gov'.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to uat.urs.earthdata.nasa.gov|198.118.243.34|:443... connected.
WARNING: certificate common name `uat.earthdata.nasa.gov' doesn't match requested host name `uat.urs.earthdata.nasa.gov'.
HTTP request sent, awaiting response... 302 Found
Location: https://54.172.97.47/opendap/login?code=a590cfc189783e29a7b8ab3ce1e0357618cbab3f590e7268a26e7ad1f7cf899d&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw [following]
--2014-11-14 11:22:20--  https://54.172.97.47/opendap/login?code=a590cfc189783e29a7b8ab3ce1e0357618cbab3f590e7268a26e7ad1f7cf899d&state=aHR0cHM6Ly81NC4xNzIuOTcuNDcvb3BlbmRhcC9kYXRhL25jL2Zub2MxLm5jLmRkcw
Connecting to 54.172.97.47:443... connected.
WARNING: cannot verify 54.172.97.47's certificate, issued by `/C=US/ST=RI/L=Narragansett/O=OPeNDAP Inc./OU=Engineering/CN=54.172.97.47/emailAddress=support@opendap.org':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 302 Found
Location: https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds [following]
--2014-11-14 11:22:21--  https://54.172.97.47/opendap/data/nc/fnoc1.nc.dds
Connecting to 54.172.97.47:443... connected.
WARNING: cannot verify 54.172.97.47's certificate, issued by `/C=US/ST=RI/L=Narragansett/O=OPeNDAP Inc./OU=Engineering/CN=54.172.97.47/emailAddress=support@opendap.org':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: `fnoc1.nc.dds'

    [ <=>                                                                                                                                                                                                            ] 197         --.-K/s   in 0s     

2014-11-14 11:22:22 (7.23 MB/s) - `fnoc1.nc.dds' saved [197]

[spooky:olfs/testsuite/urs] ndp% more fnoc1.nc.dds
Dataset {
    Int16 u[time_a = 16][lat = 17][lon = 21];
    Int16 v[time_a = 16][lat = 17][lon = 21];
    Float32 lat[lat = 17];
    Float32 lon[lon = 21];
    Float32 time[time = 16];
} fnoc1.nc;

It appears that wget followed the first redirect to uat.urs.earthdata.nasa.gov, where the URS server responded with a "401 Unauthorized" (thanks to the the app_type=401 query parameter in the redirect URL provided by mod_auth_urs). After getting the 401 wget resubmits the request with the authentication credentials and the URS server accepts them and redirects wget back to the mod_auth_urs server to complete the request.

LDAP

Shibboleth

ncdump

ncdump utilizes the NetCDF-C library (which uses the the "oc" library, which uses lib_curl under the hood) to access DAP resources so in theory if it works for ncdump then it should work for other clients that use the NetCDF-C library.

URS

The following works with the ncdump (and oc client) code bundled with NetCDF-4.3.2 Previous versions including 4.3.1 will not work.

Edit (create as needed) the file .dodsrc in your current working directory and set its file permissions to read only for owner:

[spooky:~] ndp% touch .dodsrc
[spooky:~] ndp% chmod 600  .dodsrc
[spooky:~] ndp% ls -l . dodsrc
-rw-------@ 1 ndp  staff  92 Nov 13 06:08 . dodsrc

Add your URS credentials and associate them with the URS authenticated DAP service that you wish to access. This will not work if you (correctly) associate the URS credentials with the URS server (go figure):

[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************


Here is a typical .dodsrc file.

# OPeNDAP client configuation file. See the OPeNDAP
# users guide for information.
USE_CACHE=0
# Cache and object size are given in megabytes (20 ==> 20Mb).
MAX_CACHE_SIZE=20
MAX_CACHED_OBJ=5
IGNORE_EXPIRES=0
CACHE_ROOT=/Users/ndp/.dods_cache/
DEFAULT_EXPIRES=1
ALWAYS_VALIDATE=1
# Request servers compress responses if possible?
# 1 (yes) or 0 (false).
DEFLATE=0
# Proxy configuration:
# PROXY_SERVER=<protocol>,<[username:password@]host[:port]>
# NO_PROXY_FOR=<protocol>,<host|domain>
# AIS_DATABASE=<file or url>
[https://target.dap.server.org]HTTP.CREDENTIALS.USER=your_uid
[https://target.dap.server.org]HTTP.CREDENTIALS.PASSWORD=**************

This version of ncdump (the oc client) appears to ignore the .dodsrc HTTP.COOKIEJAR and HTTP.COOKIEFILE parameters. This means that ncdump will authenticate each time it is invoked.

LDAP

Shibboleth

Integrated Data Viewer (IDV)

The Integrated Data Viewer is GUI driven data client that is based around the CDM/NetCDF data model and utilizes that NetCDF-Java (and thus the Java DAP implementation) to access remote DAP datasets. Because it has a GUI it can retrieve (and cache for later) users credentials directly from the user. Since IDV utilizes the Java-NetCDF library to access DAP resources then in theory if it works for IDV then it should work for all the other clients that use the Java-NetCDF library.

URS

For testing I utilized my AWS test service, configured to require URS authentication for all access of Hyrax.

I downloaded the latest version of IDV (5.0u2 on 11/19/14) and installed it on my local system. I launched IDV and then attempted to choose a new dataset by starting with the "Data" menu: Data > Choose Data > From A Web Server

In the resulting pane I entered the AWS test service URL for our friend coads_climatology.nc:

https://54.172.97.47/opendap/data/nc/coads_climatology.nc

When I committed the edit (aka hit Enter) IDV popped up a dialog box that indicated that the uat.urs.earthdata.nasa.gov server wanted my credentials:

IDV Authentication Dialog

I entered them, clicked the save password check box, and clicked the OK button. IDV was then able to access the requested resource. After the first successful access other resources at the AWS server were also available, but without an additional authentication challenge being presented to the user.

LDAP

Shibboleth

ToolsUI

The ToolsUI application is a simple is GUI driven data client that is based around the CDM/NetCDF data model and utilizes that NetCDF-Java (and thus the Java DAP implementation) to access remote DAP datasets. Because it has a GUI it can retrieve (and cache for later) users credentials directly from the user.

URS

For testing I utilized my AWS test service, configured to require URS authentication for all access of Hyrax.

I downloaded the latest version of ToolsUI (4.5 on 11/19/14) and installed it on my local system. I launched ToolsUI using the command line:

java -Xmx1g -jar toolsUI-4.5.jar

I selected the Viewer tab, and entered the AWS test service URL for our friend coads_climatology.nc:

https://54.172.97.47/opendap/data/nc/coads_climatology.nc

When I committed the edit (aka hit Enter) ToolsUI popped up a dialog box that indicated that the uat.urs.earthdata.nasa.gov server wanted my credentials.

ToolsUI Authentication Dialog

I entered them, clicked the save password check box, and clicked the OK button. ToolsUI was then able to access the requested resource.

LDAP

Shibboleth

Panoply

The Panoply application is a sophisticated GUI driven data client that is based around the CDM/NetCDF data model and utilizes that NetCDF-Java (and thus the Java DAP implementation) to access remote DAP datasets. Because it has a GUI it can retrieve (and cache for later) users credentials directly from the user.

URS

For testing I utilized my AWS test service, configured to require URS authentication for all access of Hyrax.

I downloaded the latest version of Panoply (4.0.5 on 11/20/14) and installed it on my local system. I launched Panoply (clicking it's icon in my Applications folder)

From the File menu, I selected "Open Remote Dataset.." and in the pop dialog I entered the URL for our friend coads_climatology.nc:

https://54.172.97.47/opendap/data/nc/coads_climatology.nc

When I committed the edit (aka hit Enter) Panoply popped up a dialog box that indicated that the uat.urs.earthdata.nasa.gov server wanted my credentials.

Panoply Authentication Dialog

I entered them, clicked the save password check box, and clicked the OK button. ToolsUI was then able to access the requested resource.

LDAP

Shibboleth