DAP Service Terminus: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
Line 119: Line 119:


=== [[DAP4: ASCII Data Service]]===
=== [[DAP4: ASCII Data Service]]===
This service provides data responses in ASCII format. When invoked the regular DAP data response will be repackaged as an ASCII representation of the data values.





Revision as of 17:55, 30 March 2012

Overview

We intend to provide a 'Services' or 'Capabilities' response for DAP4.

Currently DAP servers provide a number of dataset level responses (services). These include DAP services such as the DDS, DAS, and DDX, plus other services that have been added over time such as the RDF and ISO responses. There is currently no way for a user (or software agent) to discover which of these many services might be available for a particular dataset. While THREDDS catalogs do provide a mechanism for determining which services are associated with which datasets, they don't actually provide a mechanism for discovering the details of the various service URLs. A THREDDS catalog might identify a DAP service, but the URL that can be assembled from the THREDDS catalog for that service is limited to just the base dataset URL. Typically dereferencing this URL will provide one of two things - either the underlying data file (for example a netcdf file) or an HTTP 403 error if direct access to the underlying files has been disabled in the server configuration.

Example:

This URL points to a dataset on a DAP server:

http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc

Dereferencing the link will not connect you to a DAP service or DAP response. It will only return the underlying netcdf file. The only way to know that DAP services exist for this is by reading the URL path and recognizing that it is likely an instance of a DAP server based on the position of the 'opendap' string in the URL.

There is no guarantee that dereferencing this URL will even yield the underlying file, as that behavior is a configuration option for the server. If that access is not allowed in the configuration, then dereferencing the URL will simply return an HTTP 403 (forbidden) error.

Possible Technologies

I looked at Web Services Description Language (WSDL), Web Application Description Language (WADL), and OGC GetCapabilities as possible syntaxes to leverage. All three make assumptions about the structure of the URL and constraint expression (aka query string) that would require an extension to an existing standard be written to support our existing syntax.

Proposal

I propose that base DAP URLs should return an XML document describing the DAP services available for the dataset.

At minimum this document should provide:

  • The name of the service (DDX, DDS, etc.)
  • A link that can be dereferenced to get the service response for the dataset in question.
  • A brief description of the service
  • A link to a complete, human readable, description of the service response and it's semantics
  • A reference to an XSLT that a browser would use to render the description into HTML for a presentation view.

Jimg 12:47, 3 February 2012 (PST) This proposed feature is an example of agent-driven content negotiation. To wit:

With agent-driven negotiation, selection of the best representation for a response is performed by the user agent after receiving an initial response from the origin server. Selection is based on a list of the available representations of the response included within the header fields or body of the initial response, with each representation identified by its own URI.

Additionally this dataset services description document might contain:

  • Descriptions and syntax of server side functions available for the dataset

Static or Dynamic Response?

As the DAP servers evolve to support user authentication and sessions the content of the dataset services description document should become dynamic. By this I mean that some services and server functions might be available only to certain users. If an authenticated user asks for a particular dataset services description document they would see all the the services available to them, which might differ significantly from those available to a different user. As powerful server side functions such as re-gridding and re-projection become available we anticipate that data providers may not wish to allow just anyone to utilize them because of the potential burden they may place on the data center's computing resources.

It may be that some services may not be available for all datasets. This argues (along with the previous comment about users and roles) that the service description needs to be generated dynamically and that it should be context sensitive w.r.t. user, role, and dataset.

Implementation of a dynamic services response implies that the software generating that response can get information about what is applicable for a given user and a given dataset, where the latter could be quite complicated. The alternative is to have all possible responses described in the services description with the understanding that they might return an error code under one set of circumstances. This would be far easier for servers, and likely require no additional logic in clients (because they will have to account that unforeseen errors will happen in any case). The opposing argument is that presenting information known to be false is a poor practice - and it's hard not to agree.

I suggest that we split this into two cases:

Authenticated/privileged versus anonymous users
For this case, the response should be dynamic and reflect what the user can actually do.
Datasets for which server-side functions are/are-not applicable
For this case, however, the response should not attempt to determine what is applicable to a particular dataset, but that it optionally can indicate that a particular function can, with certainty, be applied.

These are two fundamentally different cases because the semantics or the first are under our control, while those of the latter are not. In the first cases, the server defines the classes of users and the sets that describe their capabilities. It is possible for the server to know that a given user can never run a regridding operation, for example, because they lack the authorization to do so. However, for datasets, determining which functions are applicable, for example, requires understanding the semantics of the metadata bound to that dataset which can be done in some cases, but not all.

How this limits the response's utility, plus a solution

The problem with not providing a list of functions that is known to be applicable to a dataset, is that limits an important source of semantic information about those datasets. If we know that a regridding function can be allied to dataset X, then we know a fair amount that dataset. In addition, it is possible that the service response could include, at some future time, either directly or via inference, semantic information based on server-side operations that were known to be possible.

If we extend the Function element of the service response so that the element listing a function contains an attribute that indicates that the server knows that this function can definitely be used with the dataset, then we can support semantic systems without forcing the server to know about the applicability of each function to each dataset. I would suggest that we add the attribute applicability and support the three values of yes, no, and unknown with the latter being the default value.

Services

A Service is simply a name, a URL, and an optional description. A service is accessed by dereferencing its URL, which is typically constructed by adding some type of suffix to the dataset's referent (aka base) URL. The way in which the query string (aka constraint expression) is used is defined by each service, and there is no requirement for inter-service query string API conformity.

DAP4: Dataset Services Service

Returns a listing of the services available for a particular dataset. Services accept requests in the form of a URL. They may accept query strings (Constraint Expressions). They respond with some type of specific response object. This may include, but is not limited to, any combination of the services that follow.


service url = dataset_url

DAP4: Dataset Service

The Dataset Service provides a metadata description for a dataset. The Dataset response is an XML document that contains both the 'syntactic' (structural) and 'semantic' metadata for the dataset The dataset service accepts a query string (constraint expression) that allows you to inspect the effects on the data structures when sub-setting and/or server side functions are applied.


suffix = .xml
service url = dataset_url + .xml

DAP4: Data Service

The Data Service provides DAP4 data access to a dataset. The Data response is a multipart MIME document that contains a N+1 parts for a response with N variables. The Data service accepts a query string (constraint expression) which allows you to subset the data and invoke server side functions.


suffix = .dap
service url = dataset_url + .dap

DAP4: HTML DATA Request Form Service

The HTML DATA Request Form Service provides browser based access to the Dataset. When invoked it returns a web-browser renderable document (in html) that provides a form (or other UI) that can be used to constrain and request data in accordance with the DAP4 specification as applied to the dataset .


suffix = .html
service url = dataset_url + .html

DAP4: RDF Service

The RDF service provides an RDF representation of the Dataset document (DDX). The RDF response is an XML document containing an RDF version of the DAP4: Dataset Response.


suffix = .rdf
service url = dataset_url + .rdf

DAP4: ISO 19115 Service

This service provides ISO 19115 metadata for the Dataset, if any can be found. When invoked it returns an XML document containing ISO 19115 metadata located in the DAP4: Dataset Response.


suffix = .iso
service url = dataset_url + .iso

DAP4: ISO Conformance Rubric Service

This service provides a browser renderable document that describes how well the metadata held in the Dataset conforms to ISO 19115. When invoked this service returns a browser renderable document that describes how well the metadata held in the Dataset Response conforms to ISO 19115.


suffix = .rubric
service url = dataset_url + .rubric

DAP4: NetCDF File-out Service

This service provides data responses in NetCDF-3 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 3 file.

suffix = .nc
service url = dataset_url + .nc

DAP4: ASCII Data Service

This service provides data responses in ASCII format. When invoked the regular DAP data response will be repackaged as an ASCII representation of the data values.


suffix = .ascii
service url = dataset_url + .ascii

DAP4: Native File Access Service

This service provides direct access to the data source file (or whatever else) that is creating the DAP dataset resource. When invoked it returns the "native" data from whatever store (filesystem, etc.) it may be in.


suffix = .file
service url = dataset_url + .file

DAP4: Server Version Service

This service provides software versioning information. When invoked the services returns an XML file containing a description of the version of the server and it's components.


suffix = .rdf
service url = dataset_url + .rdf


DAP2: Data Service

The DAP2 data service provides DAP2 data access to the data resource.


suffix = .dods
service url = dataset_url + .dods

DAP2: DDX Service

The DAP2 DDX service provides DAP2 access to the data resource metadata. When invoked the service returns an XML document containing both syntactic and semantic dataset metadata in DAP2 XML format.

suffix = .ddx
service url = dataset_url + .ddx

DAP2: DDS Service

The DAP2 DDS service provides access to the 'syntactic' metadata (aka use or structural metadata) for the data resource. When invoked returns a DAP2 DDS response document conforming to the DDS part of the DAP2 specification.


suffix = .dds
service url = dataset_url + .dds

DAP2: DAS Service

The DAP2 DAS service provides access to the 'semantic' metadata (aka domain metadata) for the data resource. When invoked returns a DAP2 DAS response document conforming to the DAS part of the DAP2 specification.


suffix = .das
service url = dataset_url + .das

DAP2: Info Service

The DAP2 INFO service provides a browser renderable page that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way. When invoked this service returns a web browser renderable document that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way.


suffix = .info
service url = dataset_url + .info


Template: ServiceTemplate

Prototype #1

I implemented a prototype Service Description response document for Hyrax. It is not dynamic - all datasets and all users get the same basic response (delta the embedded URL's that return the various service responses).

It looks like this:

 <?xml version="1.0" encoding="UTF-8"?>
 <?xml-stylesheet type="text/xsl" href="/opendap/xsl/serviceDescription.xsl"?>
 <DatasetServices xml:base="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc">

  <Service name="HTML Data Request Form" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.html" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/dataRequestForm.html" xlink:type="simple">OPeNDAP HTML Data Request Form for data constraints and access</Description>
  </Service>
  <Service name="DAP4 Data" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.dap" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/dap4_data.html" xlink:type="simple">DAP4 Data Object</Description>
  </Service>
  <Service name="DAP2 Data" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.dods" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/dap2_data.html" xlink:type="simple">DAP2 Data Object</Description>
  </Service>
  <Service name="DDX" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.ddx" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/ddx.html" xlink:type="simple">OPeNDAP Data Description and Attribute XML Document</Description>
  </Service>
  <Service name="DDS" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.dds" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/dds.html" xlink:type="simple">OPeNDAP Dataset Description Structure</Description>
  </Service>
  <Service name="DAS" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.das" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/das.html" xlink:type="simple">OPeNDAP Dataset Attribute Structure (DAS)</Description>
  </Service>
  <Service name="INFO" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.info" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/info.html" xlink:type="simple">OPeNDAP Dataset Information Page</Description>
  </Service>
  <Service name="RDF" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.rdf" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/rdf.html" xlink:type="simple">An RDF representation of the DDX document.</Description>
  </Service>
  <Service name="NetCDF-File" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.nc" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/netcdf_fileout.html" xlink:type="simple">NetCDF file-out response.</Description>
  </Service>
  <Service name="ISO-19115" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.iso" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/iso_metedata.html" xlink:type="simple">ISO 19115 Metadata Representation of the DDX.</Description>
  </Service>
  <Service name="ISO-19115-Score" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.rubric" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/iso_rubric.html" xlink:type="simple">ISO 19115 Metadata Representation conformance score for this dataset.</Description>
  </Service>
  <Service name="FileAccess" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc.file" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/dataset_file_access.html" xlink:type="simple">Access to dataset file.</Description>
  </Service>
  <Service name="ServiceDescription" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://localhost:8080/opendap/hyrax/NWAtlanticDec_1km.nc" xlink:type="simple">
    <Description xlink:href="http://services.opendap.org/service_description.html" xlink:type="simple">Service Description response.</Description>
  </Service>

  <ServerSideFunctions>
    <Function name="geogrid" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#geogrid">
      <Description>Allows a DAP Grid variable to be sub-sampled using georeferenced values.</Description>
    </Function>
    <Function name="grid" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#grid">
      <Description>Allows a DAP Grid variable to be sub-sampled using the values of the coordinate axes.</Description>
    </Function>
    <Function name="linear_scale" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#linear_scale">
      <Description>Applies a linear scale transform to the named variable.</Description>
    </Function>
    <Function name="version" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#version">
      <Description>Returns version information for each server side function.</Description>
    </Function>
  </ServerSideFunctions>

</DatasetServices>


I also put together a simple XSLT to make this a human friendly HTML. I added a reference to it in the XML document. When you point a browser at the resource URL (For example http://localhost:8080/opendap/data/nc/fnoc1.nc ) the Service Description response is returned as an XML file (above), the browser detects the xml-stylesheet reference, grabs the XSLT, uses it to convert the XML to HTML and renders it like this:

ServiceDescriptionPrototype-01.png

Prototype #2

This prototype response is based on the OGC Web Services Common Standard (OWS) GetCapabilities response.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Capabilities xmlns="http://www.opengis.net/wcs/1.1" updateSequence="1288821006000">
    <!--
    
    
    Service Identification
    
    
    -->
    <ows:ServiceIdentification xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xlink="http://www.w3.org/1999/xlink">
        <ows:Title>OPeNDAP DAP Service</ows:Title>
        <ows:Abstract>A server to DAP access to datasets datasets</ows:Abstract>
        <ows:Keywords>
            <ows:Keyword>OPeNDAP</ows:Keyword>
            <ows:Keyword>nectcdf</ows:Keyword>
            <ows:Keyword>DAP</ows:Keyword>
        </ows:Keywords>
        <ows:ServiceType>DAP</ows:ServiceType>
        <ows:ServiceTypeVersion>4.0.0</ows:ServiceTypeVersion>
        <ows:Fees>NONE</ows:Fees>
        <ows:AccessConstraints>NONE</ows:AccessConstraints>
    </ows:ServiceIdentification>
    <ows:ServiceProvider xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xlink="http://www.w3.org/1999/xlink">
        <ows:ProviderName>OPeNDAP Inc.</ows:ProviderName>
        <ows:ProviderSite xlink:href="http://www.opendap.org"/>
        <ows:ServiceContact>
            <ows:IndividualName>Anonymous Smith</ows:IndividualName>
            <ows:PositionName>Sys. Admin.</ows:PositionName>
            <ows:ContactInfo>
                <ows:Phone>
                    <ows:Voice>123-456-7890</ows:Voice>
                    <ows:Facsimile>234-567-8901</ows:Facsimile>
                </ows:Phone>
                <ows:Address>
                    <ows:DeliveryPoint>165 Fictitious Dr.</ows:DeliveryPoint>
                    <ows:City>AnywhereButHere</ows:City>
                    <ows:AdministrativeArea>Rhode Island</ows:AdministrativeArea>
                    <ows:PostalCode>02882</ows:PostalCode>
                    <ows:Country>USA</ows:Country>
                    <ows:ElectronicMailAddress>support[a.t]opendap[d.o.t]org</ows:ElectronicMailAddress>
                </ows:Address>
                <ows:OnlineResource xlink:href="http://your.domain.org/"/>
                <ows:HoursOfService>24x7x365</ows:HoursOfService>
                <ows:ContactInstructions>email</ows:ContactInstructions>
            </ows:ContactInfo>
            <ows:Role>Developer</ows:Role>
        </ows:ServiceContact>
    </ows:ServiceProvider>
    <!--
    
    
    Operations Metadata
    
    
    -->
    <ows:OperationsMetadata xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xlink="http://www.w3.org/1999/xlink">
        <ows:Operation name="DAP4 Data">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.dap"/>
                </ows:HTTP>
            </ows:DCP>
            <ows:Parameter name="Format">
                <ows:AllowedValues>
                    <ows:Value>application/x-netcdf-cf1.0</ows:Value>
                    <ows:Value>application/x-dap-3.2</ows:Value>
                </ows:AllowedValues>
            </ows:Parameter>
        </ows:Operation>
        <ows:Operation name="DAP2 Data">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.dods"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetDDX">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.ddx"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetDDS">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.dds"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetDAS">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.das"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetHtmlDataRequestForm">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.html"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetInfoPage">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.info"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetRDF">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.rdf"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetNetcdfFile">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.nc"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetIso">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.iso"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetIsoRubric">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.rubric"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="DataFileAccess">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc.file"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
        <ows:Operation name="GetCapabilities">
            <ows:DCP>
                <ows:HTTP>
                    <ows:Get xlink:href="http://test.opendap.org:8090/opendap/data/nc/fnoc1.nc"/>
                </ows:HTTP>
            </ows:DCP>
        </ows:Operation>
    </ows:OperationsMetadata>
    <!--
    
    
    Contents 
        
    
    -->
    <Contents>
        <Dataset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xml.opendap.org/ns/DAP2" name="fnoc1.nc" xsi:schemaLocation="http://xml.opendap.org/ns/DAP2 http://xml.opendap.org/dap/dap2.xsd">
            <Attribute name="NC_GLOBAL" type="Container">
                <Attribute name="base_time" type="String">
                    <value>88- 10-00:00:00</value>
                </Attribute>
                <Attribute name="title" type="String">
                    <value> FNOC UV wind components from 1988- 10 to 1988- 13.</value>
                </Attribute>
            </Attribute>
            <Attribute name="DODS_EXTRA" type="Container">
                <Attribute name="Unlimited_Dimension" type="String">
                    <value>time_a</value>
                </Attribute>
            </Attribute>
            <Array name="u">
                <Attribute name="units" type="String">
                    <value>meter per second</value>
                </Attribute>
                <Attribute name="long_name" type="String">
                    <value>Vector wind eastward component</value>
                </Attribute>
                <Attribute name="missing_value" type="String">
                    <value>-32767</value>
                </Attribute>
                <Attribute name="scale_factor" type="String">
                    <value>0.005</value>
                </Attribute>
                <Attribute name="DODS_Name" type="String">
                    <value>UWind</value>
                </Attribute>
                <Attribute name="b" type="Byte">
                    <value>128</value>
                </Attribute>
                <Attribute name="i" type="Int32">
                    <value>32000</value>
                </Attribute>
                <Attribute name="WOA01" type="Url">
                    <value>"http://localhost/junk"</value>
                </Attribute>
                <Int16/>
                <dimension name="time_a" size="16"/>
                <dimension name="lat" size="17"/>
                <dimension name="lon" size="21"/>
            </Array>
            <Array name="v">
                <Attribute name="units" type="String">
                    <value>meter per second</value>
                </Attribute>
                <Attribute name="long_name" type="String">
                    <value>Vector wind northward component</value>
                </Attribute>
                <Attribute name="missing_value" type="String">
                    <value>-32767</value>
                </Attribute>
                <Attribute name="scale_factor" type="String">
                    <value>0.005</value>
                </Attribute>
                <Attribute name="DODS_Name" type="String">
                    <value>VWind</value>
                </Attribute>
                <Int16/>
                <dimension name="time_a" size="16"/>
                <dimension name="lat" size="17"/>
                <dimension name="lon" size="21"/>
            </Array>
            <Array name="lat">
                <Attribute name="units" type="String">
                    <value>degree North</value>
                </Attribute>
                <Float32/>
                <dimension name="lat" size="17"/>
            </Array>
            <Array name="lon">
                <Attribute name="units" type="String">
                    <value>degree East</value>
                </Attribute>
                <Float32/>
                <dimension name="lon" size="21"/>
            </Array>
            <Array name="time">
                <Attribute name="units" type="String">
                    <value>hours from base_time</value>
                </Attribute>
                <Float32/>
                <dimension name="time" size="16"/>
            </Array>
            <dataBLOB href=""/>
        </Dataset>
    </Contents>
</Capabilities>


The first section ows:ServiceIdentification preserves the intended semantics of GetCapabilities very well, as the section really has little to do with any particular web service (REST or otherwise) and everything to do with who to contact about the service instance. Populating the Contents section with the DDX object is similarly well mapped to the original intent: "The Contents section of a service metadata document normally contains metadata about the data served by this server" (OGC 06-121r9 section 7.4.8)

However, while this document does supply the necessary links to get to the various products associated with the DAP service it does so at the expense of changing the semantics of the components of the OperationsMetadata part of the response. As far as I can tell in OGC parlance each operation in this section would be bound to the request parameter in the constraint expression. Consider this DescribeCoverage request:

http://test.opendap.org:8090/WCS?service=WCS&version=1.1.2&request=DescribeCoverage&identifiers=S1/opendap/ioos/200803061600_HFRadar_USEGC_6km_rtv_SIO.ncml

Obviously this interpretation of the semantics of the ows:Operation elements doesn't fit with the intended semantics in our example.

Also, the OWS allows for the optional inclusion of any number of ows:Parameter and ows:Constraint elements that can be used with each operation. In the schema these parameter and constraint objects are both of type ows:DomainType. The DomainType contains a description of a domain using the following terms:

  • PossibleValues
    • AllowedValues - expressed as a list of allowed values and/or range(s) of allowed values.
    • AnyValue - Any value is allowed
    • NoValues - no values allowed
    • ValuesReference - Reference to externally specified list of all the valid values and/or ranges of values for this quantity.
  • Meaning - Some kind of annotation about what the "meaning" of this parameter/constraint expression is.
  • DataType (where data type is expressed as a an "xlink:href attribute that references a URN for a well-known data type",
  • ValuesUnit - The units of the values of this parameter/constraint expression expressed as an "xlink:href attribute that references a URN for a well-known unit of measure (uom).
  • Metadata - where metadata is defined as either links to metadata documents or simply as text embedded in ows:Metadata elements.

I am not at all clear the distinction is between the ows:Parameter and ows:Constraint. It is my thinking that each of them represent a possible term in the query part of a URL. For exampleL ?p=43&z=foobar could be seen as an ows:Parameter with name "p" and an ows:Constraint with name "z".


In DAP our constraint expressions (CEs) are fluid*: Every dataset has it's own set of left-operands (the names of the variables in the dataset) and valid parameters for server side functions (the names of the variables in the dataset).

The projection section of the DAP CE doesn't make use of an equals sign, but instead uses a comma separated list of these things:

  • some/all the names of the variables in the dataset which may be followed by array/sequence subset markup on some/all variables
  • server side functions which may take variable names and other explicit values as parameters. The acceptable range of these explicitly valued parameters may be a function of some variables range, or not.

to make the projection.

Since the names, types, and ranges of variables typically change from one dataset to another the acceptable values in the query part of the request URL is different for every dataset.

After reading over the schema and the supporting documentation at OGC I do not see how the syntax provided for parameters and constraints can be used to express the details of the DAP Constraint Expression.

Where does this leave us? While many developers are already familiar with the layout and content of the OWS GetCapabilities response, I think that the semantics of it's components don't lend themselves to describing the existing DAP REST service. We can make a valid document using the OGC components and schemas that contains the information that we want, but the meaning of the parts will fail to adhere to the meanings assigned in the OGC schemas and documentation. I fear this would probably cause more confusion, consternation, and gnashing of teeth than Prototype #1 where we simply created the minimum document needed to achieve our goal for providing a service description that itemizes all of the data products and services available for a particular dataset.

Jimg 09:29, 31 January 2012 (PST) I agree. GetCapabilities is not a good fit for this feature.

*Jimg 20:57, 14 February 2012 (PST) I started to write an explanation of the differences between our constraint expressions and the WCS query string, but I think the most important aspect is that the the two use a different syntax and the description of that series of tokens (some of which are a function of the service and some of which are a function of the data being served - in both cases ) is heavily tied to the syntax of the URL and it's Query String component.

Comments

Jimg 16:22, 14 November 2011 (PST) We should think about making this a JSON response, or think about having http://server/file.nc return the XML and http://server/file.nc.json return JSON using an XSLT transform. Jimg 16:22, 14 November 2011 (PST) Nathan and I both conclude that the XML response should include the full URL and not do some weird hack where URLs are 'built'.

While this might be specific to HTTP and the web, I think we should include in the design that a JSON version of this document will be returned if the client announces that it understands such thing (the default response will be XML with an embedded XSL stylesheet that provides a transform to HTML). The HTTP/1.1 request and response would look like:

GET / HTTP/1.1
Host: api.example.com
Accept: application/vnd.example.link_templates+json

HTTP/1.1 200 OK
Content-Type: application/vnd.example.link_templates+json
Cache-Control: max-age=3600
Connection: close

Where the request header Accept: is the key bit. See the post Ethan referenced for more information about this: Web API Versioning Smackdown

Issues

  • This a change in server behavior that will change the response content of file access URLs. These URLs are not part of the regular DAP URL pattern and their current behavior is a function of server configuration.
  • How might we describe the way to elicit DAP3.x and DAP4 versions of the responses in this description given that we currently rely on the XDAP-Accept HTTP header to inform the server of the version being requested? Jimg 16:18, 14 November 2011 (PST) We can define a new set of extensions for these responses, especially since there are only two (ddx4 and dap4, e.g.). There are several other syntaxes that put this information in the URL, but if we use unique URLs to reference these new responses, then we don't break REST. This would mean that the XDAP-Accept: header would be dropped, but that's a separate discussion.
  • Jimg 10:42, 26 January 2012 (PST) I am now leaning toward using .ddx and .dap to be the way that DAP4 metadata and data are accessed, period. This means dropping the idea of version negotiation, which I think is a poor design for a web service. the extensions .das, .dds and .dods woud return the tried and true DAP2 response objects and any server hat support the five extensions clearly support DAP2 and DAP4. No client will every ask for a DAP2 DDX and be surprised by getting DAP4 (which it won't be able to parse) because to do so would be nonsensical (and anti-causal, sort of...). The one hitch with this idea is that there are some DAP2++ clients out there that gobble up DDX responses...
  • I know we discussed additional issues, but I can't recall them. Please add them if you do!