OPULS Development

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽

OPULS Process

Design Proposal Process

Short version: Each proposal is in one of the following categories:

  • New;
  • Under Consideration;
  • Accepted;
  • Declined;
  • Under Revision; or
  • Obsolete

DAP4 Specification

This section is deprecated in favor of the top-level DAP4 Specification page.


The draft specification has two volumes.

  1. Overview: New Features Introduced in DAP4
  2. Volume 1: Data Model, Persistent Representation, and Constraints
  3. Volume 2: Web Services Specification
  4. Volume 3: DAP4 Extensions
  5. DAP4 Extensions

DAP Use Cases

As part of the DAP2 --> DAP4 transition process, we are soliciting use cases regarding DAP client application development from the community. If you'd like to submit a use case, you can send it to any of the OPULS developers (e.g., James Gallagher) or the OPeNDAP technical discussion list. If you'd like you use case to be anonymous, that's fine, just let us know.

We will use these use cases to spot likely problems that may be encountered by members of the community who have developed specific client applications that interact with DAP servers.

Use cases from GSFC

These use cases were submitted by Chris Lynnes:

Use Case #1
Spatial/Variable Subsetter. Our Simple Subset Wizard (SSW) works by constructing URLs to download spatial / variable subsets from the server as netCDF files (http://disc.gsfc.nasa.gov/SSW). This has allowed us to stop writing custom subsetter programs for each dataset or group of datasets.
Use Case #2
Giovanni (http://giovanni.gsfc.nasa.gov/giovanni/) provides data exploration capabilities for datasets from GES DISC as well as a few select other providers. Data are acquired from servers via OPeNDAP, saved as netCDF/CF1, which is the Giovanni internal standard. This allows the deployment of many netCDF-capable tools for analysis and data manipulation purposes.
Use Case #3
MapServer provides WMS services for GES DISC data. Rather than site individual instances on each data server, we use the .vrt (Virtual files) capability in MapServer to set up the OPeNDAP connections needed to acquire data for the MapServer.
Use Case #4
We are currently experimenting with the NcML aggregation capability to see if it is feasible to offer the ability to generate time series at a point using just Hyrax and the NcML handler.

DAP4 Features Straw man

This is a straw man design, broken down into sections. It is incomplete and its content is subject to revision. This was our foil; don't use it. The two documents about under 'DAP Specification' are the current state of the DAP4 specification.

  1. Versions
  2. Checksum
  3. DAP Service Terminus
  4. Data Model and a XML Schema that provides one representation for the data model.
  5. Requests
  6. Responses
  7. Web Services (Starting Point)

Proposals

Data model

DAP4 Data Model Proposals
Proposal Status Author Date Proposed
Grids Ideas Accepted Dennis, et al. (Cleaned up 4/10/2012)
XML Use in DAP4 Accepted Dennis 2/25/2012
Top Level Group Accepted Dennis 4/12/2012
Proposal for Supporting Keys in Sequences Revised Dennis 5/28/2012
Constraints and Shared Dimensions Under Consideration, closely related to Grids Ideas James & Dennis Modified 04/12/2012
Ideas for Defining VLens (and Sequences) Under Consideration Dennis 02/25/2012
Path Specification in DAP4 New Dennis 3/30/2012
Coordinate Systems Element New Dennis 4/10/2012
Nested Attributes New Dennis 4/12/2012
Subsetting Arrays and Grids By Value New Nathan 4/27/2012
Proposal for a Constraint Notation New Dennis 4/30/2012
Proposal for an Abstract Constraint Specification New James, ed. 5/22/2012
Proposal for Structure Projection New Dennis 5/1/2012
Old Version of Grids Proposal Obsolete Dennis 4/10/2012
Filter Constraints new Dennis, John, Ethan 9/17/2012
Constraint Expressions new Everyone; original note by Dennis, ed. by James 10/29/2012
DAP4 Projection Syntax withdrawn Dennis 2/8/2013
Proposed Changes to Checksumming accepted with modifications 3/19/2013 Dennis 3/1/2013
Proposal for new VLEN representation New Dennis 7/21/2013
Proposal for a Comprehensive Constraint Expression Syntax New James, Dennis, et al. 7/29/2013
Alternate Proposal for a Constraint Expression Syntax New Dennis 10/16/2013
Type 1 Server Side Functions Proposal New Dennis 1/4/16

Responses Specific to DAP4

Persistent Representation
Proposal Status Author Date Proposed
Error Response new Nathan 11/19/2013
Proposal to use Multi-part Mime new 3/19/2013 Dennis 3/1/2013
DAP4: Inclusion of response metadata in the DMR New James 7/23/2013
DAP4 DDX Grammar Under Consideration Dennis




DAP4: DDX Lexical Elements Accepted Dennis 2/28/2012
Character Escape Mechanisms in DAP4 Accepted Dennis 4/5/2012
DAP4: Encoding for the Data Response Accepted James, based on Proposed DAP4 On-The-Wire Format by Dennis 6/5/2012, updated 8/31/12
DAP4: Chunked encoding Accepted James 6/8/2012, updated 8/31/12




Proposed DAP4 On-The-Wire Format Obsolete Dennis 4/8/2012
Proposed Multipart Mime Format Obsolete Dennis 4/8/2012
Proposed Error Response Format Obsolete Dennis 5/8/2012
Proposal to Replace Chunking Obsolete Dennis 8/23/2012

Web & HTTP

Web Service Proposals
Proposal Status Author Date Proposed
DAP4 Constraint expressions and query strings. Under Consideration Nathan Potter 09/24/2012
Possible Notation for Server Commands New Dennis Modified 04/12/2012




DAP4 Asynchronous Request-Response Proposal (new new version) Accepted Ethan Davis and Nathan Potter and James Gallagher 6/15/2012; updated 1 Aug 2012
DAP4 Web Services Proposal version 3 Volume 2: Web Services Specification Accepted Nathan Potter 6/18/2012
DAP4 Dataset Services Response Accepted Nathan Potter 6/19/2012




Asynchronous Responses Obsolete Nathan 4/5/2012
DAP4 Asynchronous Request-Response Proposal Obsolete EthanDavis 4/10/2012
DAP4 Capabilities and Versioning Obsolete EthanDavis 4/10/2012




Commentary on DAP4 Related Topics

These are not intended as specific proposal, but rather as commentary on some issues with DAP4 that need addressing with specific proposals.

  1. Characterization of URL Annotations (Modified: 2/26/2012)
  2. Constructing a DDX from a Query (new: 04/12/2012)
  3. Notes on Constraints (new: 04/29/2012)
  4. DAP4: An Essay on Domain Specific Models (new: 05/05/2012)

OPULS Implementation Thoughts

Unidata's Current Thoughts

Unidata needs:

 1) DAP4 C client library for use in netCDF-C library
 2) DAP4 Java client library for use in netCDF-Java library
 3) DAP4 Java server library for use in TDS
 4) Conformance test service against which a DAP4 client library can
    be tested
 5) Conformance test service against which a DAP4 server can be
    tested

The current CDM and TDS use of opendap-java library has performance problems due to extra API layers and extra copying of data. We would prefer direct serialization between CDM objects and the DAP on-the-wire data and would recommend that the OPeNDAP Java library leverage the netCDF-Java libraries.

If there are needs that the DAP4 opendap-java library be independent of the netCDF-Java library, then in order for CDM and TDS to use the opendap-java library, we would need the performance concerns mentioned above addressed. A reasonable metric would be within some percentage of cdmremote[1], say within 20-25%.

Unidata's performance requirements for Java libraries:

 1) The netCDF-Java library performance reading DAP4 through the DAP4
    Java client library must perform within 20-25% of cdmremote [1].
 2) TDS performance when handling DAP4 requests using the DAP4 Java
    server library should perform within 20-25% of cdmremote [1].

We believe this means that the DAP4 Java libraries must provide a sufficiently generic API that can be used directly with the CDM/TDS IOSP interface.

Unidata will:

 1) Write the DAP4 C client library.
 2) ??? some part of Java server library
 3) ??? some parts of conformance testing services

NOTES:

[1] cdmremote is an experimental CDM streaming service. It is basically a direct serialization of the CDM.

OPULS references

Old DAP4 Design and Implementation

  • DAP3/4
  • DAP 4.0 Design and DAP 4.0 Essential Features The design is meant to be a complete document while essential features are the minimal see we want to release. Since this task has been stalled for nearly a year, doing the latter seems like a noble goal.