OPeNDAP Developer's Workshop 2010

Where & When

• Location: Troy, NY
• Dates: 6-8 Oct

Meeting modus operandi

• mockups
• whiteboard designs
• use cases
• best to exploit face-to-face for what it affords.

Agenda

---

Wednesday

0830 - Coffee

0900 -

Server Side Functions

• Registration
  • modularized
  • each function -> class
  • version
    • can you handle this
    • service
    • how do we package this
• Discovery
  • Ask the OLFS/BES what functions for dataset
    • what functions
    • capabilities
• Format
  • function(p1,p2,p3,p4...)
• Usage
  • Run by handler during loading - slection
  • Run during constraint parsing
  • Run during transmission/serialization (constraint evaluation)
  • What about functions like "version"??

void
function_version(int, BaseType *[], DDS &, BaseType **btpp)

typedef void(*bool_func)(int argc, BaseType *argv[], DDS &dds, bool *result);
typedef void(*btp_func)(int argc, BaseType *argv[], DDS &dds, BaseType **btpp);
typedef void(*proj_func)(int argc, BaseType *argv[], DDS &dds, ConstraintEvaluator &ce);

Use Cases

• Scientist has a dataset and wants to discover what functions are available to apply to that dataset
• Data provider has a set of server-side functions that they wish to provide for data users. They want to add these modules to the OPeNDAP via dynamically loaded modules
• Scientist has a dataset and wants to apply geogrid function to that dataset
• Scientist wants to discover the the shape and size (metadata) of a particular function against a dataset

Plan: Define a class/hierarchy

• Define a set of classes for server-saide functions
• Have the BES instantiate those from .so files
• Those instances are then passed into a ConstraintEvaluator instance which is then passed to a handler
• The BES conf file(s) can include information so that some functions are always loaded and some are loaded only for specific handlers (e.g., FreeForm).

• Take the server side functions out of libdap and add to BES modules
• Add the server side functions to server side class
• These classes will be able to handle things like get version, given a ddx(?) can this function be run against it (return true/false)
• At load time the classes are created within the BES, the functions are registered with libdap. This maintains binary compatability.

---

Server Administration

Admin interface

• admin connects to a listener
• can restart a listener
• can restart the BES with ADMIN
• turn on/off debugging through admin interface
• keep track of how many connections are allowed/active
• re-write besdaemon/beslistener (2-3 hrs)
• allowed size of response from BES (1 hr)

http://docs.opendap.org/index.php/Hyrax_Admin_Interface

BES design changes

• drop the besdaemon from the architecture
• listener is started instead of daemon
• listener keeps track of connections made and socket connected to
• listener keeps track of connections dropped, removes from list
• limit the number of open connections
• besdaemon will be able to have both hard and graceful restart
• Out of band comm will enable access to logs and conf files in addition to the restarts
• capability to send back configuration information
• capability to receive new configuration information and write to disk and reload
• we don't sweat the fact that the logs are shared - we can build a fancy interface out at the servlet level to filter by specific client using the PID info in the log.
• Changes in the conf only are used when a process is started.
• Soft shutdown, listener sends certain signal to child, when done handling current request, go down
• Hard shutdown, listener sends certain signal to child, go down now
• capability to stream back the log file

Authentication and Authorization

• Will people be authenticating through ESG to get to Hyrax? Or will Hyrax need to have an authentication piece? Authorization to ESG AuthZ.
• encrypting the data? Is it through the BES or through the middle tier? If middle tier then no need to encrypt the information through the BES.
• client application to need to allow login to get data? Scientist gets a URL back to access data through OPeNDAP. To get that data, have to be logged in? Authorization needs to happen.
• Certificate is on the client disk, when request made within matlab client, grabs certificate and sends along with data URL

Throttling the response

• Function added to DDS and base types to be able to pre-compute the response size based on projections and selections
• Should also work for sequences, at least be able to say that you can have x number of rows returned.

---

Lunch

James	:	Pizza Margarita
Dan	:	Corned Beef
Patrick	:	Veggy Pizza
Michael	:	Chikin Sandwich
Nathan	:	Turkey Club Sandwich

---

Easier install for Hyrax

1. Modify nightly build to use shrew
2. Build RPMs from the NB shrew
   1. Add dependencies to all RPM spec files
3. Modify all Makefiles to support pkg/dmg binaries
4. Integrate into NB on OS/X Server VMs
5. build meta package for OS/X using above
6. Run RPM/Linux NB on VMs too
7. Test both PRMs and pkg/dmgs on completely clean VMs

SQL Handler

1. Add to README and INSTALL; fill these in and make them standard WRT the other README/INSTALL files.
2. Add to the README so that there is a simple 'How To' for the server's configutation and then write stuff up for the docs wiki.
3. Look at SQL Handler requirements in docs wiki and match up to current developed functionality
4. Getting attributes

JGOFS

• Still a lot of organizations using jgofs and wanting opendap access
• Current library is very difficult to work with and still fork/exec the methods
• Could take the library and convert to use the autotools
  • Also change from fork and exec to dynamically loaded module
• Or be a pass-thru/gateway/proxy module, like WCS, and just pass the request back to the proper JGOFS server that is already running.
  • Figure out how to return catalog information from the jgofs servers
  • How do we get attributes/metadata...

Active file system

A filesystem that creates a "signal" whenever something changes.

In many cases this boils down to caching binary objects in/around the BES

• One way to do this is to use ODB (http://www.codesynthesis.com/products/odb/)
• Another way is to use the boost library (http://www.boost.org/)

---

---

Thursday

0830 - Coffee

Mmmmmm that was tasty...

1230 - Lunch

1300 -

Strategy Breakout

• Peter, Dan, James."

NcML Ingestion Break Out

• Patrick, Micheal, Nathan
• Instead of reading NCML file, taking an NCML document (1 hr)
• Issues with adding new BES commands? While adding this new functionality document current interface for adding commands to the BES architecture.

---

Friday

Module check

Making sure a loaded module matches the version of BES and any required modules

SSL authentication/authorization issues

Currently problem with SSL authentication and keeping SSL channel open for secure communication

Multiple catalogs

BES and OLFS integration