OPeNDAP Developer's Workshop 2010: Difference between revisions
From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
Line 168: | Line 168: | ||
* One way to do this is to use ODB (http://www.codesynthesis.com/products/odb/) | * One way to do this is to use ODB (http://www.codesynthesis.com/products/odb/) | ||
* Another way is to use the boost library. | * Another way is to use the boost library (http://www.boost.org/) | ||
Revision as of 20:25, 6 October 2010
Where & When
- Location: Troy, NY
- Dates: 6-8 Oct
Meeting modus operandi
- mockups
- whiteboard designs
- use cases
- best to exploit face-to-face for what it affords.
Agenda
Wednesday
- 0830 - Coffee
- 0900 -
Server Side Functions
- Registration
- modularized
- each function -> class
- version
- can you handle this
- service
- how do we package this
- Discovery
- Ask the OLFS/BES what functions for dataset
- what functions
- capabilities
- Ask the OLFS/BES what functions for dataset
- Format
- function(p1,p2,p3,p4...)
- Usage
- Run by handler during loading - slection
- Run during constraint parsing
- Run during transmission/serialization (constraint evaluation)
- What about functions like "version"??
void function_version(int, BaseType *[], DDS &, BaseType **btpp) typedef void(*bool_func)(int argc, BaseType *argv[], DDS &dds, bool *result); typedef void(*btp_func)(int argc, BaseType *argv[], DDS &dds, BaseType **btpp); typedef void(*proj_func)(int argc, BaseType *argv[], DDS &dds, ConstraintEvaluator &ce);
Use Cases
- Scientist has a dataset and wants to discover what functions are available to apply to that dataset
- Data provider has a set of server-side functions that they wish to provide for data users. They want to add these modules to the OPeNDAP via dynamically loaded modules
- Scientist has a dataset and wants to apply geogrid function to that dataset
- Scientist wants to discover the the shape and size (metadata) of a particular function against a dataset
Plan: Define a class/hierarchy
- Define a set of classes for server-saide functions
- Have the BES instantiate those from .so files
- Those instances are then passed into a ConstraintEvaluator instance which is then passed to a handler
- The BES conf file(s) can include information so that some functions are always loaded and some are loaded only for specific handlers (e.g., FreeForm).
- Take the server side functions out of libdap and add to BES modules
- Add the server side functions to server side class
- These classes will be able to handle things like get version, given a ddx(?) can this function be run against it (return true/false)
- At load time the classes are created within the BES, the functions are registered with libdap. This maintains binary compatability.
Server Administration
Admin interface
- admin connects to a listener
- can restart a listener
- can restart the BES with ADMIN
- turn on/off debugging through admin interface
- keep track of how many connections are allowed/active
- re-write besdaemon/beslistener (2-3 hrs)
- allowed size of response from BES (1 hr)
BES design changes
- drop the besdaemon from the architecture
- listener is started instead of daemon
- listener keeps track of connections made and socket connected to
- listener keeps track of connections dropped, removes from list
- limit the number of open connections
- besdaemon will be able to have both hard and graceful restart
- Out of band comm will enable access to logs and conf files in addition to the restarts
- capability to send back configuration information
- capability to receive new configuration information and write to disk and reload
- we don't sweat the fact that the logs are shared - we can build a fancy interface out at the servlet level to filter by specific client using the PID info in the log.
- Changes in the conf only are used when a process is started.
- Soft shutdown, listener sends certain signal to child, when done handling current request, go down
- Hard shutdown, listener sends certain signal to child, go down now
- capability to stream back the log file
Authentication and Authorization
- Will people be authenticating through ESG to get to Hyrax? Or will Hyrax need to have an authentication piece? Authorization to ESG AuthZ.
- encrypting the data? Is it through the BES or through the middle tier? If middle tier then no need to encrypt the information through the BES.
- client application to need to allow login to get data? Scientist gets a URL back to access data through OPeNDAP. To get that data, have to be logged in? Authorization needs to happen.
- Certificate is on the client disk, when request made within matlab client, grabs certificate and sends along with data URL
Throttling the response
- Function added to DDS and base types to be able to pre-compute the response size based on projections and selections
- Should also work for sequences, at least be able to say that you can have x number of rows returned.
Lunch
James | : | Pizza |
Dan | : | Corned Beef |
Patrick | : | Veggy Pizza |
Michael | : | Chikin Sandwich |
Nathan | : | Turkey Club Sandwich |
Easier install for Hyrax
- Modify nightly build to use shrew
- Build RPMs from the NB shrew
- Add dependecies to all RPM spec files
- Modify all Makefiles to support pkg/dmg binaries
- Integrate into NB on OS/X Server VMs
- build meta package for OS/X using above
- Run RPM/Linux NB on VMs too
- Test both PRMs and pjg/dmgs on completely clean VMs
SQL Handler
- Modify build so that it is possible to select ODBC or MySQL or Postgres instead of the build requiring each.
- --withith-odbc
- --with-mysql
- --with-postgres
- Add to README and INSTALL
- Look at SQL Handler requirements in docs wiki and match up to current developed functionality
- Getting attributes
JGOFS
- Still a lot of organizations using jgofs and wanting opendap access
- Current library is very difficult to work with and still fork/exec the methods
- Could take the library and convert to use the autotools
- Also change from fork and exec to dynamically loaded module
- Or be a pass-thru/gateway/proxy module, like WCS, and just pass the request back to the proper JGOFS server that is already running.
- Figure out how to return catalog information from the jgofs servers
- How do we get attributes/metadata...
Active file system
A filesystem that creates a "signal" whenever something changes.
In many cases this boils down to caching binary objects in/around the BES
- One way to do this is to use ODB (http://www.codesynthesis.com/products/odb/)
- Another way is to use the boost library (http://www.boost.org/)
1800 - Dinner
Thursday
- 0830 - Coffee
- 0900 -
- NcML Ingestion Break Out
- BES Internal Caching
- 1230 - Lunch
- 1300 -
- 1400-1800
- Strategy Breakout
- Peter, Dan, James."
- 1400-1800
- NcML Ingestion Break Out
- Patrick, Micheal, Nathan
- Instead of reading NCML file, taking an NCML document (1 hr)
- Issues with adding new BES commands? While adding this new functionality document current interface for adding commands to the BES architecture.
- 1800 - Dinner
Friday
- 0830 - Coffee
- 0900 -
- Module check
- Making sure a loaded module matches the version of BES and any required modules
- SSL authentication/authorization issues
- Currently problem with SSL authentication and keeping SSL channel open for secure communication
- Multiple catalogs
- BES and OLFS integration
- 1230 - Lunch
- 1300 -
- 1400-1800
- Strategy Breakout (cont)
- 1800 - Dinner and Flee!