MediaWiki API result

{
    "batchcomplete": "",
    "continue": {
        "gapcontinue": "Report_on_the_2007_OPeNDAP_Developer's_Workshop",
        "continue": "gapcontinue||"
    },
    "warnings": {
        "main": {
            "*": "Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes."
        },
        "revisions": {
            "*": "Because \"rvslots\" was not specified, a legacy format has been used for the output. This format is deprecated, and in the future the new format will always be used."
        }
    },
    "query": {
        "pages": {
            "476": {
                "pageid": 476,
                "ns": 0,
                "title": "Relational Database Handler",
                "revisions": [
                    {
                        "contentformat": "text/x-wiki",
                        "contentmodel": "wikitext",
                        "*": "Hyrax needs a handler that allows it to access data in a Relational Database Management System (RDBMS). (This requirement stems from the observation that many of our stake holders store their in situ measurement data in RDBMSs)\n\nIt is anticipated RDBMS handler will have a broad application for observation/in-situ data sets.\n\n\n== Use Cases ==\n# [[Adding the RDH to the BES]]\n# [[RDH handles bes:showCatalog request ]]\n# [[RDH handles a DDX request ]]\n# [[RDH handles a DDS request ]]\n# [[RDH handles a DAS request ]]\n# [[RDH handles a DAP2 data request ]]\n\n== Definitions ==\n\n; row set\n: A '''row set''' is an object which encapsulates a set of [http://en.wikipedia.org/wiki/Row_(database) rows]. [http://en.wikipedia.org/wiki/Database_table Database tables] are '''row sets'''.  A database ''[http://en.wikipedia.org/wiki/View_(database) view]'' is a '''row set'''. [http://en.wikipedia.org/wiki/SQL_query SQL queries] return '''row sets'''. [http://en.wikipedia.org/wiki/Join_(SQL) SQL JOIN] operations take '''row sets''' as input and produce '''row sets'''.\n\n; Constraint Expression (CE)\n: The DAP constraint expression string as described in the [http://www.opendap.org/pdf/ESE-RFC-004v1.1.pdf DAP2 Specification].\n\n== Background ==\n\nIn the past a Java servlet called the DODS Relational Database Server (DRDS)  was used to provide DAP access to RDBMS holdings. However this older implementation has a number of shortcomings that preclude it's direct use in our current server architecture:\n* No longer supported.\n* Uses the Java DAP implementation \n* Not a BES module\n* Significant memory limitations\n* Difficult to configure/localize\n\nAlthough there has been continuing interest in a DRDS replacement within the OPeNDAP community, no funding has been available to develop a soution until recently. The IOOS project has a need to provide a [http://www.opengeospatial.org/standards/sos Sensor Observation Service (SOS)] interface for Hyrax. Since much of the sensor data is already held in RDBMSs it is a natural and necessary time to develop a DRDS replacement.\n\n== Design ==\nThere are a number of client/server interactions that can take place in the DAP protocol. However, from a programatic viewpoint, pretty much all of the activities involved in these interactions take place when a client requests DAP2 data from the server. In order for the server to respond to a DAP2 data request it must:\n* Build a DDS instance in memory.\n* Populate it with instances of DAP variables (typically from a custom class factory)\n* Parse the DAP2 Constraint Expression.\n* Read data from the underlying data source and apply the CE to the data.\n* Send the constrained data back to the client.\nIn order to illuminate the design of the RDH we will examine each component of this interaction in some detail.\n\nThe RDH will be a BES module/plug-in. It will use an implementation of the ODBC (most likely [http://www.unixodbc.org/ unixODBC]) to access the RDMS(s). ODBC Data Sources will be defined at the system level as usual.\n[http://www.unixodbc.org/odbcinst.html][http://msdn.microsoft.com/en-us/library/ms188681.aspx]\n[http://support.adobe.com/devsup/devsup.nsf/docs/53765.htm]\n[http://www.internet-webhosting.com/pleskfaq/ch03s77.html]\n\nAs with other DAP service implementations, the abstract data type classes in the [[http://www.opendap.org/api/pref/html/index.html libdap library]] get sub-classed and data source specific read methods are implemented. Because of the nature of the RDBMS data sources some data types (such as Sequence) will need to have other methods implemented.\n\n[[Image:RDH StaticClasses.png|thumb|640px|center|Data type classes for RDH]]\n\n==== New Specialized Class Methods ====\n\nHere is a (surely incomplete) list of methods that we can expect to add to some of the more complicated child classes of of the various [http://www.opendap.org/api/pref/html/index.html libdap types]. \n\n; SqlDDS\n: '''''setDataSource()''''' - Used to cache the ODBC Data Source reference associated with the DDS (probably during the construction of the DDS from ODBC introspection information) so that it may be used later for data access.\n: '''''getDataSource()''''' - Retrieves the ODBC Data Source reference so that it may used for database introspection or data access.\n\n; SqlSequence\n: '''''setDataSource()''''' - Used to cache the ODBC Data Source reference (probably during the construction of the DDS from ODBC introspection information) so that it may be used later for data access.\n: '''''getDataSource()''''' - Retrieves the ODBC Data Source reference so that it may used for database introspection or data access.\n: '''''setSqlQuery()''''' - Used to cache an SQL query string (typically generated by converting a DAP CE string to an SQL query during CE parsing activities)\n: '''''getSqlQuery()''''' - Retrieves the SQL query string for use during the read() (data access) activities.\n: '''''read()''''' - For SqlSequence the read() method will be fairly complex. It will be making one (or more?) database queries using the cached SQL query string, retrieving the '''row set''' result and passing it to it's child elements, controlling the '''row set's'''  row (and possibly column) cursor position, and cleaning up the database connection after use.\n\n; SqlTypeFactory\n: '''''NewUrl()''''' - Since the DAP URL data type has no logical counter part in the SQL data type mapping this method should probably just throw and exception if called.\n: '''''NewStructure()''''' - Since the DAP Structure data type has no logical counter part in the SQL data type mapping this method should probably just throw and exception if called.\n: '''''NewGrid()''''' - Since the DAP Grid data type has no logical counter part in the SQL data type mapping this method should probably just throw and exception if called.\n\nThis list is by no means complete and as the development proceeds the list (along with the associated figure) should be updated.\n\n===Building the DDS object in memory.===\n\n\nThe RDH will load (and cache) from it's specific configuration section of the BES configuration file a list of ODBC Data Sources that it will serve as DAP data sets (See Use Case  [[Adding the RDH to the BES]]).\nWhen the RDH receives a request for content (such as a DDS, DDX, DAP2 data etc.) it will identify the DAP dataset from the request, and determine which ODBC Data Source it must interact with to fulfill the request. \nThen, using ODBC (or SQL) introspection methods in the [http://msdn.microsoft.com/en-us/library/ms714177.aspx ODBC API], the RDH will identify the collection of tables and views available in the ODBC Data Source and use them to construct a specialized DDS instance in memory.\n\n[[Image:RDH ActivityDiagram.png|thumb|400px|center| Activity diagram for the RDH]]\n\n\nThe RDH will traverse the available tables and views in the ODBC Data Source and it will build a representation of each one as a DAP Sequence object.  The columns in each  table and view will represented by a DAP variable in the corresponding Sequence instance. \nEach of these Sequence instances  will be added to the DDS instance. \nWhen building  the Sequences and their variables the RDH will use a custom class factory to build instances of DAP objects that can utilize the ODBC API to extract data from the Result Set of an ODBC brokered database query.  Each of the data types generated by the custom factory will have implemented read methods able to read data from the '''row set''' returned through the ODBC API. The Sequence instances will each hold the information about their associated ODBC Data Source for use during the data serialization activity.\n\n\n[[Image:BuildingDDS.png|thumb|640px|center|Building a DDS instance from an ODBC Data Source]]\n\n==== Special Metadata for Sequences ====\n\n;Table Relationships\n: This implementation of the RDH will not perform SQL JOIN operations on the tables and views in the ODBC Data Source as it is beyond the current scope/funding to undertake such an effort. However, the primary and foreign key information about the tables and views should be available through the [http://msdn.microsoft.com/en-us/library/ms711722(VS.85).aspx ODBC/SQL introspection API]. This key information is what allows JOIN operations to be made, and it is considered important and useful information for users who are accessing the DAP data source. We would like to gather the primary and foreign key information for the tables and variables in the ODBC Data Source and include it in the DAP data source as DAP Attribute metadata. How this information should be represented as such is TDB. We will work out the representation as the implementation work on the RDH progresses. \n\n;ODBC Datasource\n: Because each DAP Sequence represents a table or view in the ODBC Data Source, it is equivalent to saying that a DAP Sequence represents a '''row set'''. Since it not possible to return multiple '''row sets''' from a single query, and it is possible for a ODBC Data Source to contain multiple tables and/or views, it is implied that a DAP request for a DDS representing a ODBC Data Source could return multiple Sequences. Or, in other words, multiple '''row sets'''. Thus each Sequence's read() method will have to retrieve a '''row set'''. Which is a rather long winded way of saying that when each Sequence's read() method is called it will have to transmit an SQL query to the ODBC Data Source that holds the table or view that it represents. Thus each Sequence must know which ODBC Data Source contains its associated table or view.\n\n;SQL Query String\n: Retrieving the data for each Sequence involves submitting an SQL query to the underlying ODBC Data Source. This query is a product of parsing the DAP Constraint Expression. Since the CE parsing takes place prior to data access activities the Sequence must be able to hold the SQL query string until it is needed for data access.\n\n=== Handling the DAP Constraint Expression===\n\nOnce an instance of SqlDDS has been created the Constraint Expression can be parsed and applied to it. This process is normally handled in 2 steps: First the CE is parsed and all of the projected variables in the DDS are marked as such. Second, during the reading of the data (which takes place during the serialization process), each data value is checked against the collection of Clauses in the CE. If it meets the CE criteria it is serialized back to the client. In the RDH the process is somewhat more complex because the RDBMS system can (and should) be used to apply most f the relational constraints. Thus in the RDH applying the CE works more like this:\n# Create the DDX in memory\n# Parse the CE\n# Mark all of the projected variables in the DDS.\n# Use the CE to create an SQL query for each Sequence (which represents a table in the RDBMS) in the DDS.\n## Projected variables should be \"SELECT\"ed in the SQL query.\n## Where possible convert all the clauses in the CE to SQL WHERE clauses.\n## [[RDH: Creating an SQL query from a DAP2 Constraint Expression | More details regarding the conversion of DAP Constraint Expressions to SQL queries here.]]\n[[Image:ParsingConstraintExpression.png|thumb| 640px|center|Processing a DAP Constraint Expression.]]\n\n=== Sending DAP2 Data ===\n\nWhen sending data to the requesting client via an SqlDDS:\n# As each Sequence containing projected variables in the DDS is read, the Sequence.read() method should first connect to the ODBC Data Source and issue the (previoulsy constructed) SQL query.\n# The '''row set''' returned by the query is then used as the data source for the Sequence's child variables.\n# Any CE clauses that were not converted into SQL query clauses should be applied to the data as it is read from the '''row set'''.\n# Any array sub-setting operations must be applied to the data returned in the '''row set'''.\n\n\n;Sequence.read() \n: - Make database query using SQL query string generated by constraint parsing activity..\n: - Retrieve '''row set''' response\n: - Set '''row set''' cursor to first row, first column.\n: - For each row in the '''row set''':\n:: - Serialize() each projected child variable\n:: - Pass '''row set''' to each variable\n: - Move '''row set''' cursor to next row.\n\n;BaseTypeVariable.read()\n: - Read value from '''row set'''\n: - Apply remaining constraints and array sub-setting\n: - Transmit data\n\n\n'''Details'''\n* Each implementation of a simple DAP type will need to be able to read from the '''row set''' returned by an SQL query via ODBC.\n* The implementation of Sequence type will have to manage moving the '''row set''' \"cursor\" from one row to the next.\n* The DAP constraint expressions will have to be converted into SQL queries. (By a child class of [http://scm.opendap.org/svn/trunk/libdap/ConstraintEvaluator.cc  ConstraintEvaluator] perhaps? )\n* Each Sequence requested will be associated with a separate SQL query. As the DDS is serialized, each Sequence will use the ODBC API to send the query to the Data Source and then use the returned '''row set''' as the data content for the serialization of it's (the Sequence's) variables.\n\n== [[RDH: Mapping the ODBC data model to the DAP2 data model | Mapping the ODBC data model to the DAP2 data model]] ==\n\n== [[RDH: Creating an SQL query from a DAP2 Constraint Expression | Creating an SQL query from a DAP2 Constraint Expression]] ==\n\n== [[RDH Catalog Organization]] ==\n\n== Deliverables ==\n\nThe RDH deliverables consist of:\n\n# Source code with inline (doxygen/javadoc) documentation for the various classes and components.\n# Usage, design, and configuration documentation.\n# Production rules for the software (make files, ant build files, etc.)\n# Compiled binaries for:\n#* CentOS (Linux packaged as RPMs)\n# Unit tests and regression for the software.\n\n== Period of use ==\n\n[[Category:Development|Relational Database Handler]][[Category:Hyrax Development|Relational Database Handler]]"
                    }
                ]
            },
            "760": {
                "pageid": 760,
                "ns": 0,
                "title": "ReleaseToolbox",
                "revisions": [
                    {
                        "contentformat": "text/x-wiki",
                        "contentmodel": "wikitext",
                        "*": "[[HowTo_guides|<< back to HowTo Guides]]\n\n==How to make a Matlab ocean toolbox release==\n\n===== Make sure that the source code you're working with is up to date with what is in subversion=====\n  As the GUI developer you need to  insure that the code in subversion is consistent with what you want  to release.  You can use 'svn status' to check whether or not your  version is up to date with the repository.  If any of your files show  'M' returned from 'svn status', that means they're locally modified  and your local code is different from what's in the repository.  In  that case you should double-check to make sure your local code is correct and use 'svn commit' to upload your code to the repository.   Once the repository is up to date with your local code you can continue with the next step.\n\n===== Since this is a shared repository, any new GUI version should be checked out as thoroughly as possible (on Matlab) before committing the changes to subversion.=====\n\n===== In regards to naming and numbering conventions:=====\n\n*If this is a new GUI being added to the toolbox, the name of the GUI should reflect the spelling and capitalization used in the original data set.  For example, if an acronym is used such as MODIS or GOES, the name should reflect this capitalization.\n*If you are using a function that is also used in another Matlab Ocean Toolbox GUIette, this function should be in the Common directory.  If it is not, then you should move it to the common directory and notify the developer of the other GUIette(s) to do the same.  Both of you should remove it from the individual GUIette directories.\n*Make sure that all file names (e.g., *.m files) in a GUI directory are unique and are not replicated in other GUI directories.  This is important because Matlab is not able to differentiate between files of the same name in different sub-directories.\n*Version numbers should adhere to a two element numbering scheme.  The subversion number will be equivalent to a third number, which is reserved only for minor changes being kept track of in subversion.  As such, most of the versions released will end in zero, with only the second digit usually being changed.\n\n===== Use the 'zip' command to create an archive containing the GUI code from your local SVN directory.=====\n\n===== Email the zipped file to cwolfteich@gso.uri.edu for him to test and upload to dev1.opendap.org=====\n\n===== Copy the current repository code as a release. =====\nTo do that you would 'cd' to the directory containing your local SVN code for the GUI you are releasing, then use SVN to make a copy, as in this example:\n\n<source lang=\"sh\">svn copy . https://scm.opendap.org/svn/tags/ml-ocean-toolbox/HYCOM/1.13.5</source>\n\nThis will copy all the files from the current directory ('.') to a subdirectory named '1.13.5' under ...svn/tags/ml-ocean-toolbox/HYCOM.  This way, each time a release is made, a copy of the new code is placed in a directory named with the release version number."
                    }
                ]
            }
        }
    }
}