Creating Custom Providersby John B. Moore Micro-Phyla Systems IntroductionThe Java JDBC architecture is fundamentally a "Provider/Resolver" model. In the JBuilder DataExpress, the process of providing data is handled by the StorageDataSet classes. For basic two tier configurations the built-in providing capabilities of the StorageDataSet classes will handle the majority of providing needs. For those cases where the data or datasource is non standard, or special coercion of data types is required, then a custom provider is needed. The DataExpress Provider abstract class provides just such tools. From abstract class, one can extend and utilize the structure and methods provided to build a complete Provider for any target dataset source. It is important to realize that the outline of the Provider mechanism we are about to study is the exact same mechanism that the standard StorageDataSets utilize. Therefore, in addition to understanding how to build your own Providers, you will also better understand how the DataExpress provides data to its primary built-in dataset objects.
Figure 01 - Diagram of basic DataExpress Architecture Provider Class The Provider class has a number of methods that will be used or overridden when writing a custom provider. Following is a basic discussion of each method checkIfBusy(com.borland.dx.dataset.StorageDataSet) If your implementation requires that it provides the data asynchronously, you will need to override this method. This allows the StorageDataSet to block actions such as resolving, until the asynchronous data is present. This method allows an implementation to give an appropriate error message by raising a DataSetException. The default implementation is to do nothing. checkMasterLink(com.borland.dx.dataset.StorageDataSet, com.borland.dx.dataset.MasterLinkDescriptor) This is called to validate the masterLink property of the Storagedataset. In the case of the Querydataset, when the MasterLinkDescriptor's fetchAsNeeded property is enabled (true), the QueryProvider uses this method to check if there is a WHERE clause in the query. If no WHERE clause is specified, the QueryProvider throws a DataSetException. In your own provider you could override this method to do a similar function. close(com.borland.dx.dataset.StorageDataSet, boolean) This method would be used to release resources kept for loading data on demand. A StorageDataSet calls this method when the storage is being closed and when calling StorageDataSet.closeProvider. The boolean, loadRemainingRows parameter, controls whether the rest of the data (if more data is available) is loaded or not. hasMoreData(com.borland.dx.dataset.StorageDataSet) This method will allow an implementation of a Provider to provide part of the data, then load more data on demand. This method should return true if there is more data to be loaded. To load the data, call the provideMoreData method. There are three cases where a Storagedataset will call this method:
Then, at this point, this method is called. If it returns true, the method "provideMoreData(..)" is called provideData(com.borland.dx.dataset.StorageDataSet, boolean) This method is the "meat and potatoes" of the provider class, as its job is to provide the data for a DataSet. The source of the data, and the method of retrieving the data is up to the implementation of this abstract method. This is where you do the most customization. The toOpen boolean parameter indicates whether this method is called, as part of opening this StorageDataSet. Most of the rest of this paper will be devoted to the of discussion this method. provideMoreData(com.borland.dx.dataset.StorageDataSet) This method is the partner of the "hasMoreData(..) method described above. ProvideData/LoadData Methods - The CoreProvideData/LoadData are the methods where you will do the most customizing. To this end, we will examine in detail all the aspects of these methods and how they function. Figure 02 provides a brief outline of how to wire the ProvideData and LoadData methods.
Figure 02 - Diagram of basic Provider Step 1 - Preliminary work - Setup the MetaData definition (MetaData discovery). The minimum required is the Column name and Data type. In addition, you might consider providing any of the following:
These properties are used by the ...sql.dataset classes so depending on how you want to make these classes functional, you will need to provide these properties. You can hard code these properties or invest the time and effort and use the data source's functions to query the database and dynamically provide this information, or utilize any other discovery methods provided by your datasource. Performance will suffer if you have to discover this dynamically, and since this is a "custom" provider, you would be advised to hard code this information in many cases. A flexible way to hard code the meta data is to create a "datalayout" interface that you implement in your custom provider. In the data layout interface, declare the "constants" that define the table. Also utilizing this technique the bulk of the provideData() and loadData() code can be reused with different target "tables" within a datasources, without recoding these methods. Step 2 - Now we begin laying out the actual "providedata" code. Start off by implementing a class that extends Provider and implements LoadCancel and your DataLayout interface as outlined above. This class will have to implement at least two methods: loadCancel(..) and provideData(..) plus whatever "helper" methods you require. The provideData method generally consists of several steps in this order:
Step 3 - Next we setup a loadData(...) Method. This method is the work horse of the provideData method and is where the real "customizaton" goes. Because the core of this method is so unique to each provider, I will only outline the basic generic steps (See Figure 02 for an overview) you need to take to do the job. Some examples will be provided later that will illustrate how this would be implemented.
Step 4 - Implement the cancelLoad() method. Often this involves only setting the cancel field to true. Step 5 - Decide if you want design time functionality. To determine if the provideData() method was called while in design mode, call java.beans.Beans.isDesignTime(). After making this call you can prevent data from loading, inform the user, or whatever you require as your provider's response to a design time environment. Setp 6 - Implement any other "features" via methods that are unique to the source dataset. You're done! Now, by magic, your data loads into the StorageDataSet in the same way as the DataExpress implementations: QueryDataSet and ProcedureDataSet. In fact, if the job was done correctly, a user of the provider will not be able to tell the difference. The beauty of the implementation and structure you are leveraging is that the rest of the work will be done for you. The data will be provided automatically under the follow circumstances:
Building a Basic Custom ProviderSo much for theory. The best way to get a feel for how this all works is to actually go through a simple implementation of a Provider. We will start by a quick overview of a simple text file provider that comes with JBuilder found at: ../samples/dataexpress/providerresolver/ and follow each of the outlined steps so that we can relate each step to a simple example. Step 1 - Define MetaData First we need to define a MetaData data layout of the source text file. This can be done using an interface that will be called "DataLayout", where all the column attributes: Name, Width, Caption, and Type are defined. Step 2 - Setup implementation of the provideData() method. The provideData() method must extend Provider and implement the LoadCancel() interface. In addition it can implement your DataLayout interface, but this is not required. Respond to the toOpen argument Using the following code you can respond to the boolean argument toOpen. This argument allows you to block loading of the dataset automatically on the opening of the dataset. if ( toOpen )
return;
Load column definitions The DataLayout information is loaded in the beginning of the "provideData()" method, with the code: Column[] columns = new Column[COLUMN_COUNT];
for (int i=0; i < COLUMN_COUNT; i++)
columns[i] = new Column( COLUMN_NAMES[i],
COLUMN_CAPTIONS[i],
COLUMN_TYPES[i] );
Because this is a text provider in which we have to know the datasource schema in advance, hence this step is hard coded. When working with other types of providers, you might have the ability to "query" the datasource for this information. If so, this would be where you do this step. Regardless, at this point you need to have defined your column object. Call providerHelp.initData to merge columns to StorageDataSet which returns a columnMap. Simply plug in the correct arguments. The work of merging the columns to the StorageDataSet is done for you. The call in this case will look like: int[] columnMap = ProviderHelp.initData(dataSet, columns, true, false, true); Where the arguments are:
Set the RowID Next set the column or columns that will act as the primary key of this dataset. This is needed for various reasons: both to maintain uniqueness on the local data, but also to provide a generic way for your "resolver" to know how to post data back to the data source. Think of it as a generic way to communicate with your "resolver". Of course, if your data is read-only and no resolution will occur then your only need for a rowID will be sorting. It then, may not be a necessary step, and therefore is optional based on your situation. The code is simple and straight forward: dataSet.setAllRowIds(false); Column rowid = dataSet.getColumn(ROWID_NAME); rowid.setRowId(true); rowid.setHidden(true); This last method setting, setHidden(), allows you to keep the ID row hidden by default. Your IDE or user code can override this at any time. Again, the need to do this, will be determined by the way your data will be used. Empty the StorageDataSet Now we make sure the dataset is empty. It may well be full of data if this is a repeat call to the provideData method. Remember this method is called at various times and situations and therefore, we need to insure an accurate set of data based on the request. If this is a call to "load more data" and you are providing this functionality, you will need to pass over this call in those situations. Utilizing a boolean flag that you set, will do this job. In cases were FetchAsNeeded is required your need to handle this possiblity. Emptying a fetchAsNeeded dataset will cause a NullPointerException in StorageDataSet.recordDetailsFetched since a protected member: fetchDataSet ends up being null. Make sure dataSet.empty is not called for datail datasets with when "fetchAsNeeded == true". dataSet.empty(); Define your MaxRows Here we set the maxRows. There are couple of things worth noticing in this line of code. First, that there is two sources of "maxRow": DesignTime and the regular source from the property setting for the dataset. As a provider designer, you can make a decision here to prevent design time access to data. int maxRows = java.beans.Beans.isDesignTime() ? dataSet.getMaxDesignRows() : dataSet.getMaxRows(); There are a number of choices you can make. You can wrap your setting of maxRows such that if design time is detected, it always is set to zero, or you can do what is done in the example above, and that is to place a strict limit on design time regardless of what the user of your provider sets as a maxRow. Hand the "dataset", "maxRows" and "columnMap" to the overridden loadData(..) method Our last step for the provideData() method is to hand all the pieces we have gathered to the loadData() method. We will build that next. Step 3 - Implement the loadData() method Implementing the loadData() method requires some basic steps be followed in order to insure that your provider is responsive to navigation and loading events, similar to what is expected from the standard providers: QueryDataSet and ProcedureDataSet. Each type of provider will implement these steps in different way, therefore focus on the basic ideas behind each step. The details will vary with the type of data being loaded. This contrast will be more evident as we implement an XML provider following this example. You should come to realize that this is the location for the majority of the required customization. Set the "cancel" variable Set the variable "cancel" to false so that we can detect if our public loadCancel() method (which we will build later) can shut down the loading process. cancel = false; Call dataSet.startLoading(..) The synchronized method, startLoading(), returns a variant array reference that gives us access to an internal array of Variants in the StorageDataSet. We will use this array during our custom loading code as hook into the StorageDataSet. Each array variant is preset with its datatype that you defined earlier, when we called the initData() method. Variant[] variants = dataSet.startLoading(this, RowStatus.LOADED, false); Only one "startLoading" can be active at a time. There are three versions of this method, depending on the degree of control you require. startLoading(LoadCancel loader, int loadStatus, boolean loadAsync) In this version loadUncached and loadValidate are set to false by default. This means that the variant added is cached and persisted and not validated by any column domain settings. This method call results in the fastest loading combination. startLoading(LoadCancel loader, int loadStatus, boolean loadAsync, boolean loadUncached) You would call this version if you needed to set loadUncached to true. LoadUncached is not documented very clearly in the help. When set to , this means that the row is not persisted in the StorageDataSet. If another row is loaded, it replaces this row; hence the effect is a single row in the StorageDataSet at any time. Note that loadUncached can be thought of as a special case of Load.AS_NEEDED.The exception is that as Load.UNCACHED it will only hold on to one RowStatus.LOADED row. If either provideMoreData() explicitly called, or navigation past the end of the DataSet occurs, a new row is loaded to replace the current row. There is also a StorageDataSet.provideMoreData() that will ask the provider to provideMoreData() and return true if it can. startLoading(LoadCancel loader, int loadStatus, boolean loadAsync, boolean loadUncached, boolean loadValidate) Use this version if you want to both; force loadUncached and/or validate (via DataRow.validate()) each row of data. This method checks all columns in the DataRow for any defined constraints on the data, such as minimum or maximum value, readOnly, and etc.) Calling this method in this fashion, would result in the slowest loading of your data. Parameters for startLoading():
Load the data This step is the most custom part of the provider. This is the point at which the data is actually retrieved, be it a file, a stream from a http request, or a method call to an EJB server. In this case a file is opened with a call to FileInputStream(). FileInputStream fs = null; fs = new FileInputStream(getTextDataFilePath("DataExpress/ProviderResolver", "data.txt")); Setup a for-loop to move the data into the dataset. (with a nested for-loop described next) that loads each row, and at the end of that loop calls dataset.loadRow(). This method takes your variant array and loads it into the StorageDataSet. Nested within the above for-loop create another for-loop that walks the columns and loads the columns into the variant array. Be sure to check the "cancel" variable with each loop, to insure that your loading will respond quickly to a call to cancelLoad(). Once you exit the outer for-loop place the call to dataset.endLoading() To close the load buffer in the StorageDataSet. Whew! It seems like a lot to do, just to load a bit of data, but hopefully it is clear that this structure allows for a lot of flexibility and robustness in loading your data. There are other issues we have not covered, because they were not relevant to loading data from a file. The most often needed issue is the idea of "load as needed". This concept is important for situations like large datasets that are being viewed in a small, grid or situations where your data is linked to a master record. The last thing you would want is the loading of the entire table to view five records! Advanced ProvidersAt this point you should have all the basics well in mind. Therefore the following examples will focus on the unique issues of that provider and skip over the more obvious steps covered previously. If you need to review those issues, just browse the source code of these examples. With the XML provider there will still be steps where a bit more detail is required, because this provider is especially unique. XML ProviderThere may be situations when you need to connect to a datasource that only provides XML in response to data requests. This scenario will become very common for Business-to-Business interactions. In fact I am currently working on a system that will provide insurance quotes via XML to any client that both understands the standardized insurance XML and , of course, has the correct authorization. In the following example, the details of how to setup a SAX parser are not included, but are in the sample code for you to study. Appendix A reviews a few key points to hopefully get you going, if XML parsing is new to you. The Motivation Why create an XML provider? Might it be simpler just to parse the XML and stick it into a TableDataSet. Several issues need to be considered in deciding what we gain by trying to create an XML provider. Consider the following reasons:
Unlike the previous example we will now skip some of the more obvious steps, and only focus on discussing those steps that set up the needed XML functionality. Setup Tasks Two rather lengthy tasks need to be completed before working on the XML Provider. First is the XML tag scheme. If you are interested in the details, review Appendix A where that step is documented. Next, is the servlet that will supply the XML in response to our request. The code for the servlet is provided and will be left for you to study at a later time. You will find though, that much of the XML part of the code is very similar to the Provider code that is outlined below. ProvideData() Method Notice that one of the first steps in the provideData() method, is the metadata discovery of the database. Before we can do this, we need to design a request for the data structure (See Appendix A). We could just hard code this, similar to the text file example, but it might be more instructive if we do some "metadata discovery" To this end we create the class XMLMetaData. In this class, we have the usual static COLUMN_COUNT, COLUMN_WIDTH..etc, and we implement several methods that we need to call in the provideData() method. They are: public Column[] getXMLColumns()throws DataSetException This method sends back a column array that we will need for ProviderHelp.initData() method. Internally, we have passed the XML request and returned the table information as an XML string. This is passed to a SAX parser (See Appendix B). We then have to implement a DocumentHandler, that will respond to the tags and their attributes that we are seeking. This handler then loads the static fields that define our table, row id, and field data types. This method is used as follows: Column[] columns = xmlData.getXMLColumns(); Like our previous example this is then used to initialize the dataset as follows: int[] columnMap = ProviderHelp.initData(dataSet, columns, true, false, true); public String getRowIDName() The rowID is set for the table. This information is required in the following code: dataSet.setAllRowIds(false); Column rowid = dataSet.getColumn(xmlData.getRowIDName()); rowid.setRowId(true); LoadData() method The last step in the provideData() method, is the actual retrieving of the data that is handled by the loadData() method. In a commercial version, this should be designed such that a URL and the request string would be handed to the desired StorageDataSet. By subclassing, new TableDataSet properties can be added that the provider would retrieve as needed. For the purpose of keeping this demonstration simple, this has not been done, and instead the URL and the request string has been hardcoded. The first step in this method is to call startLoading(), and get a handle on the variant array of the StorageDataSet. The next group of steps are simply standard communications with a servlet. They are:
Then the whole process is wrapped up by calling dataSet.endLoading(). The dataset is loaded and ready. EJB ProviderCreating an EJB provider is a next logical topic to cover. It should be clear at this point all the issues that need to be addressed. It also should be clear that the two key areas that need to be "customized" are the definitions of the columns and the LoadData() method. There you are free to literally "load" the data and make sure that each field is loaded into the variant array in the correct order that you defined the columns. You will need to either hardwire the column metadata or provide some methods in a EJB Session Bean that will provide this information on request. Then, in the loadData() method, the code must then call another EJB Session Bean that returns the requested data. Most of your work will be in the back end and the actual provider will be no more involved that the previous examples. An example has been provided for you to study. Wrap UpDataExpress provides a contract structure that you can utilize to bring data from any provider to a TableDataSet. Utilizing this contract, you are free to develop any type of provider from any data source. Creating a provider for a very specific data source is involved, but fairly simple as illustrated. It is worth noting that in designing these examples, I basically cut and pasted the basic provider into the needed class name, and then made changes to that code where necessary, mostly in the loadData() method. The real challenge will be to create a provider that can retrieve data from any data source of "that type" In the book "Java and XML" (see References below) the author discusses just such a scheme for XML sources. This can easily be another future topic, but is beyond the scope of this paper. ReferencesFollowing are a number of references and links that at some point provided help or ideas that were used in this paper. Many of these references are more of the nature of "background" information, and might be necessary to help make the core issues more understandable for the reader. Books Java and XML Programming with servlets and JSP, ISBN 1-861002-85-8 XML and Java - Developing Web Applications, ISBN 0-201-48543-5 Java Servlets, ISBN 0-07-913779-2 XML Applications, ISBN 1-861001-5-25 Links http://www.xml.com/pub XMX.com - Plenty of good ideas on use of XML http://www.borland.com/techpubs/jbuilder/jbuilder3-5/database/prov_custom.html - Writing a Custom Provider such as SAP, BAAN, IMS, OS/390, CICS, VSAM, DB2, etc. http://www.borland.com/techpubs/jbuilder/jbuilder3/ref/dx/com.borland.dx.dataset.Provider.html - Description of provider class. http://www.borland.com/techpubs/jbuilder/jbuilder3/database/app_distrdb.html - Applications on custom providers in distributive application designs. It discusses creating a distributed database application using the DataSetData component and Remote Method Invocation (RMI) for creating a distributed application. http://www.borland.com/techpubs/jbuilder/jbuilder3/ref/dx/com.borland.dx.dataset.ProviderHelp.html - Description of providerHelp class. http://www.borland.com/techpubs/jbuilder/jbuilder3-5/database/prov_providingintro.html - Using JBuilder's DataExpress architecture to retrieve data from a data source, and provide data to an application. The components in the DataExpress packages encapsulate both the connection between the application and its source of the data, as well as the behavior needed to manipulate the data. http://www.borland.com/midas/startclient/page3.html - To add special functionality to providing data or resolving modifications, it is possible to build custom providers and resolvers to overwrite the built-in functionality in CorbaConnection. http://www.borland.com/techpubs/jbuilder/jbuilder3-5/database/dh_concepts.html - DataExpress components were designed to be modular to allow the separation of key functionality. This design allows the DataExpress components to handle a broad variety of applications http://www.borland.com/techpubs/jbuilder/jbuilder3/database/ap_howto.html - This link is comprised of answers to questions posted on the JBuilder Database newsgroup. This document will be posted on the newsgroup, and updated there periodically. To access the Database newsgroup, borland.public.jbuilder.database, point your browser to http://www.borland.com/newsgroups/. Appendix AXML Tags Primer For this demonstration, a simple XML tag schema was developed. A full discussion of DTD , DOM and tag component structure and development, is beyond the scope of this paper. For this example we will play it loose, and just outline what we need to express a simple table of data. Following, is a brief description of the tag scheme develop for this example. Request Tags Our request tag needs the ability to send a simple query to the servlet for data. It therefore needs to provide a table name, a list of requested fields, a where clause, and a count of records requested. <REQUEST></REQUEST> <QUERYDATA table="" fields="" where="" returncnt=0></QUERYDATA> </QUERYDATA> A basic request might look like: <REQUEST> <QUERYDATA table="Product" fields="PROD_CODE,PROD_DESCRIPT" where="PRO_PAR < 1.00" returncnt=50></QUERYDATA> </REQUEST> <MetaData table="" fields=""/> A basic request might look like: <REQUEST> <METADATA table="Product" fields="PROD_CODE,PROD_DESCRIPT" /> </REQUEST> Reponse Tags <RESPONSE></RESPONSE> <TABLEDATA tblname="" rowid="" morerecords="true"></TABLEDATA> <TBLROW></TBLROW> <TBLFIELD colname="" datatype="">value<TBLFIELD> A basic table response might look like: <RESPONSE> <TABLEDATA tblname="Product" rowid="PROD_CODE" morerecords="true"> <TBLROW> <TBLFIELD colname="PROD_CODE" datatype="varchar10">16901</TBLFIELD> <TBLFIELD colname="PROD_DESCRIPT" datatype="varchar20">Static Cling Decal</TBLFIELD> </TBLROW> <TBLROW> <TBLFIELD colname="PROD_CODE" datatype="varchar10">BS1S</TBLFIELD> <TBLFIELD colname="PROD_DESCRIPT" datatype="varchar20">Bumper Sticker</TBLFIELD> </TBLROW> <TBLROW> <TBLFIELD colname="PROD_CODE" datatype="varchar10">DSBA</TBLFIELD> <TBLFIELD colname="PROD_DESCRIPT" datatype="varchar20">Assorted Sports Btls</TBLFIELD> </TBLROW> </TABLEDATA> </RESPONSE> <METADATA table="" columncount=0 rowid=""> </METADATA> <COLUMN name="" columnwidth=0 columndatatype="" columncaption=""/> A basic response might look like: <RESPONSE> <METADATA table="Product", columncount=2 rowid="PROD_CODE"> <COLUMN name="PROD_CODE" columnwidth=10 columndatatype="varchar" columncaption="Product Code"/> <COLUMN name="PROD_DESCRIPT" columnwidth=20 columndatatype="varchar" columncaption="Description"/> </METADATA> </RESPONSE>
Appendex BSAX - Simple API for XML There are a number of methods by which information can be extracted from an XML string. SAX provides a lightweight, event-driven mechanism. Setting up SAX involves several steps that I will outline here. You should consult a good XML book for a more detailed discussion. The purpose of this appendix is to provide you with a quick overview of the process such that the discussion on the XML provider makes a bit more sense. Implement a DocumentHandler or extend a HandlerBase The DocumentHandler requires that you implement a number of methods that will be triggered by the parser. These events will allow you to extract information during the parse. The basic methods you need to implement are:
By their names, you can guess fairly well their purpose. In addition to implementing all of the methods you can extend the adapter class HandleBase, and using that, can override only those methods you need. This would be useful if you only need one or two methods. |
