• Tax authorities change their rules and forms every year, but don’t want to do total rewrites of their electronic submission and processing software.
• The financial services industry keeps inventing new products, which don’t just have different terms and conditions, but may also have different kinds of terms and conditions.
• The same, to some extent, goes for the travel industry, which also keeps adding different and different kinds of destinations.
• The energy industry keeps adding new kinds of highly complex equipment it has to manage.

Conor also thinks market evidence shows that XML’s schema flexibility is important for data interchange. For example, hospitals (especially in the US) have disparate medical records and billing systems, which can make information interchange a chore.

The second suggestion is probably the less controversial of the two. After all, everybody knows that data is very commonly exchanged in XML formats. So if it gets persisted in XML format somewhere along the way, even relational purists shouldn’t much mind, as long as it eventually gets into what they regard as a more properly structured database. (Besides — if the data is going on long, challenging, multi-stage journeys, then nobody should much blame it if it indeed wants to stop along the way somewhere and rest. )

In the first group of examples, there’s usually also a kind of cooperation between native XML and other kinds of database managers. Before those users had access to XML, they were getting by just fine using other database technology. So XML can be used in conjunction with other systems, not as complete replacement. Even so, it’s reasonable to consider scenarios in which XML is the primary data model of record, and relational/tabular copies of the information are secondary.

For example, an income tax authority wants to store your tax form in its entirety, so that they can check both your truthfulness and your arithmetic. This is most naturally done in XML, although for many years it’s been done in relational or pre-relational technologies. They also want to aggregate a limited amount of information from each taxpayer’s form for all sorts of aggregation and administrative purposes; that’s best done in a relational database. But the part that belongs in XML is the most fundamental.

As another example, the core information of a derivatives transaction is:

• The derivatives contract (naturally stored in XML)
• The actual purchase/sale information (traditionally stored in relational systems)
• Account balances of various kinds altered by the transaction (a classic case where relational databases guarantee much-needed data integrity)

Here the majority of the basic record fits best in XML. The minority that fits best in a relational system is small enough that a good XML DBMS can probably handle it as well. Neither the superior OLTP performance nor data integrity safeguards of a relational DBMS are needed for the purchase/sale information. They are needed for the general account management – but again, that’s a relatively secondary or (no pun intended!) derivative part of the overall database.

So what we’re coming up with here is a strategy along the lines of:

1. Use XML for your system of record.
2. Spawn transactions in your relational/tabular data stores right away.

And by the way, while I haven’t dwelled on this – those relational/tabular data stores could be data warehouses instead of or in addition to transactional systems.

Obviously, there are two major classes of objections to this strategy (when it is contrasted with a traditional relational approach):

• Assertions that the extra programming effort needed to assure data integrity are so important as to outweigh all other consideration.
• Assertions that the need for schema flexibility isn’t really that high, or at least wouldn’t be if the enterprises’ database designers were sufficiently competent.

Well, we’ll see. So far the customer uptake for the native XML approach is small but non-zero. And thus the issue is far from being decided.

]]> http://www.dbms2.com/2008/10/05/schema-flexibility-and-xml-data-management/feed/ 3 http://www.dbms2.com/2008/10/05/vertical-market-xml-standards/ http://www.dbms2.com/2008/10/05/vertical-market-xml-standards/#comments Sun, 05 Oct 2008 12:43:31 +0000 Curt Monash http://www.dbms2.com/?p=593 Tracking the alphabet soup of vertical market XML standards is hard. So as a starting point, I’m splitting a list I got from IBM into a standalone post.

Among the most important or successful IBM pureXML-supported standards, in terms of downloads and other evidence of customer interest, are:

• UNIFI (UNIversal Financial Industry message scheme). According to ISO 20022, it’s a standard that any electronic fund transfer must be accompanied by a UNIFI document. This standard seems important at least in SEPA (the Single European Payment Area). UNIFI seems to be a major pureXML use case.
• FpML (Financial Products Markup Language). FpML is used in the derivatives market, to actually create contracts and hence derivative securities.
• ACORD (Association for Cooperative Operations, Research and Development). When one discusses XML industry standards, ACORD is usually one of the first to come up. It is used in the life insurance agency, also for contract definition/creation. The top use case is agents, although speculate it’s used among investment managers and reinsurers as well.
• STAR (Standards for Technology in Automotive Retail). STAR is used by car manufacturers and retailers to exchange information – about cars, no doubt, and probably some other things as well.
• HR-XML, which is used by human resource departments for resumes and other kinds of employee information.
• OTA (Open Travel Alliance). This is an XML schema used by usual-suspect players in the travel business – hotel chains, car renters, travel agents, and probably anybody else who deals in travel reservations.

I’ve heard at least about the financial services ones from other XML database vendors as well. I’m a little surprised that nothing from the health area made the top of the list.

As I write it, I see there are a considerable number of holes, but that’s the way it seems to go when researching XML storage. I’m also writing up a September call from which I finally figured out (I think) the essence of how MarkLogic Server works – but only after five months of trying. It turns out that MarkLogic works rather differently from DB2 pureXML. Not coincidentally, IBM and Mark Logic focus on rather different use cases for native XML storage.

What I understand so far about the basic DB2 pureXML architecture goes like this:

• DB2 pureXML stores XML in “true hierarchical format.” Based on all the discussion of indexing, I’m guessing that the way it does this is somewhat similar to that in MarkLogic.
• Unlike MarkLogic, DB2 pureXML gives you the choice of what tags to index on.
• In a big difference from Marklogic, text search on DB2 pureXML involves two separate indexes – XML and text (the latter being of the usual inverted-list variety). You can text-search both contents and tags, with the usual CONTAINS semantics.
• PureXML has a data store separate from the rest of DB2′s, notwithstanding IBM’s references to XML “columns.” DB2′s general datatype extensibility framework is not used; I don’t wholly understand why.
• I neglected to ask how well DB2 backup, management tools, and so on extend to DB2 pureXML.
• You can talk to DB2 pureXML via two data manipulation languages: SQL/XML, and XQuery. Both are compiled down to the same run-time instructions. IBM said there’s an abstraction layer sitting over both the relational store and the XML store that allows for this I don’t totally understand what that means, since presumably the SQL/XML starts out by being sent to DB2′s parser.

A big part of IBM’s XML business strategy is to support various (typically vertical market) XML standards. IBM has implemented support for these standards and made it freely downloadable. What does “support” mean? It surely starts with a DTD (Document Type Definition), and apparently also includes mappings to generic web services interfaces. It turns out that there are a lot of them, so I’m listing some in a separate post.

More generally, it seems that the sales and uses for IBM pureXML are concentrated in two main (overlapping) cases:

• When XML was going to be used anyway. One big example of this is the case of the standards-based industry data interchanges. Another example is when pureXML, albeit disk-based, acts as a kind of quasi-cache or mini-MDM hub (Master Data Management) for WebSphere-based enterprise application integration (EAI). IBM reports that DB2 pureXML has been sold as an intermediate EAI data store at least once each in banking, retailing, health care, and insurance.
• When schema flexibility is of great importance.

Experience teaches me that schema flexibility is a subject that can attract considerable flames, in the general vein of “Omigod! The relational model is perfect because it’s mathematically proven to ensure referential integrity!!” So I’ll split out the main discussion of that into yet another separate post, and keep going.

IBM actually breaks out the pureXML use cases into four main groups:

1. Transactional. This comprises the transactional logging of information that just happens to be XML, such as in financial services.
2. Forms-oriented. This comprises, for example, the tax authority use case.
3. Service bus acceleration. That’s a fancy phrase to cover both the standards-based interchanges and the other EAI-related uses.
4. Event-driven data warehousing. This one was kind of blurry to me. What I think it means is that if you have transactional data in XML, and you want to use it in near-real-time business intelligence, DB2 pureXML can help you with that.

#1, 3, and 4 seem to fit into my “When XML was going to be used anyway” category. Part of “Schema flexibility” matches #2; I’m not clear on where in IBM’s four buckets the rest of schema flexibility goes.

Finally, I asked directly in what areas there were significant numbers of DB2 pureXML customers. IBM offered two examples. One was financial services in general — especially in North America, notwithstanding the importance of the UNIFI standard in Europe. The other was health care data interchange outside the United States — especially in China, where regional and national centers are being established to more closely oversee local hospitals.

Related links

• IBM kindly gave me permission to make available the slide presentation from our August 29 briefing. The last page has a large number of links to further IBM pureXML resources.
• Conor O’Mahony has a good blog.
• As noted above, I am putting up separate posts on standards-based data interchange and schema flexibility.