DBMS2 -- DataBase Management System Services » Structured documents

Advice for some non-clients

Curt Monash — Fri, 30 Jul 2010 14:35:52 +0000

Most of what I get paid for is in some form or other consulting. (The same would be true for many other analysts.) And so I can be a bit stingy with my advice toward non-clients. But my non-clients are a distinguished and powerful group, including in their number Oracle, IBM, Microsoft, and most of the BI vendors. So here’s a bit of advice for them too.

Oracle. On the plus side, you guys have been making progress against your reputation for untruthfulness. Oh, I’ve dinged you for some past slip-ups, but on the whole they’ve been no worse than other vendors.’ But recently you pulled a doozy. The analyst reports section of your website fails to distinguish between unsponsored and sponsored work.* That is a horrible ethical stumble. Fix it fast. Then put processes in place to ensure nothing that dishonest happens again for a good long time.

*Merv Adrian’s “report” listed high on that page is actually a sponsored white paper. That Merv himself screwed up by not labeling it clearly as such in no way exonerates Oracle. Besides, I’m sure Merv won’t soon repeat the error — but for Oracle, this represents a whole pattern of behavior.

Oracle. And while I’m at it, outright dishonesty isn’t your only unnecessary credibility problem. You’re also playing too many games in analyst relations.

HP. Neoview will never succeed. Admit it to yourselves. Go buy something that can.

Smaller BI vendors. Analytic DBMS evaluations commonly include BI strategy and tool selection as well. If an analytic DBMS expert tells you he needs to learn more about your product line, don’t blow him off. In fact, you should be particularly embracing anybody who’s shown a fondness for small DBMS vendors; maybe he or his clients will like small BI vendors as well. That means (among others) you, Jaspersoft, Endeca, and Tableau.

Information Builders. Is there anything about your BI products that is in any way technologically differentiated? If so, you might want to mention some examples to somebody some time.

Kalido. I’ve said this to you before, but it bears repeating — your positioning translates to “I-CASE for analytics,” and that’s not a good thing. If your product is not as cumbersome and entrapping as that sounds, you need to do a much better job of explaining why not.

SenSage. You are what you are. Sell out while the selling is good. You don’t have the corporate personality to make it into the analytic DBMS mainstream on your own.

Ingres. You need to be more engaged with analysts than you are. Ingres navel-gazed too much 25 years ago, and evidently you haven’t outgrown it yet.

TIBCO. You probably have a lot of cool analytic technology, but I don’t know of an influencer who has much relationship with or trust in you. Rethink how you’re approaching influencer relations top to bottom.

Tableau. You had a lot of mindshare, but it’s fading. Do something.

MarkLogic, graph DBMS vendors, etc. You’re clinging too hard to the NoSQL label. Nobody is out there deciding among Cassandra, neo4j, and MarkLogic. They might be deciding between MongoDB and MarkLogic, I guess, but if you admit to yourself that’s all it is you’ll probably change your messaging somewhat.

Objectivity. Get real about marketing. Infinite Graph is a cool opportunity. But I didn’t even ping you for a meeting when I’m in your area next week, because I wouldn’t have known who to reach out to.

Everybody (especially Objectivity). “First X deployed in the cloud” is almost surely an inaccurate claim. Don’t make it. And by the way, even if it were true, it probably wouldn’t be interesting.

Toward a NoSQL taxonomy

Curt Monash — Sun, 14 Mar 2010 23:24:45 +0000

I talked Friday with Dwight Merriman, founder of 10gen (the MongoDB company). He more or less convinced me of his definition of NoSQL systems, which in my adaptation goes:

NoSQL = HVSP (High Volume Simple Processing) without joins or explicit transactions

Within that realm, Dwight offered a two-part taxonomy of NoSQL systems, according to their data model and replication/sharding strategy. I’d be happier, however, with at least three parts to the taxonomy:

How data looks logically on a single node
How data is stored physically on a single node
How data is distributed, replicated, and reconciled across multiple nodes, and whether applications have to be aware of how the data is partitioned among nodes/shards.

After talking with Dwight, and also with Cassandra project chair Jonathan Ellis, I feel I’m doing decently in understanding the first of those three areas. But there’s a long way yet to go on the other two.

In Dwight’s opinion, as I understand it, NoSQL data models come in four general kinds.

Key-value stores, more or less pure. I.e., they store keys+BLOBs (Binary Large OBjects), except that the “Large” part of “BLOB” may not come into play.
Table-oriented, more or less. The major examples here are Google’s BigTable, and Cassandra.
Document-oriented, where a “document” is more like XML than free text. MongoDB and CouchDB are the big examples here.
Graph-oriented. To date, this is the smallest area of the four. I’m reserving judgment as to whether I agree it’s properly included in HVSP and NoSQL.

As Dwight sees it, JSON (JavaScript Object Notation) is the emerging markup standard for the document-oriented data models, and to some extent the BLOB part of key-value models as well. Reasons seem to include:

JSON is something web developers are likely to know anyway.
JSON, unlike XML, is schema-less. In the NoSQL world, that’s perceived as a good thing.
Perhaps for both these reasons, JSON is perceived as easier to use than XML.

Except as noted, I’m not aware of anything that solidly contradicts the above.

Dwight went on to say that there are two main NoSQL replication/sharding models, in line with the seminal papers to which I previously linked:

Based on or resembling Dynamo. The core idea here is accepting eventual consistency among nodes as being good enough, even if that means you sometimes read dirty data. The benefit is that you never are blocked from writing. By way of contrast, systems that enforce true inter-node consistency (think of a two-phase commit) can shut you down from writing if consistency guarantees aren’t being confirmed in a timely manner. Thus, in a Dynamo-like scheme you write data to multiple nodes, via consistent hashing; then when the time comes you read one or more nodes, and hope that what you’re getting back is a correct result.
Based on or resembling BigTable. In this model you’re trying to keep the nodes fully consistent in the usual way, e.g. by synchronous replication. Indeed, what’s being kept consistent is both data itself, and metadata about the data’s location. Details surely vary a lot from implementation to implementation.

I’m fuzzier on this stuff than on the data models, because to date nobody has ever explained to me how an actual live system (MongoDB, Cassandra, whatever) implements its replication strategy. Also, while I think that in both these models applications are allowed to be ignorant of the replication/sharding strategy, I’m not as sure of that as I’d like to be.

If we stop here, we already have something useful. MongoDB has a document data model, and is in the BigTable-like replication camp, at least at first. Cassandra has a table-like data model, and is on the Dynamo-like eventual consistency side. But to say those are the only differences that matter would be like saying that all shared-disk RDBMS (e.g., Oracle and Sybase IQ) are essentially alike. That, of course, would be nonsense.

So a third dimension needed in this taxonomy is how the systems actually bang data on and off of disk (or silicon, as the case may be). I don’t yet have an overview of that. I know something of how Cassandra does it, and will write about same in a future post, but that’s about it. So please stay tuned.

This and that

Curt Monash — Tue, 29 Dec 2009 09:14:46 +0000

I have various subjects backed up that I don’t really want to write about at traditional blog-post length. Here are a few of them.

Vertica offers a post on its 3.5 release, with a riff on the popular theme “We’ve fixed some weaknesses in our prior versions that we didn’t previously say we had.” More important, Vertica is pretty clear on the virtues of its hybrid columnar architecture.

Speaking of which — Oracle is going true hybrid columnar as well. I don’t have details or timing, however.

Dave Kellogg of Mark Logic wrote in to amusedly point out Oracle’s anti-MarkLogic collateral. The very first charge Oracle levies is that MarkLogic goes beyond the emerging XQuery standard to add additional functionality. Considering Oracle’s approach to SQL standards, I tend to share Dave’s amusement.

Bill Conniff of Xponent LLC wrote in to tell of a vastly cheaper and less functional approach to XML management, apparently geared to looking at very large XML files one at a time.

Cayuga is a Cornell research project in complex event processing (CEP). There’s a Cayuga academic home page, a Sourceforge page for some open source Cayuga CEP code, and so on. Minsheng Hong, writing from a Vertica email address, tipped me off some months ago. The basic idea seems to be to do lots of queries very quickly, rather than a smaller number of queries over and over again. Whether this is an advance in anything but open-sourceness over Apama or Aleri I couldn’t say, but I do think it’s a different focus than that of StreamBase or pre-Aleri Coral8.

And finally, editor Doug Henschen listed his 15 favorite Intelligent Enterprise blog posts of 2009 — four each by Seth Grimes and Doug himself, three by Cindi Howson, two by me,* and one each by Mark Smith and Neil Raden.

*Doug selects up to three posts a month from here to republish.

Technical introduction to Splunk

Curt Monash — Sun, 18 Oct 2009 16:01:06 +0000

As noted in my other introductory post, Splunk sells software called Splunk, which is used for log analysis. These can be logs of various kinds, but for the purpose of understanding Splunk technology, it’s probably OK to assume they’re clickstream/network event logs. In addition, Splunk seems to have some aspirations of having its software used for general schema-free analytics, but that’s in early days at best.

Splunk’s core technology indexes text and XML files or streams, especially log files. Technical highlights of that part include:

Splunk software both reads logs and indexes them. The same code runs both on the nodes that do the indexing and on machines that simply emit logs. However, in the latter case indexing is turned off. Thus, Splunk does not portray its software as “agentless.” However, it asserts that its agent-like software runs without “material” overhead.
The fundamental thing that Splunk looks at is an increment to a log – i.e., whatever has been added to the log since Splunk last looked at it.
Splunk tries to figure out what the individual entries are in a section of log it looks at. In particular:
- Time stamps are a big clue in this “inferencing” process, but they are not the be-all and end-all.
- Nor are line boundaries, if logs are naturally broken up into lines. (Splunk threw that latter comment in as a shot at SenSage.)
I get the impression that most Splunk entity extraction is done at search time, not at indexing time. Splunk says that, if a pair is clearly marked, its software does a good job of recognizing same. Beyond that, fields seem to be specified by users when they define searches.
Splunk has a simple ILM (Information Lifecycle management) story based on time. I didn’t probe for details.

Given its text search engine, Splunk does – well, it does text searches. And it stores searches, so they can be used for alerting or reporting. Indeed, Splunk persists and presumably updates results to stored searches, in a rough analog to materialized views.

Apparently, Splunk’s indexing is typically done via MapReduce jobs. I don’t know whether any actual Splunk searches are also done via MapReduce; surely they aren’t all, given the discussion of a near-real-time alerting engine and so on. Splunk fondly believes its MapReduce is an order of magnitude faster than SQL (I didn’t ask which SQL engines Splunk has in mind when they say this), and 5-10X faster than Hadoop. One efficiency trick is to look ahead and do Reduces in place where possible. This seems to be done automatically in the execution plan, ala Aster’s SQL-MapReduce, rather than having to be hand-coded. Splunk says its software can “easily” index 1-200 gigabytes of data per day on a commodity 8-core server, while maintaining an active search load, and 3-400 gigabytes are doable.

Splunk’s capabilities right now in tabular-style analytics seem to be limited to a command-line report builder, plus a GUI wizard that generates the command line. A few users have asked for support of third-party business intelligence tools, but Splunk hasn’t provided that yet. Nor can I find much evidence of ODBC/JDBC drivers for Splunk. But then, I have trouble understanding how Splunk could provide flexible and robust reporting unless it tokenized and indexed specific fields more aggressively than I think it now does.

How 30+ enterprises are using Hadoop

Curt Monash — Sat, 10 Oct 2009 10:19:29 +0000

MapReduce is definitely gaining traction, especially but by no means only in the form of Hadoop. In the aftermath of Hadoop World, Jeff Hammerbacher of Cloudera walked me quickly through 25 customers he pulled from Cloudera’s files. Facts and metrics ranged widely, of course:

Some are in heavy production with Hadoop, and closely engaged with Cloudera. Others are active Hadoop users but are very secretive. Yet others signed up for initial Hadoop training last week.
Some have Hadoop clusters in the thousands of nodes. Many have Hadoop clusters in the 50-100 node range. Others are just prototyping Hadoop use. And one seems to be “OEMing” a small Hadoop cluster in each piece of equipment sold.
Many export data from Hadoop to a relational DBMS; many others just leave it in HDFS (Hadoop Distributed File System), e.g. with Hive as the query language, or in exactly one case Jaql.
Some are household names, in web businesses or otherwise. Others seem to be pretty obscure.
Industries include financial services, telecom (Asia only, and quite new), bioinformatics (and other research), intelligence, and lots of web and/or advertising/media.
Application areas mentioned — and these overlap in some cases — include:
- Log and/or clickstream analysis of various kinds
- Marketing analytics
- Machine learning and/or sophisticated data mining
- Image processing
- Processing of XML messages
- Web crawling and/or text processing
- General archiving, including of relational/tabular data, e.g. for compliance

We went over this list so quickly that we didn’t go into much detail on any one user. But one example that stood out was of an ad serving firm that had an “aggregation pipeline” consisting of 70-80 MapReduce jobs.

I also talked yesterday again w/ Omer Trajman of Vertica, who surprised me by indicating a high single-digit number of Vertica’s customers were in production with Hadoop — i.e., over 10% of Vertica’s production customers. (Vertica recently made its 100th sale, and of course not all those buyers are in production yet.) Vertica/Hadoop usage seems to have started in Vertica’s financial services stronghold — specifically in financial trading — with web analytics and the like coming on afterwards. Based on current prototyping efforts, Omer expects bioinformatics to be the third production market for Vertica/Hadoop, with telecommunications coming in fourth.

Unsurprisingly, the general Vertica/Hadoop usage model seems to be:

Do something to the data in Hadoop
Dump it into Vertica to be queried

What I did find surprising is that the data often isn’t reduced by this analysis, but rather exploded in size. E.g., a complete store of mortgage trading data might be a few terabytes in size, but Hadoop-based post processing can increase that by 1 or 2 orders of magnitude. (Analogies to the importance and magnitude of “cooked” data in scientific data processing come to mind.)

And finally, I talked to Aster a few days ago about the usage of its nCluster/Hadoop connector. Aster characterized Aster/Hadoop users’ Hadoop usage as being of the batch/ETL variety, which is the classic use case one concedes to Hadoop even if one believes that MapReduce should commonly be done right in the DBMS.

Related link

An August, 2008 round-up of MapReduce applications.

HadoopDB

Curt Monash — Sun, 13 Sep 2009 04:59:39 +0000

Despite a thoughtful heads-up from Daniel Abadi at the time of his original posting about HadoopDB, I’m just getting around to writing about it now. HadoopDB is a research project carried out by a couple of Abadi’s students. Further research is definitely planned. But it seems too early to say that HadoopDB will ever get past the “research and oh by the way the code is open sourced” stage and become a real code line — whether commercialized, open source, or both.

The basic idea of HadoopDB is to put copies of a DBMS at different nodes of a grid, and use Hadoop to parcel work among them. Major benefits when compared with massively parallel DBMS are said to be:

Open/cheap/free
Query fault-tolerance
The related concept of tolerating node degradation that isn’t an outright node failure.

HadoopDB has actually been built with PostgreSQL. That version achieved performance well below that of a commercial DBMS “DBX”, where X=2. Column-store guru Abadi has repeatedly signaled his intention to try out HadoopDB with VectorWise at the nodes instead. (Recall that VectorWise is shared-everything.) It will be interesting to see how that configuration performs.

The real opportunity for HadoopDB, however, in my opinion may lie elsewhere. Rather than trying to compete with parallel relational DBMS, HadoopDB might do more good parallelizing more specialized kinds of database engines. How about, for example, a massively parallel XML manager to compete with MarkLogic? Or a massively parallel array processor other than the still-nascent SciDB? Or, even more to the point, something that parallelizes a yet-more-specialized scientific data management engine? That kind of area is where I suspect the potential for HadoopDB really lives.

IBM’s Oracle emulation strategy reconsidered

Curt Monash — Sat, 25 Apr 2009 02:10:58 +0000

I’ve now had a chance to talk with IBM about its recently-announced Oracle emulation strategy for DB2. (This is for DB2 9.7, which I gather has been quasi-announced in April, will be re-announced in May, and will be re-re-announced as being in general availability in June.)

Key points include:

This really is more like Oracle emulation than it is transparency, a term I carelessly used before.
IBM’s Oracle emulation effort is focused on two technological goals:
- Making it easy for an Oracle application to be ported to DB2.
- Making it easy for an Oracle developer to develop for DB2.
The initial target market for DB2’s Oracle emulation is ISVs (Independent Software Vendors) much more than it is enterprises. IBM suggested there were a couple hundred early adopters, and those are primarily in the ISV area.

Because of Oracle’s market share, many ISVs focus on Oracle as the underlying database management system for their applications, whether or not they actually resell it along with their own software. IBM proposed three reasons why such ISVs might want to support DB2:

Oracle is expensive. In particular, IBM suggested it is more flexible on licensing terms for resale than Oracle is. I find that easy to believe.
Hey, there’s a DB2 market or installed base out there of some size — why not address it?
Acquisition-fueled expansion in applications makes Oracle a much bigger competitor to many ISVs (all around the world) than it used to be before. That one makes all kinds of sense.

And by the way — if I wanted an Oracle-emulating DBMS, I’d feel a lot happier about doing business with IBM than I would with EnterpriseDB.

IBM feels that DB2’s Oracle compatibility is a strict superset of EnterpriseDB’s, which it presumably has carried over more or less in its entirety. I didn’t press too hard for examples of what Oracle emulation DB2 offers and EnterpriseDB doesn’t, but IBM did say something about support for more programming languages. IBM was clear on one broad area where DB2 does not offer Oracle emulation, which is the specifics of various kinds of datatype support or other specialized data access methods. For example, IBM has its own syntax for querying text, geospatial, or XML data, and has not added support for Oracle’s alternative approaches.

Schema flexibility and XML data management

Curt Monash — Sun, 05 Oct 2008 12:53:36 +0000

Conor O’Mahony, marketing manager for IBM’s DB2 pureXML talks a lot about one of my favorite hobbyhorses — schema flexibility — as a reason to use an XML data model. In a number of industries he sees use cases based around ongoing change in the information being managed:

Tax authorities change their rules and forms every year, but don’t want to do total rewrites of their electronic submission and processing software.
The financial services industry keeps inventing new products, which don’t just have different terms and conditions, but may also have different kinds of terms and conditions.
The same, to some extent, goes for the travel industry, which also keeps adding different and different kinds of destinations.
The energy industry keeps adding new kinds of highly complex equipment it has to manage.

Conor also thinks market evidence shows that XML’s schema flexibility is important for data interchange. For example, hospitals (especially in the US) have disparate medical records and billing systems, which can make information interchange a chore.

The second suggestion is probably the less controversial of the two. After all, everybody knows that data is very commonly exchanged in XML formats. So if it gets persisted in XML format somewhere along the way, even relational purists shouldn’t much mind, as long as it eventually gets into what they regard as a more properly structured database. (Besides — if the data is going on long, challenging, multi-stage journeys, then nobody should much blame it if it indeed wants to stop along the way somewhere and rest. )

In the first group of examples, there’s usually also a kind of cooperation between native XML and other kinds of database managers. Before those users had access to XML, they were getting by just fine using other database technology. So XML can be used in conjunction with other systems, not as complete replacement. Even so, it’s reasonable to consider scenarios in which XML is the primary data model of record, and relational/tabular copies of the information are secondary.

For example, an income tax authority wants to store your tax form in its entirety, so that they can check both your truthfulness and your arithmetic. This is most naturally done in XML, although for many years it’s been done in relational or pre-relational technologies. They also want to aggregate a limited amount of information from each taxpayer’s form for all sorts of aggregation and administrative purposes; that’s best done in a relational database. But the part that belongs in XML is the most fundamental.

As another example, the core information of a derivatives transaction is:

The derivatives contract (naturally stored in XML)
The actual purchase/sale information (traditionally stored in relational systems)
Account balances of various kinds altered by the transaction (a classic case where relational databases guarantee much-needed data integrity)

Here the majority of the basic record fits best in XML. The minority that fits best in a relational system is small enough that a good XML DBMS can probably handle it as well. Neither the superior OLTP performance nor data integrity safeguards of a relational DBMS are needed for the purchase/sale information. They are needed for the general account management – but again, that’s a relatively secondary or (no pun intended!) derivative part of the overall database.

So what we’re coming up with here is a strategy along the lines of:

Use XML for your system of record.
Spawn transactions in your relational/tabular data stores right away.

And by the way, while I haven’t dwelled on this – those relational/tabular data stores could be data warehouses instead of or in addition to transactional systems.

Obviously, there are two major classes of objections to this strategy (when it is contrasted with a traditional relational approach):

Assertions that the extra programming effort needed to assure data integrity are so important as to outweigh all other consideration.
Assertions that the need for schema flexibility isn’t really that high, or at least wouldn’t be if the enterprises’ database designers were sufficiently competent.

Well, we’ll see. So far the customer uptake for the native XML approach is small but non-zero. And thus the issue is far from being decided.

Vertical market XML standards

Curt Monash — Sun, 05 Oct 2008 12:43:31 +0000

Tracking the alphabet soup of vertical market XML standards is hard. So as a starting point, I’m splitting a list I got from IBM into a standalone post.

Among the most important or successful IBM pureXML-supported standards, in terms of downloads and other evidence of customer interest, are:

UNIFI (UNIversal Financial Industry message scheme). According to ISO 20022, it’s a standard that any electronic fund transfer must be accompanied by a UNIFI document. This standard seems important at least in SEPA (the Single European Payment Area). UNIFI seems to be a major pureXML use case.
FpML (Financial Products Markup Language). FpML is used in the derivatives market, to actually create contracts and hence derivative securities.
ACORD (Association for Cooperative Operations, Research and Development). When one discusses XML industry standards, ACORD is usually one of the first to come up. It is used in the life insurance agency, also for contract definition/creation. The top use case is agents, although speculate it’s used among investment managers and reinsurers as well.
STAR (Standards for Technology in Automotive Retail). STAR is used by car manufacturers and retailers to exchange information – about cars, no doubt, and probably some other things as well.
HR-XML, which is used by human resource departments for resumes and other kinds of employee information.
OTA (Open Travel Alliance). This is an XML schema used by usual-suspect players in the travel business – hotel chains, car renters, travel agents, and probably anybody else who deals in travel reservations.

I’ve heard at least about the financial services ones from other XML database vendors as well. I’m a little surprised that nothing from the health area made the top of the list.

Overview of IBM DB2 pureXML

Curt Monash — Sun, 05 Oct 2008 12:34:51 +0000

On August 29, I had a great call with IBM about DB2 pureXML (most of the IBM side of the talking was done by Conor O’Mahony and Qi Jin). I’m finally getting around to writing it up now. (The world of tabular data warehousing has kept me just a wee bit busy …)

As I write it, I see there are a considerable number of holes, but that’s the way it seems to go when researching XML storage. I’m also writing up a September call from which I finally figured out (I think) the essence of how MarkLogic Server works – but only after five months of trying. It turns out that MarkLogic works rather differently from DB2 pureXML. Not coincidentally, IBM and Mark Logic focus on rather different use cases for native XML storage.

What I understand so far about the basic DB2 pureXML architecture goes like this:

DB2 pureXML stores XML in “true hierarchical format.” Based on all the discussion of indexing, I’m guessing that the way it does this is somewhat similar to that in MarkLogic.
Unlike MarkLogic, DB2 pureXML gives you the choice of what tags to index on.
In a big difference from Marklogic, text search on DB2 pureXML involves two separate indexes – XML and text (the latter being of the usual inverted-list variety). You can text-search both contents and tags, with the usual CONTAINS semantics.
PureXML has a data store separate from the rest of DB2’s, notwithstanding IBM’s references to XML “columns.” DB2’s general datatype extensibility framework is not used; I don’t wholly understand why.
I neglected to ask how well DB2 backup, management tools, and so on extend to DB2 pureXML.
You can talk to DB2 pureXML via two data manipulation languages: SQL/XML, and XQuery. Both are compiled down to the same run-time instructions. IBM said there’s an abstraction layer sitting over both the relational store and the XML store that allows for this I don’t totally understand what that means, since presumably the SQL/XML starts out by being sent to DB2’s parser.

A big part of IBM’s XML business strategy is to support various (typically vertical market) XML standards. IBM has implemented support for these standards and made it freely downloadable. What does “support” mean? It surely starts with a DTD (Document Type Definition), and apparently also includes mappings to generic web services interfaces. It turns out that there are a lot of them, so I’m listing some in a separate post.

More generally, it seems that the sales and uses for IBM pureXML are concentrated in two main (overlapping) cases:

When XML was going to be used anyway. One big example of this is the case of the standards-based industry data interchanges. Another example is when pureXML, albeit disk-based, acts as a kind of quasi-cache or mini-MDM hub (Master Data Management) for WebSphere-based enterprise application integration (EAI). IBM reports that DB2 pureXML has been sold as an intermediate EAI data store at least once each in banking, retailing, health care, and insurance.
When schema flexibility is of great importance.

Experience teaches me that schema flexibility is a subject that can attract considerable flames, in the general vein of “Omigod! The relational model is perfect because it’s mathematically proven to ensure referential integrity!!” So I’ll split out the main discussion of that into yet another separate post, and keep going.

IBM actually breaks out the pureXML use cases into four main groups:

Transactional. This comprises the transactional logging of information that just happens to be XML, such as in financial services.
Forms-oriented. This comprises, for example, the tax authority use case.
Service bus acceleration. That’s a fancy phrase to cover both the standards-based interchanges and the other EAI-related uses.
Event-driven data warehousing. This one was kind of blurry to me. What I think it means is that if you have transactional data in XML, and you want to use it in near-real-time business intelligence, DB2 pureXML can help you with that.

#1, 3, and 4 seem to fit into my “When XML was going to be used anyway” category. Part of “Schema flexibility” matches #2; I’m not clear on where in IBM’s four buckets the rest of schema flexibility goes.

Finally, I asked directly in what areas there were significant numbers of DB2 pureXML customers. IBM offered two examples. One was financial services in general — especially in North America, notwithstanding the importance of the UNIFI standard in Europe. The other was health care data interchange outside the United States — especially in China, where regional and national centers are being established to more closely oversee local hospitals.

Related links

IBM kindly gave me permission to make available the slide presentation from our August 29 briefing. The last page has a large number of links to further IBM pureXML resources.
Conor O’Mahony has a good blog.
As noted above, I am putting up separate posts on standards-based data interchange and schema flexibility.