Reporter: [Care to comment]?
CAM: SQL Server is an adequate product if you don’t mind being locked into the Microsoft stack. For example, the ColumnStore feature is very partial, given that it can’t be updated; but Oracle doesn’t have columnar storage at all.

Reporter: Is the lock-in overall worse than IBM DB2, Oracle?
CAM: Microsoft locks you into an operating system, so yes.

Reporter: Is this release something larger Oracle or IBM shops could consider as a lower-cost alternative a co-habitation scenario, in the event they’re mulling whether to buy more Oracle or IBM licenses?
CAM: If they have a strong Microsoft-stack investment already, sure. Otherwise, why?

Reporter: [How about] just cost?
CAM: DB2 works just as well to keep Oracle honest as SQL Server does, and without a major operating system commitment. For analytic databases you want an analytic DBMS or appliance anyway.

Best is to have one major vendor of OTLP/general-purpose DBMS, a web DBMS, a DBMS for disposable projects (that may be the same as one of the first two), plus however many different analytic data stores you need to get the job done.

By “web DBMS” I mean MySQL, NewSQL, or NoSQL. Actually, you might need more than one product in that area.

• “Deliver products that really meet customers’ desires and needs.”
• “Successfully convince them that you’re doing so …”
• “… at an attractive overall cost.”

Vendors mentioned here are Oracle, SAP, HP, and IBM. Anybody smaller got left out due to the length of this post. Among the bigger omissions were:

• salesforce.com (multiple subjects).
• SAS HPA.
• The evolution of Hadoop.

A (lingering) issue for SAP and Oracle alike

As I noted in January of this year, integration of business intelligence into operational apps is making very slow progress. Even so, it’s a huge part of the apparent strategy at SAP and Oracle alike, as well it should be. Much of the benefit from automating routine desk work has already happened. The areas ripest for exploitation are the ones where analytics are part of the equation.

Given the lack of tangible progress, why do I think this is a genuine area of Oracle and SAP emphasis? Three reasons of many are:

• Why else did SAP buy Business Objects?
• If they’re not trying to integrate operational apps and analytics, why else does SAP’s emphasis on HANA make sense?
• Without business intelligence in the picture, how does Oracle’s integrated-stack story promise any direct user benefits?*

*As opposed to IT concerns — integration, administration, TCO (Total Cost of Ownership), etc.

After so many years of disappointment, I’m not going to forecast 2012 as a pivotal year for the integration of business intelligence into operational applications. But if one of SAP or Oracle ever does get a significant BI/operational app integration advantage over the other, it could be a major competitive advantage in those application market segments that are still up for grabs. It also is an opportunity for both vendors to gain BI market share in their respective application customer bases.

A more urgent issue for SAP

SAP has put huge amounts of credibility on the line for HANA, the integration of two different and not particularly mature in-memory database technologies. So far, it is difficult to find evidence that HANA is robust enough for widespread adoption. Whether or not SAP can fix that is a huge open question, which could have significant impact on the course of several technology areas: applications, business intelligence, in-memory DBMS, and maybe even hardware.

Based on current information, which is admittedly partial, I’m a short-term pessimist on HANA. Longer-term, I’m on record as saying that traditional databases will eventually wind up in RAM. SAP will surely get that technology right some day, whether or not the way it does so has anything to do with present-day HANA code.

Four more issues for Oracle

Oracle’s ambitions are near-endless, and so also therefore is its list of execution challenges. Four in the analytics area that I find particularly interesting are:

• True hybrid columnar DBMS. I was guessing that Oracle, like Teradata, would announce true hybrid columnar the week of Oracle OpenWorld. I was wrong. But if Oracle can’t bring out true hybrid columnar DBMS functionality relatively soon, Exadata will lose credibility as a competitor to more specialized analytic DBMS.
• Oracle Exalytics. With Exalytics in the mix, Oracle’s technology stack has HANA-like potential. But will Exalytics even ship in 2012? (I think so.) Will it be good for much in the first release? (I’m skeptical.)
• Oracle’s Big Data Appliance. I’m skeptical both about Oracle’s NoSQL product — a favorable InfoWorld review notwithstanding — and Hadoop appliances. But if I’m wrong, and Oracle can successfully embrace/extend the new non-relational paradigms, then it really might regain control over the evolution of data management.
• Oracle’s Endeca acquisition — will Oracle prove me wrong and integrate Endeca effectively into its overall analytic product line? If it does, we might finally see effective text (and eventually speech) navigation of enterprise software. (But as with all Oracle issues cited here, this is something that probably won’t amount to much in 2012 even if it does later go well.)

Three issues for IBM

Like Oracle, IBM is a huge company with many ambitions and hence many execution challenges. The biggest of those is surely: How effective can IBM be at selling outside its existing customer base? I don’t hear as much competitively about IBM DataStage, IBM SPSS or now IBM Netezza as I did when their vendors were independent companies. Even Cognos may not be much of an exception to the rule, although it has its own large customer base outside of IBM’s traditional one. (To lesser extents , the same is of course true of Netezza and numerous other IBM acquisitions.)

Another general issue for IBM is substantively integrating its various product lines, at least to the extent that makes sense. DB2/Netezza integration sounds good, but even that is a matter more of product marketing (the admirable part of that discipline) more than of actual technology. Other integrations (e.g. Cognos/DB2 in various bundles) have tended toward the dubious side.*

*I’m still waiting for IBM to get back to me with examples of how Cognos/DB2 joint tuning amounts to anything. It’s been more than a year, so I’m glad I didn’t hold my breath.

In a somewhat narrower vein, I wonder: Will IBM be able to gain traction for InfoSphere Streams? And if so, when and where will the traction be?

Will HP screw up Vertica?

Vertica has a very attractive product offering. It’s perhaps the most scalable analytic DBMS outside of Teradata, running on the hardware of your reasonable choice.  It’s also the one I recommend most often to clients in the 1-50 terabyte range.

So far HP doesn’t seem to have done much to leadfoot Vertica. (About all I’ve heard from competitors is that Vertica seems to have faded somewhat in the financial services market, and there could be multiple explanations if that is indeed true.) But if HP Vertica does somehow manage to botch things, opportunities will open up for a range of columnar analytic DBMS competitors.

• The upshot of my what to call CEP thread in August was that “streaming” and “event processing” are not the same concept, but it so happens that they have the most traction where they intersect. That said, I both observe and endorse an apparent shift from “event” to “stream” as the core of the terminology, in a reversal of my opinion of several years ago.
• IBM continues to throw a lot of resources at its System S/ InfoSphere Streams product, but I haven’t heard yet of much marketplace success. That said, I believe IBM is still pretty serious about Streams, as one would expect from an effort whose code name so cheekily references System R. In particular, Streams shows up prominently on IBM’s top-level analytic architecture slide.
• Sybase recently released its ESP (Event Stream Processor) 5.0, which it says is the full merger of the Aleri and Coral8 predecessors. You can still get Sybase ESP without buying into the full Sybase RAP stack, and Sybase has no plans to change that.
• Sybase has discontinued all the business intelligence types of products Aleri and Coral8 were developing. Rather, Sybase is OEMing Panopticon, which it reports has been well received. Other than the discontinuation of the BI efforts, there seem to be few Aleri or Coral8 features missing from the merged Sybase ESP product.
• Truviso continues to be out of the picture.
• I have more to say about StreamBase separately.
• I have more to say about Sybase and IBM, which I’ll get to when I can.
• I have nothing new on Progress Apama. I also know little about any of the open source efforts.

Meanwhile, if you want to see technically nitty-gritty posts about the CEP/streaming area, you may want to look at my CEP/streaming coverage circa 2007-9, based on conversations with (among others) Mike Stonebraker, John Bates, and Mark Tsimelzon.

My favorite relational OLTP scale-out choice these days is the SchoonerSQL/dbShards partnership. Schooner Information Technology (SchoonerSQL) and Code Futures (dbShards) are young, small companies, but I’m not too concerned about that, because the APIs they want you to write to are just MySQL’s. The main scenarios in which I can see them failing are ones in which they are competitively leapfrogged, either by other small competitors – e.g. ScaleBase, Akiban, TokuDB, or ScaleDB — or by Oracle/MySQL itself. While that could suck for my clients Schooner and Code Futures, it would still provide users relying on MySQL scale-out with one or more good product alternatives.

Relying on non-MySQL NewSQL startups, by way of contrast, would leave me somewhat more concerned. (However, if their code is open sourced. you have at least some vendor-failure protection.) And big-vendor scale-out offerings, such as Oracle RAC or DB2 pureScale, may be more complex to deploy and administer than the MySQL and NewSQL alternatives.

• Platform Computing started ~20 years ago.
• Platform Computing claimed close to $100 million in revenue and >500 people.
• (This is Platform Computing’s most famous splash to date.) Platform Computing technology underlies SAS Institute’s preferred method of parallelization, which may variously be called:
  • SAS Grid Manager (the more or less official brand name).
  • SAS HPA (High Performance Analytics), sort of an alternate brand name.
  • MPI (Message Passing Interface), the industry’s name for the underlying semantics/syntax/API.
• Platform Computing’s original business was scientific grid computing.
• Platform Computing’s second major business was its “Symphony” product line. According to Platform Computing, Symphony:
  • Debuted 6-7 years ago.
  • Is more commercially oriented.
  • Is what supports SAS HPA.
  • SAS aside, has been sold to Wall Street and so on.
  • Is sometimes used in conjunction with CEP/streaming, mainly for backtesting.
  • Can be used to build global (parallel) persistent memory for R.
• (This is probably why IBM is buying Platform Computing.) Platform Computing’s has a new MapReduce offering that:
  • Is based on Symphony.
  • Shipped last July, except that early access was a couple months before that.
  • Is focused on:
    • Lowering the latency of MapReduce.
    • Consolidating multiple MapReduce use cases into one high(er)-utilization cluster.
    • Offering workload management in support of those goals.
    • Reliability, availability, predictability, puppies, kittens, and apple pie.
• Is most specifically a MapReduce run-time engine, with other stuff beyond that.

Unfortunately, I’m not precisely clear as to how tied this offering is to Hadoop, but using it with Hadoop is at least the base case. But Platform Computing did say:

• It can support multiple virtual Hadoop clusters, which can be grown or shrunk at will.
• Non-Hadoop workloads can be mixed in.

Platform Computing said that key technical benefits of this offering included:

• 1-3 seconds to start a job, vs. 40-50 in generic Hadoop.
• Automatic recovery of JobTracker nodes.
• Failover for NameNodes.
• Workload management that:
  • Manages all of CPU, I/O, and RAM (this is quickly becoming an industry standard level of capability, although I’m judging more by the standards of the analytic DBMS world).
  • Monitors but doesn’t actively manage network resources.
  • Can reprioritize jobs that are in flight. (Also an industry-standard capability.)

This conflation of scientific, commercial analytic, streaming, and MapReduce is right in IBM’s philosophical wheelhouse. I base that comment on, among other factors:

• How IBM positions “Big Insights”.
• IBM’s “smart consolidation” picture/pitch (which I really should get around to posting).
• The fuss IBM makes about Watson, Blue Gene, and so on.

The IBM acquisition probably obviates a lot of Platform Computing’s previous business comments, but at the time they included:

• POCs (Proofs of Concept):
  • Mainly in financial services, government, and telecom.
  • At both existing customers and new prospects.
  • Typically running 30-50 nodes, 2-50 terabytes.* The smallest databases evidently tended to be an financial services firms.
• Pricing that was starting out:
  • Perpetual license: $3450/server, 21% annual maintenance after the first year.
  • Subscription: $2070/server annually, or $3070 with HDFS support bundled in.

*1 terabyte or less per node is probably the lowest data-per-node figure I’ve heard for anything Hadoop-like — even below Hadapt, and well below what Cloudera and Hortonworks usually see.

• In May I surveyed relational vs. non-relational pros and cons at some length.
• Last November I mused about when it might be OK to do without joins.
• The question is implicit in a variety of posts about, say, document-oriented or object-oriented DBMS.

Before going further in that vein, I’d like to do a quick review of what E. F. “Ted” Codd was getting at with the relational model in the first place. 

The first sentence of Codd’s famous 1970 paper introducing the relational database concept reads:

Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation).

In modern terms, that means “all you have to know to use the database is its logical schema; you don’t need to know anything about its physical representation.”

Over the next 15 years, Codd’s thinking — and his employer IBM’s technology — evolved to the point that Codd proposed 12 rules for a relational DBMS, the three most fundamental of which are:

Foundation Rule
A relational database management system must manage its stored data using only its relational capabilities.

Information Rule
All information in the database should be represented in one and only one way — as values in a table.

Guaranteed Access Rule
Each and every datum (atomic value) is guaranteed to be logically accessible by resorting to a combination of table name, primary key value and column name.

I.e., Codd was positively asserting that a database should have a fixed logical schema, in a tabular form. The clear implication was that programmers could or should be able to write anything they wanted to against that schema, without database performance being unduly compromised.

Of course, things never quite worked out that way. For most of the history of tabular DBMS, the best-performing short-request and analytic DBMS have been designed quite differently from each other.* Non-relational systems — from IBM’s own IMS to various object-oriented DBMS — outperformed relational DBMS on particular applications. Designers of high-performance applications were sensitive to the database’s physical design, sometimes even going to the extreme of non-transparent sharding. But on the whole, it was generally agreed that programming against a fixed logical schema is a good thing.

*Codd acknowledged this himself by promoting multidimensional OLAP over traditional RDBMS. (I regard the multidimensional/relational divide to be a distinction without significant difference; it’s all just fixed-logical-schema tabular processing with different data manipulation languages.)

In my next post, I’ll return to the subject of why fixed schemas might not always be such a good idea after all.

• Packaging up a Cloudera distribution of Apache Hadoop. This distribution doesn’t have proprietary code; it’s just packaged by Cloudera from Apache projects (with a decent minority of the code happening to have been contributed by Cloudera engineers).
• Paid subscription support for Apache Hadoop and, in connection with that …
• …  proprietary software that all support customers automatically get. There are two points to this proprietary software:
  • It adds value for the customer.
  • It makes Cloudera’s support job easier.
• Professional services around Hadoop.
• Training and conferences around Hadoop, which probably don’t generate all that much money, but are great marketing in terms of visibility, thought leadership, and lead generation.

Hortonworks spun out of Yahoo last week, with parts of the Cloudera business model, namely Hadoop support, training, and I guess conferences. Hortonworks emphatically rules out professional services, and says that it will contribute all code back to Apache Hadoop. Hortonworks does grudgingly admit that it might get into the proprietary software business at some point — but evidently hopes that day will never actually come.

Hortonworks’ two main initial marketing messages — and there’s some synergy between these — boil down to:

• Open source purism
• “We have most of the Hadoop developers, so we’re better”*

Frankly, the open source purism part sounds like doubletalk to me, in that Hortonworks has trouble articulating what supposedly-less-pure Cloudera does wrong that Hortonworks will do better. However, I’ve been hearing for a long time that Yahoo’s MapReduce developers feel very strongly about open source, so perhaps this is in part an emotional issue for them. More substantively, it fits well with the pro-Hortonworks story I’ve outlined below.

*”We have most of the Hadoop developers” seems fairly defensible, give or take dueling definitions of “committer,” “core developer,” “patch” or for that matter “Hadoop.”

The other branch of the Hortonworks marketing message can be lampooned as “We’re the right folks to identify your bugs, since we’re probably the ones who put them there in the first place.” More darkly, that pitch could be “If you want the bugs fixed that bother you, we’re the ones who have control over whether or not that happens.” Well, maybe. But I also see Cloudera having a couple years experience supporting Hadoop, as well as shipping some code that perhaps makes Hadoop more supportable.

That’s the skeptical view. A more favorable view of Hortonworks’ prospects would go something like this:

• One version of Apache Hadoop is plenty.
• Cloudera (and arguably other Hadoop platform software vendors) sell capabilities that will soon be eclipsed by core Apache Hadoop. Folks should just please wait.
• Now that Hortonworks is an independent company focused on the task, it will speedily solve the packaging problems that have made Cloudera’s Hadoop distribution (perceived to be) necessary.
• Yahoo and IBM both back Hortonworks’ approach. That’s got to count for something.
• Apache Hadoop will be quickly enhanced, and Hortonworks will be driving the enhancements. Hortonworks simply is the top Hadoop authority.

We’ll see. Cloudera’s been around for a couple years, has smart people, and by definition has no technical inferiority to Hortonworks (since it has access to all Hortonworks’ code). What’s more, it will be a long time before Hadoop technology is so mature that there’s nothing left to do; add-on software should long prove to be useful. As for “We’re purer about open source than the other guys” — well, I’m dubious that that will turn out to be a great marketing message.

And so I think Cloudera is the early favorite in the competition. But perhaps Hadoop users will be able to play Cloudera and Hortonworks off  against each other in price negotiations. Perhaps, notwithstanding my skepticism about Hadoop appliances, some hardware vendors will play them against each other for appliance partnerships.

Meanwhile, whatever else happens, I’m pretty psyched about some enhancements the Hortonworks folks plan to lead for Hadoop.

Related links

• A Hortonworks/Apache Hadoop slide deck Hortonworks graciously allowed me to post
• Cloudera’s post about it’s recent 3.5 release of Cloudera Enterprise
• Pros and cons of professional services efforts at young software companies

Let’s try eight categories instead. While no categorization is ever perfect, these each have at least some degree of technical homogeneity. Figuring out which types of analytic database you have or need — and in most cases you’ll need several — is a great early step in your analytic technology planning. 

Enterprise data warehouse (Full or partial)

• Kinds of data likely to be included: All, but especially operational
• Likely use styles: All
• Canonical example: Central EDW for a big enterprise
• Stresses: Concurrency, reliability, workload management

The enterprise data warehouse (EDW) ideal says that you copy all your data into one place, and drive all decision-making from there. Full EDWs are pipedreams. Still, a partial EDW makes sense for most large enterprises, and many indeed already have one. The first product lines to consider for classical EDWs are Teradata, DB2, Exadata, and maybe Microsoft SQL Server, especially if you’re going to stress concurrency and/or operational use cases.

Traditional data mart

• Kinds of data likely to be included: All
• Likely use styles: Business intelligence, budgeting/consolidation, investigative
• Examples: Reporting servers, planning/consolidation servers, anything MOLAP, etc.
• Stresses: Performance, concurrency, TCO

Whether or not you have something like an enterprise data warehouse, it’s common to have lighter-weight data marts as well. A traditional data mart might drive reports and dashboards. Or it might be specialized for budgeting, planning, and/or consolidation.  Some investigative analytics may be in the mix as well.

Any DBMS that can support an EDW can also support a data mart, but it may not be the most cost-effective way to do so. Columnar DBMS might have more attractive performance and TCO (Total Cost of Ownership); the same goes for Netezza. Some of them — e.g. Sybase IQ and Vertica — have excellent track records in concurrent usage as well. Ted Codd pushed what amounts to MOLAP (Multidimensional OnLine Analytic Processing) systems for these use cases. But relational DBMS commonly do a better job, which is one reason most major MOLAP products have wound up at RDBMS companies.

Investigative data mart — agile

• Kinds of data likely to be included: All, especially customer-centric
• Likely use styles: Investigative
• Canonical example: A few analysts getting a few TB to examine
• Stresses: Ease of setup/load, ease of admin, price/performance

Besides the traditional data mart, there are at least two other kinds. Both are focused on investigative analytics, but they’re differentiated by database size.

If you have just a few analysts,* looking at no more than a few terabytes of data (perhaps even just some gigabytes) — and if that data is “single-subject” and fairly homogenous — your watchwords should be “cheap”, “easy”, and “fast”. You don’t need to invest in much hardware, in expensive software, in much administrative effort (the analysts can be their own DBAs),  nor should you endure much set-up time. Just grab a product, grab some data, and start running queries (or extracts into the statistical tool of your choice).

*If you have dozens or even hundreds of analysts hitting the same database, you’re probably back to the more concurrency-oriented scenarios outlined above.

Infobright is often cost-effective among columnar analytic DBMS. Other vendors might cut you a price break as well. If you have multiple terabytes of data, don’t rule out Netezza’s lowest-end products (even if they’d really rather sell you something bigger). Or, if you’re in the sub-terabyte range, maybe you can get by with an in-memory BI tool such as QlikView, and not do anything special on the DBMS side at all.

Investigative data mart — big

• Kinds of data likely to be included: All, especially customer-centric, logs, financial trade, scientific
• Likely use styles: Investigative
• Canonical example: Single-subject 20 TB – 20 PB relational database
• Stresses: Performance, scale-out, analytic functionality

But if you’re looking at tens of terabytes of relational data, or even more, you really do have a “big data” problem. Performance and scalability are major challenges, usually best addressed by MPP (Massively Parallel Processing) systems, such as Netezza, Vertica, Aster Data, ParAccel, Teradata, or Greenplum. Performance POCs (Proofs Of Concept) are a big part of the buying process. Vendor price negotiations are crucial too.

Actually, in the low tens of terabytes you might be able to get away with a shared-disk system that has excellent compression — e.g., columnar products like Sybase IQ, Infobright, or SAND, rather than just Vertica and ParAccel.

Assuming you have affordable, scalable query performance, the competitive differentiator can switch to additional analytic functionality. Aster, Netezza, ParAccel, Vertica, and Greenplum either offer full analytic platforms, or seem to be on the path to doing so. Teradata, which now owns Aster Data, offers substantial built-in analytic capability in its traditional products as well, and the same goes for Sybase IQ.

Continued in Part 2, where we cover some of the more difficult use cases.

An enterprise data warehouse should:

• Manage data to high standards of accuracy, consistency, cleanliness, clarity, and security.
• Manage all the data in your organization.

Pick ONE.

IBM’s main theme at the Enzee Universe conference has been to say the same thing.

Merv Adrian’s talk at the same conference made it clear that Gartner feels the same way, as does he personally. Indeed, like me, he’s racked up multiple decades of industry experience without ever finding a single theoretically ideal grand central EDW.

Forrester Research has been a little less clear on the point, but generally seems to be on the correct side of the issue as well.

If somebody is still saying that one central enterprise data warehouse can hold all the information or data you need on which to base your business decisions, they’re probably not somebody you should be listening to very hard.

Is that clear, or should I hammer home the point even harder?

*Here I’m using the term “database” literally, rather than as a concise synonym for “database management system”. But see below.

Second, a more accurate distinction is not whether a database has one structure or none – it’s whether a database has one structure or many. The easiest way to see this is for databases that have clearly-defined schemas. A relational database has one schema (even if it is just the union of various unrelated sub-schemas); an XML database, however, can have as many schemas as it contains documents.

One small terminological problem is easily handled, namely that people don’t talk about true databases very often, at least when they’re discussing generalities; rather, they talk about data and DBMS.* So let’s talk of DBMS being “structured” singly or multiply or whatever, just as the databases they’re designed to manage are.

*And they refer to the DBMS as “databases,” because they don’t have much other use for the word.

All that said — I think that single vs. multiple database structures isn’t a bright-line binary distinction; rather, it’s a spectrum. For example: 

• IMS is the most structured DBMS of all. The data in an IMS database is in a hierarchy, and that’s that.
• CODASYL and other kinds of what used to be called network DBMS (before the word got so overloaded) — e.g. RDB, IDMS, or TOTAL — are/were almost as structured as IMS.
• Relational databases were invented because their structure was more flexible than that of linked-list databases. The whole point of relational DBMS is that you can view the data in a multitude of ways. Still, I see classical relational databases as being toward the single-structure end of the spectrum. (I say “classical” because Oracle and DB2 actually can manage combinations of XML, text, and traditional relational tables, if you choose.)
• A multivalue DBMS is a little more multi-structured than a relational one, because of how a field can be filled one or multiple times.
• eBay Singularity (as implemented on Teradata gear) has, in essence, two structures (that I know of). One structure is just the relational schema. The other is the structure you would get if each kind of name-value pair truly had its own column.
• A Splunk collection of log data can reasonably be said to have a different structure for every type or source of log. It further can be said to have multiple structures in somewhat the same way that eBay Singularity does.
• So-called document stores can be very multi-structured. MongoDB, Couchbase, et al. let you have a different structure for every document, if you choose. The same goes for XML-based MarkLogic.
• HBase and Cassandra are also very multi-structured. Theoretically, each record gets to decide which column sets it does or doesn’t fit into.

As a general rule — the more structures a database can have at once, the easier it is to change those structures, even on the fly (e.g., by inserting yet another bit of self-describing data). Thus, I sometimes use the term polystructured instead of multi-structured or multistructured. Thoughts as to which term I should choose going forward would be much appreciated.

As for an actual definition — well, here’s something I drafted 3 1/2 years ago but never published:

These problems with the relational paradigm are big enough to be worth coining a word for – polystructured. Polystructured data is data with structure that:

• Can be exploited to provide most of the benefits of a highly structured database (e.g., a tabular/relational one) …
• … but cannot be described in the concise, consistent form such highly structured systems require.

Specifically, we’ll call a database “polystructured” if it is characterized by at least two of the following:

1. Data suitable for being queried by simple predicate-based matching (e.g., equality to certain values, falling with in ranges, etc.)
2. (Other) data suitable for being queried by more complex matching (e.g., text search relevancy rankings)
3. Subsets that are more neatly structured than the whole.

Equivalently, we’ll just say that polystructured data is data that has considerable structure, but whose structure is in some important way unpredictable.

NoSQL document or “column” stores would satisfy #1 and #3, as would Splunk. MarkLogic would satisfy all three criteria. #1 + #2 is sort of like what happens when text queries are allowed to go against (groups of) relational columns … and the vagueness with which I’m saying that makes me suspect that at least the unbolded/first definition doesn’t really fly.

Finally, here’s what led up to those definitions (the whole thing is from the introduction to a never-completed white paper). Please forgive any anachronisms in it. A number of the points in it have also been addressed in posts here; e.g.,

• In December, 2005 I expounded on the mismatch between text data and the relational model.
• In June, 2010 I elucidated the variety of data that could go into an individual’s marketing-oriented profile.
• In February, 2008 I predicted that flexible-schema DBMS would gain share.

The case for polystructured data

Traditional computer databases amount to sets of records.   There usually are a limited number of record formats, which each instance of a particular format containing parallel kinds of information.  Business transactions, web page visits, instrument readings– whatever the nature of the information, application designers stick it into the simplest structure they think makes sense.

These records are arranged into a variety of data structures.

• Log files are widely used, especially to track web site visits, in other networking uses, and for other kinds of instrument readings.
• Computer user administration is commonly in LDAP (Lightweight Directory Access Protocol) format.
• There are still a lot of installations of legacy “linked-list” DBMS (DataBase Management Systems) such as IBM’s IMS.
• Some decision support applications use data in multidimensional arrays.

Even so, most new business applications are written over relational DBMS, in the well-known rows-and-tables paradigm.

There are good reasons for the dominance of the relational model and of rows and tables.  (Strictly speaking, “relational” equates neither to “rows and tables” nor to “SQL”, but in practice the three concepts are closely linked.) In particular:

• Data integrity is (fairly) easy to ensure.
• From some standpoints, relational databases are flexible; you can construct almost any kind of query, without having to do any kind of database reorganization (except perhaps for performance).
• SQL programmers are easy to find.
• There’s simply been much more engineering effort invested in making good relational DBMS than in any other kind.

But the relational database paradigm also has some major drawbacks.  Three of the big ones are:

• Queries must have strictly match/fail answers; there’s no natural way for a relational DBMS to handle “somewhat relevant” hits.
• Relational databases can get cumbersome when large fractions of the potential data happen to be missing. (Hence the decades-long debates about the problems with NULL values.)
• While you have good flexibility in querying against any particular data structure, you do have to predefine your structure before you start accepting input.

The last point is why you wind up with all those NULL values in the first place; if a kind of information can be in any record in a set, the database is set up to assume that its present in all of them.  Or if you normalize your database so highly as to avert missing values, then you wind up with a huge number of tables, making queries (and updates) complicated from both the programmer’s and the machine’s standpoint.

Text apps suffer from RDBMS’ inelegant handing of relevancy. What’s more, documents can have almost unlimited internal structures, in three senses:

1. They can have chapters, sections, subsections, sidebars, footnotes, and so on, in any combination.
2. Semantic references can link words, phrases, sentences, and paragraphs in a near-infinite number of ways.
3. Documents can explicitly contain fielded data, such as numbers, addresses, dates, or geo-encodings.

Another group of apps that suffer from RDBMS’ limitations are in the area of personalization and similar fine-grained marketing analysis. Analysis of web clicks throws away most kinds of path information.  Analysis of written or verbal communication isn’t well-integrated with that of fielded data.  Different customers and prospects give different kinds of contact information, and are “touched” by different marketing initiatives; current systems do a poor job of integrating all that scattered information.