• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

Most of what’s confusing about the MarkLogic Hadoop Connector lies in two pairs of options it presents you:

• Hadoop can talk XQuery to MarkLogic. But alternatively, Hadoop can use a long-established simple(r) Java API for streaming documents into or out of a MarkLogic database.
• Hadoop can make requests to MarkLogic in MarkLogic’s normal mode of operation, namely to address any node in the MarkLogic cluster, which then serves as a “head” node for the duration of that particular request. But alternatively, Hadoop can use a long-standing MarkLogic option to circumvent the whole DBMS cluster and only talk to one specific MarkLogic node.

Otherwise, the whole thing is just what you would think:

• Hadoop can read from and write to MarkLogic, in parallel at both ends.
• If Hadoop is just writing to MarkLogic, there’s a good chance the process is properly called “ETL.”
• If Hadoop is reading a lot from MarkLogic, there’s a good chance the process is properly called “batch analytics.”

MarkLogic said that it wrote this Hadoop connector itself.

When I realized MarkLogic was claiming the ability to seamlessly integrate short-request and batch analytic processing, I asked about workload management. I gathered that:

• MarkLogic believes that MarkLogic 5 does a great job of granular workload monitoring.
• However, MarkLogic doesn’t have a strong workload management administrative interface. Rather, you may have to do workload management programmatically.

Overall, I think the MarkLogic Hadoop connector could prove pretty useful. The first question I ask somebody who wants to process relational data in Hadoop is “Why not just an analytic RDBMS?” But the natural use cases for MarkLogic are often ones in which you might as well do your analytics in Hadoop, including a 4 billion Word/PDF/image document insurance-industry example I recently encountered, and for which I favor MarkLogic over MongoDB or straight Hadoop alike.

• More-of-the-same in line with MarkLogic’s core positioning.
• A new bi-directional Hadoop connector.
• A free MarkLogic Express edition, limited in license terms more than in actual features, as per Slide 27 of the deck MarkLogic graciously supplied for me to post.

Also, MarkLogic is early with a feature that most serious DBMS vendors will soon have – support for tiered storage, with writes going first to solid-state storage, then being flushed to disk via a caching-style algorithm.* And as befits a sometime search-engine-substitute, MarkLogic has finally licensed a large set of document filters, from an Australian company called Isys. Apparently, the special virtue of the Isys filters is that they’re good at extracting not only text, but metadata as well.

*If there’s a caching algorithm that doesn’t contain a major element of LRU (Least Recently Used), I don’t recall ever hearing about it.

MarkLogic seems to have settled on a positioning that, although distressingly buzzword-heavy, is at least partly based upon reality. The real part includes:

• MarkLogic is a serious, enterprise-class DBMS (see for example Slide 12 of the MarkLogic deck) …
• … which has been optimized from the getgo for poly-structured data.
• MarkLogic can and does scale out to handle large amounts of data.
• MarkLogic is a general-purpose DBMS, suitable for both short-request and analytic tasks.
• MarkLogic is particularly well suited for analyses with long chains of “progressive enhancement” (MarkLogic’s favorite term when talking about derived data).
• MarkLogic often plays the role of a content assembler and/or search engine, and the people who use MarkLogic in those ways are commonly doing things that can be described as research and analysis.

Based on that reality, MarkLogic talks a lot about Volume, Velocity, Variety, Big Data, unstructured data, semi-structured data, and big data analytics.

My November, 2010 overview of MarkLogic technology remains pretty relevant. One correction, however: Node heterogeneity configurations, in which “data” and “evaluation” nodes reside on separate servers, are the exception rather than the rule.

Like Vertica, MarkLogic has laudably said that true academic researchers can get MarkLogic for free without the severe license restrictions. Free MarkLogic should be of particular interest to researchers who:

• Are studying natural networks or graphs, such as social networks or biological pathways. (This might be a fit in the social or biological sciences.)
• Are managing metadata for, say, a variety of disparate kinds of experimental files. (This might be a fit anywhere in the natural sciences.)
• Are managing actual documents, images, videos, etc., or data about such things. (This might be a fit in the humanities or social sciences.)

MarkLogic provided some disclosable financial substance by email, which I shall quote verbatim:

• MarkLogic has 45% revenue growth and 55-60% license growth year over year.
• We expect to finish this year with over $85 million in revenue, up from $55 million last year.

Arithmetical purists might note that 85/55 is more than 145%, but I’m just going to settle for the information I got and move on.

Edit: I posted separately about the MarkLogic Hadoop connector. As for that Hadoop connector – stay tuned for a short follow-up post, as writing about it now would not be convenient. (My backup discipline isn’t what it should be, and the only copy of my notes about that product is on a heavy tower computer in a house that doesn’t have working power.)

1. Confusion about text data management.
2. Choices for text data management (general and short-request).
3. Choices for text data management (analytic).

I’ve recently given widely varied advice about managing text (and similar files — images and so on), ranging from

Sure, just keep going with your old strategy of keeping .PDFs in the file system and pointing to them from the relational database. That’s an easy performance optimization vs. having the RDBMS manage them as BLOBs.

to

I suspect MongoDB isn’t heavyweight enough for your document management needs, let alone just dumping everything into Hadoop. Why don’t you take a look at MarkLogic?

Here are some reasons why.

There are three basic kinds of text management use case:

• Text as payload.
• Text as search parameter.
• Text as analytic input.

The simplest way to manage text electronically is to:

• Store it as whole documents — scanned images, .PDFs, word processing files, whatever.
• Find it only via fielded metadata, perhaps manually created — title, author, date, and so on.

For example, an application for college admission is accompanied by recommendation letters, transcripts, and so on; those are then moved around as dumb payloads, until such time as an admissions officer reads them. Most relational database management systems can manage BLOBs (Binary Large OBjects), but performance may be better if you leave the big objects outside the relational system. For text-as-payload, that way of managing documents can often suffice.

In other cases, the text may be so short that it naturally fits into a character field in a relational database. This is particularly likely when the text is typed in at the time of record creation, e.g. by a call center operator, a doctor, or a customer entering a support ticket. In such cases, leaving it under the management of an RDBMS makes perfect sense.

In many situations you actually want to search based on the context of the text. Unless you’re doing simple search on short text snippets in relational character fields, that generally calls for some kind of text index. Text indexes are generally found in text search engines. That said, however:

• Anything that can be done in a standalone text search engine can in principle also be integrated into relational DBMS and other data stores. (How well that works in practice is another matter.)
• Text search engines commonly index data in situ that they don’t actually manage.

In theory, then:

• You can store your text in your chosen DBMS or outside it, as pleases you.
• Your chosen DBMS can have a text search capability.
• This text search capability can be integrated with the query method used to get at the rest of the data managed by that DBMS.

That theory is commonly reflected in actual products, such as Oracle and DB2.

As so often, then, the choice of how to manage text comes down to issues such as performance, programming ease, and other components of total cost of ownership (or of some other general “goodness” metric, such as time-to-value). As a general rule, it seems:

• Text indexing inside relational DBMS has poorer performance than in, say, text search engines, often drastically so.
• BLOB management inside relational databases has poorer performance than leaving the files outside the DBMS’ purview.
• Relational DBMS do just fine at managing text strings up to, say, 2048 characters long.
• Tight integration between  text search and SQL is valuable in a few applications, but irrelevant to many others.

And so, to a first approximation:

• If you just have short text snippets, it can make sense to leave them in your relational database.
• If performance is not an issue, you can just leave your BLOBs in your relational database too.
• If performance is an issue, you probably want to have your larger text files outside your RDBMS’ control.

However, that phrasing assumes the default option is a relational DBMS, which may not be the case at all. Other choices include:

• Standalone text search engines. If you want the best available text search, get a search engine. But attempts to start with a search engine and wind up with an application platform have generally run into difficulty.
• Document-oriented (or other) NoSQL systems. The story here is surprisingly like that for relational DBMS. I’ve previously noted that document-oriented NoSQL systems manage objects, not “documents” in the ordinary sense of the word. Even so, conceptually they’re no less suited for management of true documents than relational DBMS are. I’d guess that the correlation between use cases involving true documents and use cases where document-oriented NoSQL is suitable is positive, but not very strong.
• MarkLogic. From one standpoint, MarkLogic is just a heavier-weight version of document-oriented NoSQL. But MarkLogic’s XML (and XQuery) orientation, tuned-for-years indexing, and built-in search engine put it on a different level for document management than the upstarts have reached.

I’ll cover the Hadoop option in the next, more analytically-focused post.

I hope I’ve demonstrated that there are appropriate use cases for each of:

• Letting documents be managed by the file system (and pointing to them from your preferred DBMS).
• Sticking documents straight into your preferred DBMS (SQL or non-SQL as the case may be).
• Using a specialty system such as MarkLogic (or of course, in some cases, an enterprise search engine).

And that’s even before we move on to analytically-oriented text data management.

1. Confusion about text data management.
2. Choices for text data management (general and short-request).
3. Choices for text data management (analytic).

There’s much confusion about the management of text data, among technology users, vendors, and investors alike. Reasons seems to include:

• The terminology around text data is inaccurate.
• Data volume estimates for text are misleading.
• Multiple different technologies are in the mix, including:
  • Enterprise text search.
  • Text analytics — text mining, sentiment analysis, etc.
  • Document stores — e.g. document-oriented NoSQL, or MarkLogic.
  • Log management and parsing — e.g. Splunk.
  • Text archiving — e.g., various specialty email archiving products I couldn’t even name.
  • Public web search — Google et al.
• Text search vendors have disappointed, especially technically.
• Text analytics vendors have disappointed, especially financially.
• Other analytic technology vendors ignore what the text analytic vendors actually have accomplished, and reinvent inferior wheels rather than OEM the state of the art.

Above all: The use cases for text data vary greatly, just as the use cases for simply-structured databases do.

There are probably fewer people now than there were six years ago who need to be told that text and relational database management are very different things. Other misconceptions, however, appear to be on the rise. Specific points that are commonly overlooked include:

• The terms “unstructured” or “semi-structured” data are inherently misleading. That’s why I favor “multi-structured” or “poly-structured” instead. (“Multi-structured” seems to be winning; e.g., it’s been adopted by Teradata and Teradata/Aster.)
• The “social media” text data any one enterprise brings in house isn’t all that much. For example, Attensity serves many different enterprises’ social media needs from a single 20-terabyte data store, and reports that no single enterprise has required as much as 1 terabyte of text yet. Text data may consume a lot of storage on spinning disks somewhere, but it’s not that big a factor in future DBMS industry growth. (That 20 terabyte figure does seem low.)
• Structured databases are typically worth a lot more per bit than other kinds. The most valuable electronic data, per-bit, is probably records of significant economic transactions — purchases, sales, money transfers, etc. The least valuable may be sensor log files, whose contents consist mainly of “Nothing going on here; ping you again in a minute.” Email logs, web interaction data and many other kinds fall somewhere in between. Highly valuable documents — such as signed contracts — generally persist in paper as well as electronic forms. Investors commonly overlook this point.
• The enterprise text search industry is screwed up.
  • FAST was a goofy company before it was acquired for far too much money by Microsoft.
  • Autonomy was a goofy company before it was acquired for far too much money by HP.
  • Google’s enterprise efforts are quiet.
  • The integration of text search and relational DBMS — e.g. at Oracle — has languished, with poor performance and evident lack of management attention.
  • Smaller text search vendors don’t seem to be getting a lot of traction — e.g., Coveo has a decent reputation, but when’s the last time you heard much about them? What has Attivio actually accomplished?
• Text analytics is a small business. Add up the revenue for Attensity, Clarabridge, Lexalytics, Temis, and all the others, and you might poke above $100 million, especially now that Attensity had a three-way merger. Then again, you might not.
• Even so, the text analytics vendors have developed sophisticated technology. In particular, you can use it to get a pretty good idea as to what people are writing about you, individually or as groups.

NoSQL is most easily defined by what it excludes: SQL, joins, strong analytic alternatives to those, and some forms of database integrity. If you leave all four out, and you have a strong scale-out story, you’re in the NoSQL mainstream.

• Thus, I’d say Cassandra, HBase, Mongo DB, and Couchbase are prime examples, in no particular order. Riak as well.
• I might have phrased that better if I’d used a different word than simply “strong” — but hey, there was a 140-character limit, and he was on deadline.

Using NoSQL can make sense when at least one of two things is paramount: low-cost scale-out or dynamic schemas.

• There are some seriously sensible use cases for dynamic schemas.
• “Low-cost” generally boils down to:
  • Performance.
  • Open source free-like-beer.
  • Not a lot of database administration.

I’ve generally given object-oriented DBMS vendors and also MarkLogic hard times whenever they consider saying they’re “NoSQL”. Reasons include:

• Closed source.
• Database administration overhead (even if you get good stuff for incurring that overhead, like MarkLogic’s comprehensive indexing).

Also, NoSQL started out being ACID-unfriendly.

What you give up are the query flexibility and the easily automatic data integrity of SQL-based systems. I should have added something about a mature ecosystem.

In the most recent live example, I influenced a client away from Cassandra and toward scale-out MySQL (dbShards and/or Schooner flavors, most likely). Part of the reason was the ability to do joins, which are useful in their application. Another part is that their development practices obviated any significant benefit from dynamic schemas. But perhaps the most important — or at least resonant — reason of all was that they really, really cared about .NET support.

• Derived data.
• Many-step processes to produce derived data.
• Schema evolution.
• Temporary data constructs.

So let’s dive in. 

When I started my discussion of derived data, I focused on five kinds:

Aggregates, when they are maintained, generally for reasons of performance or response time.

Calculated scores, commonly based on data mining/predictive analytics.

Text analytics.

The kinds of ETL (Extract/Transform/Load) Hadoop and other forms of MapReduce are commonly used for.

Adjusted data, especially in scientific contexts.

Later I added a sixth kind:

• Derived metadata, commonly for polystructured data sets (logs, text, images, video, whatever).

More kinds may yet follow.

In all cases, I was (and am) talking about data that is actually persisted into the database. Temporary tables — for example the kind frequently created by Microstrategy — are also important in data processing, as is temp space managed solely for the convenience of the DBMS. But neither are what I mean when I talk about “derived data.”

As I noted back in June, derived data naturally leads to schema evolution. You load data into an analytic database. You do some analysis and get some interesting results — interesting enough for you to want to keep them persistently. So you extend the schema to include them. You do more research; you discover something else interesting; you extend the schema again. Repeat as needed.

However, in no way is derived data the only source of analytic schema evolution. Duh. Sometimes you just have new kinds of information coming in. Of course, once it’s there, you may want to derive something from it. In marketing contexts, both parts of that might be true in spades.

When I mentioned all this to my clients at MarkLogic — which was my inspiration for the polystructured/metadata example — they perked up and said “Oh! Progressive enhancement.” Indeed, it’s long been the case that a simple text processing pipeline could have >15 steps of extraction; indeed, I learned about the “tokenization chain” in 1997. If all the “progression” in  the data enhancement occurs in a single processing run, that wouldn’t necessarily spawn much schema evolution. On the other hand, if you think of additional steps to add every now and then — in that case your schema might indeed evolve over time.

Somewhat similarly, Hadoop can be used to run “aggregation pipelines” of many 10s of steps. The output of the whole thing might be a relatively small number of fields. But again, if the number or nature of the fields changes over time, schemas will need to evolve accordingly.

So to sum up:

• Derived data — of multiple kinds — is very important.
• If you want to increase the value you get from derived data, you might need to evolve your schema accordingly.
• Data derivation happens to sometimes have long processing pipelines; those pipelines might happen to offer clues as how to do yet better at data derivation in the future; those improvements might happen to lead to schema evolution over time.

“Just the raw facts” analytic databases are, for the most part, obsolete.

… there’s no escaping the importance of derived/augmented/enhanced/cooked/adjusted data for analytic data processing. The five areas I have in mind are, loosely speaking:

• Aggregates, when they are maintained, generally for reasons of performance or response time.
• Calculated scores, commonly based on data mining/predictive analytics.
• Text analytics.
• The kinds of ETL (Extract/Transform/Load) Hadoop and other forms of MapReduce are commonly used for.
• Adjusted data, especially in scientific contexts.

Probably there are yet more examples that I am at the moment overlooking.

Well, I did overlook at least one category.

A surprisingly important kind of derived data is metadata, especially for large, poly-structured data sets. For example, CERN has vastly quantities of experiment sensor data, stored as files; just the metadata alone fills over 10 terabytes in an Oracle database. MarkLogic is big on storing derived metadata, both on the publishing/media and intelligence sides of the business.

Actually, what made me think of writing this post was a few conversations at MarkLogic’s April user conference. For example, MarkLogic likes to break lunch up into subject-specific tables, hosted either by a partner company, or by one of the analysts who is attending anyway. So they asked me to hold a table about having Hadoop and MarkLogic work together. When I showed up, I discovered that most of the users at the table worked for a single organization; what’s more, they were skeptical about the table’s discussion subject, and wanted to be see if I could persuade them otherwise. I gently pointed out that I hadn’t actually picked the subject, and asked them what their use cases might be like. Those turned out to be classified …

… but have no fear! Your hero thought quickly, and soon was holding forth about various ways one might combine the two technologies for various intelligence tasks. The one that finally struck a chord was — you guessed it! — metadata management. It seems they had colleagues with a lot of machine-generated data maintained in Hadoop and, upon reflection, they thought MarkLogic might be a good way to manage the metadata for same.

So should metadata management be handled relationally? Looking at my first three tests for when going relational is a slam-dunk choice:

• I don’t think the application suites exploiting derived metadata are complex enough to support a strong pro-relational bias.
• I don’t think the benefits of normalization are intense enough to mandate relational storage. (Also, since provenance matters, some of the traditional benefits of normalization are obviated — you may actually want out-of-date information in some cases.)
• There certainly are some cases where you can set up a fixed schema, have one row of metadata per object, and be happy. In those cases, a relational database likely suffices, and is probably the right choice, but …

… I’m not sure how numerous the cases are where a simple, fixed database design isn’t a good fit. Thoughts?

*Here I’m using the term “database” literally, rather than as a concise synonym for “database management system”. But see below.

Second, a more accurate distinction is not whether a database has one structure or none – it’s whether a database has one structure or many. The easiest way to see this is for databases that have clearly-defined schemas. A relational database has one schema (even if it is just the union of various unrelated sub-schemas); an XML database, however, can have as many schemas as it contains documents.

One small terminological problem is easily handled, namely that people don’t talk about true databases very often, at least when they’re discussing generalities; rather, they talk about data and DBMS.* So let’s talk of DBMS being “structured” singly or multiply or whatever, just as the databases they’re designed to manage are.

*And they refer to the DBMS as “databases,” because they don’t have much other use for the word.

All that said — I think that single vs. multiple database structures isn’t a bright-line binary distinction; rather, it’s a spectrum. For example: 

• IMS is the most structured DBMS of all. The data in an IMS database is in a hierarchy, and that’s that.
• CODASYL and other kinds of what used to be called network DBMS (before the word got so overloaded) — e.g. RDB, IDMS, or TOTAL — are/were almost as structured as IMS.
• Relational databases were invented because their structure was more flexible than that of linked-list databases. The whole point of relational DBMS is that you can view the data in a multitude of ways. Still, I see classical relational databases as being toward the single-structure end of the spectrum. (I say “classical” because Oracle and DB2 actually can manage combinations of XML, text, and traditional relational tables, if you choose.)
• A multivalue DBMS is a little more multi-structured than a relational one, because of how a field can be filled one or multiple times.
• eBay Singularity (as implemented on Teradata gear) has, in essence, two structures (that I know of). One structure is just the relational schema. The other is the structure you would get if each kind of name-value pair truly had its own column.
• A Splunk collection of log data can reasonably be said to have a different structure for every type or source of log. It further can be said to have multiple structures in somewhat the same way that eBay Singularity does.
• So-called document stores can be very multi-structured. MongoDB, Couchbase, et al. let you have a different structure for every document, if you choose. The same goes for XML-based MarkLogic.
• HBase and Cassandra are also very multi-structured. Theoretically, each record gets to decide which column sets it does or doesn’t fit into.

As a general rule — the more structures a database can have at once, the easier it is to change those structures, even on the fly (e.g., by inserting yet another bit of self-describing data). Thus, I sometimes use the term polystructured instead of multi-structured or multistructured. Thoughts as to which term I should choose going forward would be much appreciated.

As for an actual definition — well, here’s something I drafted 3 1/2 years ago but never published:

These problems with the relational paradigm are big enough to be worth coining a word for – polystructured. Polystructured data is data with structure that:

• Can be exploited to provide most of the benefits of a highly structured database (e.g., a tabular/relational one) …
• … but cannot be described in the concise, consistent form such highly structured systems require.

Specifically, we’ll call a database “polystructured” if it is characterized by at least two of the following:

1. Data suitable for being queried by simple predicate-based matching (e.g., equality to certain values, falling with in ranges, etc.)
2. (Other) data suitable for being queried by more complex matching (e.g., text search relevancy rankings)
3. Subsets that are more neatly structured than the whole.

Equivalently, we’ll just say that polystructured data is data that has considerable structure, but whose structure is in some important way unpredictable.

NoSQL document or “column” stores would satisfy #1 and #3, as would Splunk. MarkLogic would satisfy all three criteria. #1 + #2 is sort of like what happens when text queries are allowed to go against (groups of) relational columns … and the vagueness with which I’m saying that makes me suspect that at least the unbolded/first definition doesn’t really fly.

Finally, here’s what led up to those definitions (the whole thing is from the introduction to a never-completed white paper). Please forgive any anachronisms in it. A number of the points in it have also been addressed in posts here; e.g.,

• In December, 2005 I expounded on the mismatch between text data and the relational model.
• In June, 2010 I elucidated the variety of data that could go into an individual’s marketing-oriented profile.
• In February, 2008 I predicted that flexible-schema DBMS would gain share.

The case for polystructured data

Traditional computer databases amount to sets of records.   There usually are a limited number of record formats, which each instance of a particular format containing parallel kinds of information.  Business transactions, web page visits, instrument readings– whatever the nature of the information, application designers stick it into the simplest structure they think makes sense.

These records are arranged into a variety of data structures.

• Log files are widely used, especially to track web site visits, in other networking uses, and for other kinds of instrument readings.
• Computer user administration is commonly in LDAP (Lightweight Directory Access Protocol) format.
• There are still a lot of installations of legacy “linked-list” DBMS (DataBase Management Systems) such as IBM’s IMS.
• Some decision support applications use data in multidimensional arrays.

Even so, most new business applications are written over relational DBMS, in the well-known rows-and-tables paradigm.

There are good reasons for the dominance of the relational model and of rows and tables.  (Strictly speaking, “relational” equates neither to “rows and tables” nor to “SQL”, but in practice the three concepts are closely linked.) In particular:

• Data integrity is (fairly) easy to ensure.
• From some standpoints, relational databases are flexible; you can construct almost any kind of query, without having to do any kind of database reorganization (except perhaps for performance).
• SQL programmers are easy to find.
• There’s simply been much more engineering effort invested in making good relational DBMS than in any other kind.

But the relational database paradigm also has some major drawbacks.  Three of the big ones are:

• Queries must have strictly match/fail answers; there’s no natural way for a relational DBMS to handle “somewhat relevant” hits.
• Relational databases can get cumbersome when large fractions of the potential data happen to be missing. (Hence the decades-long debates about the problems with NULL values.)
• While you have good flexibility in querying against any particular data structure, you do have to predefine your structure before you start accepting input.

The last point is why you wind up with all those NULL values in the first place; if a kind of information can be in any record in a set, the database is set up to assume that its present in all of them.  Or if you normalize your database so highly as to avert missing values, then you wind up with a huge number of tables, making queries (and updates) complicated from both the programmer’s and the machine’s standpoint.

Text apps suffer from RDBMS’ inelegant handing of relevancy. What’s more, documents can have almost unlimited internal structures, in three senses:

1. They can have chapters, sections, subsections, sidebars, footnotes, and so on, in any combination.
2. Semantic references can link words, phrases, sentences, and paragraphs in a near-infinite number of ways.
3. Documents can explicitly contain fielded data, such as numbers, addresses, dates, or geo-encodings.

Another group of apps that suffer from RDBMS’ limitations are in the area of personalization and similar fine-grained marketing analysis. Analysis of web clicks throws away most kinds of path information.  Analysis of written or verbal communication isn’t well-integrated with that of fielded data.  Different customers and prospects give different kinds of contact information, and are “touched” by different marketing initiatives; current systems do a poor job of integrating all that scattered information.

So what other markets could MarkLogic pursue? Before Ken even started work, I sent over some thoughts. They included (but were not limited to): 

• Everybody now knows that not all problems require a relational DBMS. The NoSQL movement has seen to that.
• Not everybody agrees that “heavyweight” enterprise DBMS are needed for everything. The NoSQL movement has seen to that too.
• The “document”/”object” DBMS distinction has long been blurry. XML is full of documents, but they’re really objects. The same goes for the JSON/quasi-JSON objects of CouchDB/Couchbase and MongoDB. Object-oriented DBMS vendors have dabbled in XML on and off over the years because of technical similarity. Etc.
• MarkLogic has always focused on markets where the database truly was about documents in the conventional sense — especially long text documents — aka “content”. I always thought that focus was over-narrow.
• There are various cases where law, regulation, compliance etc. mandate the production of long text documents. I’m not sure MarkLogic has penetrated those as well as it could have.
• Graph DBMS technology is going nowhere fast, largely because nobody has solved the data distribution problem in cases big enough to need scale-out, and the technology isn’t obviously needed in single-server cases. (But see my post on Objectivity’s Infinite Graph.) Even so, graph-oriented apps are exploding, and MarkLogic should think about playing in the graph area, even if by acquisition.
• I think what I described in http://www.dbms2.com/2010/06/08/profile-of-revealed-preferences/ is non-relational and a very big market. Playing there with a “heavyweight” DBMS is of course a challenge.
• Coming over from Autodesk, Ken Bado hopefully knows more about the product data management business than I do.

It will be interesting to see what MarkLogic actually does.