• Splunk is first and foremost an analytic DBMS …
• … used to manage logs and similar multistructured data.
• Splunk’s DML (Data Manipulation Language) is based on text search, not on SQL.
• Splunk has extended its DML in natural ways (e.g., you can use it to do calculations and even some statistics).
• Splunk bundles some (very) basic, Splunk-specific business intelligence capabilities.
• The paradigmatic use of Splunk is to monitor IT operations in real time. However:
  • There also are plenty of non-real-time uses for Splunk.
  • Splunk is proudest of its growth in non-IT quasi-real-time uses, such as the marketing side of web operations.

As in any release, a lot of Splunk 4.3 is about “Oh, you didn’t have that before?” features and Bottleneck Whack-A-Mole performance speed-up. One performance enhancement is Bloom filters, which are a very hot topic these days. More important is a switch from Flash to HTML5, so as to accommodate mobile devices with less server-side rendering. Splunk reports that its users — especially the non-IT ones — really want to get Splunk information on the tablet devices. While this somewhat contradicts what I wrote a few days ago pooh-poohing mobile BI, let me hasten to point out:

• Splunk is used for a lot of (quasi) real-time monitoring.
• Splunk’s desktop user interfaces are, by BI standards, quite primitive.

That’s pretty much the ideal scenario for mobile BI: Timeliness matters and prettiness doesn’t.

Hmm. Maybe StreamBase LiveView needs a mobile option as well …

Splunk’s basic use is to take the text string that is a log and make sense of it. But Splunk now also supports JSON structures. It does this via something called spath, which as you might guess from the name has XPath similarities. That probably bore more discussion than we found the time to have.

By the way: If you’re interested in BI over XML, that’s what my former clients at Skytide were founded to do, before they pivoted a bit. I don’t think those capabilities have disappeared from the product.

Splunk has graciously allowed me to post a slide deck. More stuff in there, including quotes from a customer — Expedia — that has 2700 Splunk users.

• More-of-the-same in line with MarkLogic’s core positioning.
• A new bi-directional Hadoop connector.
• A free MarkLogic Express edition, limited in license terms more than in actual features, as per Slide 27 of the deck MarkLogic graciously supplied for me to post.

Also, MarkLogic is early with a feature that most serious DBMS vendors will soon have – support for tiered storage, with writes going first to solid-state storage, then being flushed to disk via a caching-style algorithm.* And as befits a sometime search-engine-substitute, MarkLogic has finally licensed a large set of document filters, from an Australian company called Isys. Apparently, the special virtue of the Isys filters is that they’re good at extracting not only text, but metadata as well.

*If there’s a caching algorithm that doesn’t contain a major element of LRU (Least Recently Used), I don’t recall ever hearing about it.

MarkLogic seems to have settled on a positioning that, although distressingly buzzword-heavy, is at least partly based upon reality. The real part includes:

• MarkLogic is a serious, enterprise-class DBMS (see for example Slide 12 of the MarkLogic deck) …
• … which has been optimized from the getgo for poly-structured data.
• MarkLogic can and does scale out to handle large amounts of data.
• MarkLogic is a general-purpose DBMS, suitable for both short-request and analytic tasks.
• MarkLogic is particularly well suited for analyses with long chains of “progressive enhancement” (MarkLogic’s favorite term when talking about derived data).
• MarkLogic often plays the role of a content assembler and/or search engine, and the people who use MarkLogic in those ways are commonly doing things that can be described as research and analysis.

Based on that reality, MarkLogic talks a lot about Volume, Velocity, Variety, Big Data, unstructured data, semi-structured data, and big data analytics.

My November, 2010 overview of MarkLogic technology remains pretty relevant. One correction, however: Node heterogeneity configurations, in which “data” and “evaluation” nodes reside on separate servers, are the exception rather than the rule.

Like Vertica, MarkLogic has laudably said that true academic researchers can get MarkLogic for free without the severe license restrictions. Free MarkLogic should be of particular interest to researchers who:

• Are studying natural networks or graphs, such as social networks or biological pathways. (This might be a fit in the social or biological sciences.)
• Are managing metadata for, say, a variety of disparate kinds of experimental files. (This might be a fit anywhere in the natural sciences.)
• Are managing actual documents, images, videos, etc., or data about such things. (This might be a fit in the humanities or social sciences.)

MarkLogic provided some disclosable financial substance by email, which I shall quote verbatim:

• MarkLogic has 45% revenue growth and 55-60% license growth year over year.
• We expect to finish this year with over $85 million in revenue, up from $55 million last year.

Arithmetical purists might note that 85/55 is more than 145%, but I’m just going to settle for the information I got and move on.

Edit: I posted separately about the MarkLogic Hadoop connector. As for that Hadoop connector – stay tuned for a short follow-up post, as writing about it now would not be convenient. (My backup discipline isn’t what it should be, and the only copy of my notes about that product is on a heavy tower computer in a house that doesn’t have working power.)

There really are three major points to NoSQL.

• Dynamic schemas. This is the only one of the three that truly depends on NoSQL.
• Scale-out short-request processing. If you want to scale out efficiently at high request volumes, you’re best off not using all the flexibility SQL/relational DBMS offer. (In particular, you don’t want to do cross-node joins). Not coincidentally, a number of the best scale-out offerings were built to be NoSQL.
• Open source. Doing a relational DBMS is a big project. It seems easier to build NoSQL ones.

Oracle can address the latter two points as aggressively as it wishes via MySQL. It so happens I would generally recommend MySQL enhanced by dbShards, Schooner, and/or dbShards/Schooner, rather than Oracle-only MySQL … but that’s a detail. In some form or other, Oracle’s MySQL is a huge player in the scale-out, open source, short-request database management market.

So that leaves us with dynamic schemas. Oracle has at least four different sets of technology in that area:

• As Workday noticed years ago, MySQL can be used as a functional, basic key-value store.
• Oracle also has XML-based Berkeley DB/SleepyCat kicking around.*
• The XML extensions to Oracle’s core DBMS could be alleged to have a dynamic schema/NoSQL flavor. (Blech.)
• A dynamic schema argument could also be made for object-oriented DBMS technology. While Oracle doesn’t to my knowledge exactly sell that, it does have the Tangosol Coherence line of technology, with a potentially similar programming model.

If Oracle is now refreshing and rebranding one or more of these as “NoSQL”, there’s no reason to view that as a big deal at all.

*That’s Mike Olson’s former company, if you’re keeping score at home.

• HP is buying Autonomy.
• HP is pulling back from WebOS.
• HP may spin off its PC business altogether.

On a high level, this means:

• HP is doubling down on enterprise IT.
• HP is taking a more software-centric approach to the enterprise IT business.
• HP is backing away from the consumer electronics business.
• HP in particular is backing away from the generic desktop/laptop PC business, which may with only moderate exaggeration be regarded as:
  • The intersection of the enterprise IT and consumer electronics businesses.
  • The least attractive sector of each.

My coverage of Autonomy isn’t exactly current, but I don’t know of anything that contradicts long-time competitor* Dave Kellogg’s skeptical view of Autonomy. Autonomy is a collection of businesses involved in the management, search, and retrieval of poly-structured data, in some cases with strong market share, but even so not necessarily with the strongest of reputations for technology or technology momentum. Autonomy started from a text search engine and a Bayesian search algorithm on top of that, which did a decent job for many customers. But if there’s been much in the way of impressive enhancement over the past 8-10 years, I’ve missed the news.

*Dave, of course, was CEO of MarkLogic.

Questions obviously arise about how the Autonomy acquisition relates to other HP businesses. My early thoughts include: 

• HP has clearly signaled that it intends to pursue and focus on the data management business. Thus, we can anticipate marketing messages spanning Autonomy and Vertica. It may be helpful to recall that Vertica plays nicely with both Hadoop and Attensity.
• The first two natural tuck-in acquisitions I can think to add are Attensity and MarkLogic.
• One place I’d look for synergy is with HP’s system management software business. HP has previously acquired its way into a strong position there. If you add in knowledge of how many kinds of data are used, you have a chance to set yourself apart in the system management area.
• I had enough trouble advising Vertica about how to explain what they do in terms that HP’s hardware sales force can comfortably embrace. I think I did OK with that. But Autonomy? Youch. On the other hand, …
• … HP is run by guys from SAP (Leo Apotheker) and Oracle (Ray Lane), both of whom have dealt with similarly tough sales challenges before. But even at best, HP’s sales force organization, commission structure, and training is going to consume a lot of attention at the very highest levels of HP.
• Autonomy manages documents electronically. HP prints them. The markets where that seems synergistic, however, are fairly specialized or small. (E.g., equipment for printing on demand.) Perhaps there’s some grand joint venture possibility with Xerox here, antitrust permitting.

Context for any comments on customer traction includes:

• Membase went into limited production release in October, and full release in January. Similar things are true of CouchDB.
• Hence, most sales of Couchbase’s products have been made over the past 6 months.
• Couchbase (the merged product) is at this point only in a pre-production developer’s release.
• Couchbase has both a direct sales force and a classic open-source “funnel”-based online selling model. Naturally, Couchbase’s understanding of what its customers are doing is more solid with respect to the direct sales base.
• Most of Couchbase’s revenue to date seems to have come from a limited number of big-ticket “lighthouse” accounts (as opposed to, say, the larger number of smaller deals that come in through the online funnel).

That said,

• Most Membase purchases are for new applications, as opposed to memcached migrations. However, customers are the kinds of companies that probably also are using memcached elsewhere.
• Most other Membase purchases are replacements for the Membase/MySQL combination. Bob says those are easy sales with short sales cycles.
• Pure memcached support is a small but non-zero business for Couchbase, and a fine source of upsell opportunities.
• In the pipeline but not so much yet in the customer base are SaaS vendors and the like who use and may want to replace traditional DBMS such as Oracle. Other than among those, Couchbase doesn’t compete much yet with Oracle et al.
• Pure CouchDB isn’t all that much of a business, at least relative to community size, as CouchDB is a single-server product commonly used by people who are content not to pay for support.

Membase sales are concentrated in five kinds of internet-centric companies, which in declining order are:

• Social gaming
• Ad platforms
• Online retail
• Online business, including B2B  SaaS
• Social networking

Bob said that Couchbase often sees MongoDB competitively, but never Riak, HBase, or Redis. I got the impression Couchbase sees at least a little Cassandra. That would, of course, all pertain only to direct sales, rather than download/community kinds of usage.

Couchbase is also excited about the potential for the CouchDB-based Couchbase Mobile occasionally-connected offering. The hottest use cases, interestingly, seem to be non-consumer; Bob rattled off military, farming, and health care, and surely could have named more besides. However, the Couchbase Mobile sales effort still seems to be in early days, as is evidenced by the fact that Couchbase has not yet competitively encountered Sybase SQL Anywhere.

With all that said, I’ll go now to a separate post for a Couchbase technical update.

• “A bunch of apps in one, similar but not the same” (in my recent post on MongoDB).
• Out-of-control product catalogs (ditto).
• Analytic use cases in which one keeps enhancing the database with derived data.

And if marketing profile analysis is ever done correctly, that will be a huge example for the list.

So what do we call those DBMS — for example NoSQL, object-oriented, or XML-based systems — that bake the schema into the applications or the records themselves? In the MongoDB post I went with “schemaless,” but I wasn’t really comfortable with that, so I took the discussion to Twitter. Comments from Vlad Didenko (in particular), Ryan Prociuk, Merv Adrian, and Roland Bouman favored the idea that schemas in such systems are changeable or late-bound, rather than entirely absent. I quickly agreed.

The discussion wasn’t entirely serious; wise-ass comments were contributed by at least Merv, Neil Raden, Yiorgos Adamopoulos, and myself.

I like that approach for the same reason I favor saying that databases are poly- or multi-structured (rather than un- or semi-):  Every database has structure, the only question being when that structure is determined. I wouldn’t precisely equate “poly-structured” to “has a late-bound schema”; for example, I’d say that mucking with the DDL (Data Description Language) of a relational database shows that it’s a little bit poly-structured, even though it’s not at all late-bound. But the concepts are definitely related.

So what actual wording should we use here? The only alternative I see to fixed schema is “static”, and that feels like it has too much of a connotation of “unchangeable”. The simplest word I can think of for changeable/late-bound/whatever is dynamic schema; that choice also has the virtue of some traction, as per the Vlad Didenko tweet linked above. Casual googling is also supportive of “fixed” and “dynamic”, at least over whatever alternatives I came up with. So those are my choices.

For actual definitions, I’ll say:

• A (logical) schema is fixed if it is defined before a program is written, but dynamic if it is defined by the program or data itself.
• A database is fixed- or dynamic-schema depending on whether its schemas are fixed or dynamic respectively.
• A DBMS is fixed- or dynamic-schema depending on whether databases created in it tend to have fixed or dynamic schemas respectively.

Suit yourself as to what you say about relational DBMS when they also have a bit of XML, text, or whatever support.

By these definitions:

• Relational databases are fixed-schema (within the caveat above about XML or text data).
• MOLAP databases are fixed-schema.
• Pre-relational network and hierarchical DBMS (e.g. IMS) are fixed-schema.
• Most other DBMS are dynamic-schema.

What do you think? Do these definitions work for you?

*Everybody seems to like “poly-structured” better when it has a hyphen in it — including me.

The big difference between the two is that “multi-” just means there are multiple structures, while “poly-” further means that the structures are subject to change. Upon reflection, I think the “subject to change” part is essential, so poly-structured it is.

The definitions I’m proposing are:

• A database is poly-structured to the extent that its structure is apt to be changed in the ordinary course of query, update, or programming.
• Data is poly-structured to the extent that it is best represented in a poly-structured database.
• A DBMS is poly-structured to the extent that it is oriented to managing poly-structured databases.

Please note:

• There are many different degrees of being poly-structured; that’s why I used the phrase “to the extent that”, instead of a simple “if”.
• And as always, no technology categorization is ever precise.

Examples of poly-structure include:

• XML or JSON documents/objects describe themselves. Add a new one to a database with a different structure than the others and — presto! — you have changed the overall structure. Thus:
  • XML and JSON data is apt to be poly-structured.
  • XML and JSON databases are apt to be poly-structured.
  • MarkLogic, MongoDB, et al. are poly-structured DBMS.
• A text document is inherently poly-structured. Some queries might look at it as a bag of words; others might group the words via stemming and synonyms; others might actually exploit the document’s grammatical structure. Text search engines are poly-structured because they support all those kinds of queries.
• A single log file can be somewhat poly-structured, in that different views of it might extract different kinds of name-value pair, or different temporal relationships.
• A database that seamlessly includes a variety of log files, each with its own structure(s), is quite poly-structured.
• A classic relational database is not very poly-structured, because DDL (Data Description Language) isn’t really in “the ordinary course” of programming or update.
• However, views add a bit of poly-structure to relational databases that is not present in, say, IMS databases.
• An object-oriented DBMS is highly poly-structured, as is Workday’s internal data store.

So what do you think? Do these definitions work?

So what other markets could MarkLogic pursue? Before Ken even started work, I sent over some thoughts. They included (but were not limited to): 

• Everybody now knows that not all problems require a relational DBMS. The NoSQL movement has seen to that.
• Not everybody agrees that “heavyweight” enterprise DBMS are needed for everything. The NoSQL movement has seen to that too.
• The “document”/”object” DBMS distinction has long been blurry. XML is full of documents, but they’re really objects. The same goes for the JSON/quasi-JSON objects of CouchDB/Couchbase and MongoDB. Object-oriented DBMS vendors have dabbled in XML on and off over the years because of technical similarity. Etc.
• MarkLogic has always focused on markets where the database truly was about documents in the conventional sense — especially long text documents — aka “content”. I always thought that focus was over-narrow.
• There are various cases where law, regulation, compliance etc. mandate the production of long text documents. I’m not sure MarkLogic has penetrated those as well as it could have.
• Graph DBMS technology is going nowhere fast, largely because nobody has solved the data distribution problem in cases big enough to need scale-out, and the technology isn’t obviously needed in single-server cases. (But see my post on Objectivity’s Infinite Graph.) Even so, graph-oriented apps are exploding, and MarkLogic should think about playing in the graph area, even if by acquisition.
• I think what I described in http://www.dbms2.com/2010/06/08/profile-of-revealed-preferences/ is non-relational and a very big market. Playing there with a “heavyweight” DBMS is of course a challenge.
• Coming over from Autodesk, Ken Bado hopefully knows more about the product data management business than I do.

It will be interesting to see what MarkLogic actually does.

Database management that uses a JSON model and gives you reasonably robust access to individual field values inside a JSON (JavaScript Object Notation) object.

Or, if they really mean,

The essence of whatever it is that CouchDB and MongoDB have in common.

well, that’s pretty much the same thing as what I said in the first place.

Of the various questions that might arise, three of the more definitional ones are:

• Why JSON rather than XML?
• What’s with this fluidity between the terms “document” and “object”?
• Are you serious about the lack of joins?

Let me take a crack at each. 

Like XML, JSON is a data-interchange format that has been repurposed as a data persistence model. JSON is evidently beating out XML in web applications, for reasons including:

• XML is more verbose and slower than JSON. (Whether this matters or not is of course use-case-dependent.)
• Like SQL, XML requires what some web programmers regard as too much formalism and up-front specification.
• JSON is associated with JavaScript.
• JSON is regarded as being more suited to straightforwardly fielded data, while XML is regarded as being more suited to “mixed content” — e.g., real text documents.
• In general, XML feels “enterprisey” to developers who don’t like that feel.

One good starting point for recent JSON vs. XML discussion is here. My favorite from the 2007 iteration of the debate is this one.

So, in essence:

• The reasons JSON beats XML for web application data interchange have some applicability to web application data storage as well.
• There’s ever more JSON around, at the expense of XML.

But truth be told, I don’t think XML and JSON actually go head to head against each other on the DBMS side very often at all. E.g., Dwight Merriman (the 10gen/MongoDB guy) told me he never, ever competes against MarkLogic, and I found that very credible.*

*Proof point: Dwight was clueless about MarkLogic specifics in a way he never would be if they were any kind of competitive consideration for him.

Note that the one area where (almost) everybody agrees XML wins is for what one might call “real” documents. By way of contrast, JSON is best suited for stringing data attributes and values together. So the “documents” that JSON models can indeed just as reasonably be called “objects.”

That said, JSON-based DBMS are not what one would normally call object-oriented DBMS; for an example of those, consider Intersystems Cache’. And just to close the loop on confusion — Cache’ can also be used as an XML DBMS.

As I previously noted, one downside to today’s document-oriented DBMS is that you can’t do joins. Let me now add that I think joins will be added to document DBMS in the future. Plausibility arguments for this opinion include:

• MarkLogic — the XML database gold standard — sells to enterprises, and enterprises like joins.
• The alternative to joins in CouchDB and MongoDB is in essence MapReduce. Well, Hive proves that you can do joins on top of MapReduce if you want to. (So, for that matter, does Aster Data nCluster; Aster says its SQL parallelism is built on top of MapReduce.)
• Intersystems quite happily put SQL on top of an object-oriented DBMS, Cache’. And Cache’ is so similar to an XML DBMS that it in fact is sometimes used as one.

But that is indeed a future. For discussion of the current state of affairs, I refer you to my earlier post on the subject of joinlessness linked above.

Since MarkLogic and other XML DBMS are used in applications for brokerage trades and the like, I used that area as my example for a challenge question: What happens when one brokerage firm buys another? (Similar challenges could be made about medical records or consumer profiling.) The answer was that you just have to update or augment each existing record with the new firm’s information. And by the way, if you choose to augment, then you have the new and old information side-by-side, both of which could conceivably come in handy.

Document-oriented NoSQL DBMS such as CouchDB and MongoDB face similar challenges, of course. I didn’t pursue the matter in depth in either case, but:

• If I understood Damien Katz correctly, CouchDB has a view capability that provides some kind of workaround. (A quick web search turned up this page on a kind of entity-relationship modeling in CouchDB and the associated querying.)
• Dwight Merriman suggested to me that in MongoDB, you can work around the lack of joins via client-side logic, or by embedding lots of data in each document (e.g., all the line items for an order) and extracting what you need via MapReduce jobs.

I’m not totally sure what I think about joinlessness, but one way of looking at it could be:

• The reason we have joins is because we normalize. If it’s OK to be highly denormalized, then it’s less important to have joins.
• When normalization is good and denormalization is bad, one or both of two reasons are commonly in play:
  • The logical burden of keeping straight all the different places you’d have to update the same data is too great for the poor, overburdened programmers.
  • The performance burden of doing all that updating is too great for the poor, overburdened hardware.
• For the logical reason to have great force, there has to be a pretty complex schema, or else a frequently changing one. But when schemas change frequently, relational designs have their own problems.
• The physical reason automatically has great force if you have huge update volumes and keep many copies of the same data. Otherwise, its strength has a lot to do with the specific architecture of the DBMS. E.g., if it’s a lot cheaper to update a small record than a big one, short rows are better. But otherwise, denormalization may not have that much effect on performance.

Putting all that together, I’m inclined to think that for many applications, it’s OK to denormalize, or to have such a simple schema that normalization is moot. But even so, I’d be a lot more comfortable if a DBMS offered at least some way of doing a join.

All this raises a related question: What are transactions like in document-oriented DBMS? I’ve never pushed the point with MarkLogic, but when they talk of their ACID compliance they sound as if they are using the phrase in the usual way. MongoDB only lets you do transactions in single documents. I’ve never asked the question about CouchDB, but I do note with interest CouchDB’s “crash-only” architecture, which boils down to:

• CouchDB shutdown is “instantaneous.”
• You can only shut down CouchDB by crashing it.
• There’s no way to shut down or crash CouchDB that causes data to be inconsistent.