NoSQL is most easily defined by what it excludes: SQL, joins, strong analytic alternatives to those, and some forms of database integrity. If you leave all four out, and you have a strong scale-out story, you’re in the NoSQL mainstream.

• Thus, I’d say Cassandra, HBase, Mongo DB, and Couchbase are prime examples, in no particular order. Riak as well.
• I might have phrased that better if I’d used a different word than simply “strong” — but hey, there was a 140-character limit, and he was on deadline.

Using NoSQL can make sense when at least one of two things is paramount: low-cost scale-out or dynamic schemas.

• There are some seriously sensible use cases for dynamic schemas.
• “Low-cost” generally boils down to:
  • Performance.
  • Open source free-like-beer.
  • Not a lot of database administration.

I’ve generally given object-oriented DBMS vendors and also MarkLogic hard times whenever they consider saying they’re “NoSQL”. Reasons include:

• Closed source.
• Database administration overhead (even if you get good stuff for incurring that overhead, like MarkLogic’s comprehensive indexing).

Also, NoSQL started out being ACID-unfriendly.

What you give up are the query flexibility and the easily automatic data integrity of SQL-based systems. I should have added something about a mature ecosystem.

In the most recent live example, I influenced a client away from Cassandra and toward scale-out MySQL (dbShards and/or Schooner flavors, most likely). Part of the reason was the ability to do joins, which are useful in their application. Another part is that their development practices obviated any significant benefit from dynamic schemas. But perhaps the most important — or at least resonant — reason of all was that they really, really cared about .NET support.

There really are three major points to NoSQL.

• Dynamic schemas. This is the only one of the three that truly depends on NoSQL.
• Scale-out short-request processing. If you want to scale out efficiently at high request volumes, you’re best off not using all the flexibility SQL/relational DBMS offer. (In particular, you don’t want to do cross-node joins). Not coincidentally, a number of the best scale-out offerings were built to be NoSQL.
• Open source. Doing a relational DBMS is a big project. It seems easier to build NoSQL ones.

Oracle can address the latter two points as aggressively as it wishes via MySQL. It so happens I would generally recommend MySQL enhanced by dbShards, Schooner, and/or dbShards/Schooner, rather than Oracle-only MySQL … but that’s a detail. In some form or other, Oracle’s MySQL is a huge player in the scale-out, open source, short-request database management market.

So that leaves us with dynamic schemas. Oracle has at least four different sets of technology in that area:

• As Workday noticed years ago, MySQL can be used as a functional, basic key-value store.
• Oracle also has XML-based Berkeley DB/SleepyCat kicking around.*
• The XML extensions to Oracle’s core DBMS could be alleged to have a dynamic schema/NoSQL flavor. (Blech.)
• A dynamic schema argument could also be made for object-oriented DBMS technology. While Oracle doesn’t to my knowledge exactly sell that, it does have the Tangosol Coherence line of technology, with a potentially similar programming model.

If Oracle is now refreshing and rebranding one or more of these as “NoSQL”, there’s no reason to view that as a big deal at all.

*That’s Mike Olson’s former company, if you’re keeping score at home.

• A long-ago marketing decision to not give infrastructure details, so as to convey a “Don’t worry; we’ll take care of everything” message.
• Even so, a long-ago and perhaps now-regretted marketing decision to disclose and even exaggerate salesforce.com’s reliance on Oracle, as part of an early-days attempt to prove salesforce was using enterprise-class technology.
• A desire to hide the recipe for salesforce.com’s secret sauce.
• Force of habit — I’m not sure salesforce even knows how to tell its technical story with any clarity.

Actually, salesforce.com has moved some kinds of data out of Oracle that previously used to be stored there. Besides Oracle, salesforce uses at least a file system and a RAM-based data store about which I have no details. Even so, much of salesforce.com’s data is stored in Oracle — a single instance of Oracle, which it believes may be the largest instance of Oracle in the world.

Salesforce did spell out some of its database story in a 2008 force.com white paper, which is good stuff, but potentially misleading in one important way. The paper tells of a level of abstraction, whereby what the application sees as logical “columns” are stored in a very different schema than one might assume. However, it doesn’t spell out a second level of abstraction, whereby that logical schema also isn’t how the database is actually laid out.

Another flaw in the paper is that it spins “We had to do this, to support multitenancy, so we did.” issues as “Because we’re multitenant, we can do this, while single-tenant systems can’t.” One example is the query optimization step around “user visibility” in Figure 11. Welcome to marketing.

At the first level of abstraction, data seems to be kept mainly in a single wide table, with hundreds of columns. What’s more, many of those are “flex columns”; a flex column can hold data of many different kinds and even datatypes. Notwithstanding the second level of abstraction, I imagine the idea of stuffing different kinds of thing into the same column has something to do with the fact that Oracle’s physical limit on columns falls far short of the number of logical columns salesforce wants to use.

If we imagine that the different kinds of data in a flex column were each in their own column instead, the whole thing might sound like BigTable/Cassandra/HBase-style column-group NoSQL. Thus, much as Workday uses MySQL to simulate a key-value store, salesforce.com can be said to use Oracle to simulate a different kind of NoSQL. In both cases, what’s going on seems to be a kind of object/relational mapping, but with the relational aspect strongly deemphasized. Or, if you take a more relational view, we could say that salesforce.com’s tables are a lot wider than any one user organization’s, because each user sees only its own custom columns (plus the standard ones common to all users).

The second layer of abstraction has a lot to do with multitenancy. If you want to stick data for many different user organizations into the same huge table, then you have to label it in some way to show who is permitted to see or update each part. Logically, this leads to a join, between one table carrying data plus a simple key showing which users/roles are entitled to see it, and a second table showing who actually is that kind of user/has that kind of role. But that join makes a lot of sense to store in a denormalized way, all the more because data is partitioned across the computer cluster in line with which user organization it actually belongs to.

Multitenant security isn’t the only reason for this denormalization, but it appears to be the biggest one.

The whole thing is doing 550 million or so transactions per day. salesforce.com thinks that fact should be regarded as evidence that it works.

• “A bunch of apps in one, similar but not the same” (in my recent post on MongoDB).
• Out-of-control product catalogs (ditto).
• Analytic use cases in which one keeps enhancing the database with derived data.

And if marketing profile analysis is ever done correctly, that will be a huge example for the list.

So what do we call those DBMS — for example NoSQL, object-oriented, or XML-based systems — that bake the schema into the applications or the records themselves? In the MongoDB post I went with “schemaless,” but I wasn’t really comfortable with that, so I took the discussion to Twitter. Comments from Vlad Didenko (in particular), Ryan Prociuk, Merv Adrian, and Roland Bouman favored the idea that schemas in such systems are changeable or late-bound, rather than entirely absent. I quickly agreed.

The discussion wasn’t entirely serious; wise-ass comments were contributed by at least Merv, Neil Raden, Yiorgos Adamopoulos, and myself.

I like that approach for the same reason I favor saying that databases are poly- or multi-structured (rather than un- or semi-):  Every database has structure, the only question being when that structure is determined. I wouldn’t precisely equate “poly-structured” to “has a late-bound schema”; for example, I’d say that mucking with the DDL (Data Description Language) of a relational database shows that it’s a little bit poly-structured, even though it’s not at all late-bound. But the concepts are definitely related.

So what actual wording should we use here? The only alternative I see to fixed schema is “static”, and that feels like it has too much of a connotation of “unchangeable”. The simplest word I can think of for changeable/late-bound/whatever is dynamic schema; that choice also has the virtue of some traction, as per the Vlad Didenko tweet linked above. Casual googling is also supportive of “fixed” and “dynamic”, at least over whatever alternatives I came up with. So those are my choices.

For actual definitions, I’ll say:

• A (logical) schema is fixed if it is defined before a program is written, but dynamic if it is defined by the program or data itself.
• A database is fixed- or dynamic-schema depending on whether its schemas are fixed or dynamic respectively.
• A DBMS is fixed- or dynamic-schema depending on whether databases created in it tend to have fixed or dynamic schemas respectively.

Suit yourself as to what you say about relational DBMS when they also have a bit of XML, text, or whatever support.

By these definitions:

• Relational databases are fixed-schema (within the caveat above about XML or text data).
• MOLAP databases are fixed-schema.
• Pre-relational network and hierarchical DBMS (e.g. IMS) are fixed-schema.
• Most other DBMS are dynamic-schema.

What do you think? Do these definitions work for you?

*I could be wrong in those guesses; as small companies go, McObject is unusually prone to secrecy games.

As best I understand:

• eXtremeDB is something like an in-memory object-oriented DBMS, designed to be embeddable.
• However, much as with Objectivity and other old-school OODBMS, eXtremeDB winds up being more of a toolkit with which to build DBMS than a full DBMS.
• eXtremeDB has a few indexing schemes. The main one is good old B-trees. One customer wanted Patricia tries, so they’re in there. (Perhaps not coincidentally, solidDB relies on Patricia tries.) At least one wanted R-trees, so they’re in there too.
• eXtremeDB has long had the option of persistent logs.
• eXtremeDB newly has a hybrid memory-centric option, in which you can have more data in the database than fits into RAM.
• eXtremeDB newly has multi-master two-phase-commit clustering.

My guess three years ago that eXtremeDB might emerge as an alternative to solidDB seems to have been borne out. McObject CEO Steve Graves says that the core of McObject’s business is OEMs, in sectors such as telecom equipment and defense/aerospace. That’s exactly solidDB’s traditional market, except that solidDB got acquired by IBM and deemphasized it.

I’ve said before that if I were starting a SaaS effort — and it wasn’t just focused on analytics — I’d look at using a memory-centric OODBMS. Perhaps eXtremeDB is worth looking at in such scenarios.

Oracle seems to have said on yesterday’s conference call Oracle OpenWorld (first week in October) will feature appliances based on Tangosol and Hadoop. As I post this, the Seeking Alpha transcript of Oracle’s call is riddled with typos. Bolded comments below are by me. 

Well, we’re planning to add a couple of appliances and announcing them this fall. One appliance, that should surprise you is a large memory addition to Exadata for analytics and memory, so we continue to invest. We thought that would — we’ve been the leader of in-memory database technology ever since we bought Tungsten. I presume that’s a typo for “Tangosol”. And it sort of denigrates Oracle TimesTen. And that’s for both for transactions and for preprocessing. We are, as memories become cheaper and larger scale, we’ve changed as much of our algorithms and this in-memory analytics accelerator is going to be, again, coming out and we’ll be announcing it in the fall at Oracle OpenWorld.

That part, especially in connection with the last sentence of the next quote, sounds almost as if Tangosol will be positioned as a kind of memory-centric object-oriented DBMS, albeit with Oracle as its persistence layer. Well, I favor both in-memory and object-oriented DBMS, and especially the intersection of those two categories. So in principle this could be a very cool product. Exploiting that coolness, however, may require one heck of a missionary sell.

In addition, attaching to our Exalogic box, there’s a lot of misunderstanding about what’s a dupe is, and is it a replacement for database. I presume “a dupe” is a typo for “Hadoop”. So the dupe is not a replacement for database. It’s an adjunct to the database, which we think, is very, very important. It really is a tool for Java programmers. And we’re the world leader in Java technology and we are building a big data accelerator to attach to our Exalogic box, which comes out also this fall. The big data accelerator includes some of the standard open source heavy software, HTFF, the heavy file system and a number of other pieces, but also some Oracle components that we think can dramatically speed up the entire math-produced process. I presume that’s a series of typos for “HDFS” and “MapReduce“. And will be particularly attractive to Java programmers who are the ones, who asked for — aspire to do. There are some interesting applications they do, ETL is one. Log processing is another. Those last two sentences are more evidence for the theory that this is about Hadoop. Besides, I spoke with somebody who listened to the call. We’re going to have a lot of those features, functions and prebuilt applications in our big data accelerator. So, Oracle has always followed database technology trends, whether it’s object databases, in-memory databases and kept up with this technology and some, quite often led on innovation.

And that part sounds as if Oracle will announce a Hadoop appliance, positioning it more as a Java software accelerator than a place to store cheap data. Be the positioning as it may, my objections to the idea of a Hadoop appliance still stand, although Amr Awadallah’s counterarguments make sense as well.

*Actually, he didn’t admit that what he was advocating was a different kind of DBMS, namely a MOLAP one — but he was. And he was wrong anyway about the necessity for MOLAP. But let’s overlook those details.

Nonetheless, relational DBMS dominate the market. As I see it, the reasons for relational dominance cluster into four areas (which of course overlap):

• Data re-use. Ted Codd’s famed original paper referred to shared data banks for a reason.
• The benefits of normalization, which include:
  • You only have to do programming work of writing something once …
  • … and you don’t have to do the programming work of keeping multiple versions of the information consistent.
  • You only have to do processing work of writing something once.
  • You only have to buy storage to hold each fact once.
• Separation of concerns.
  • Different people can worry about programming and “database stuff.”
  • Indeed, even performance optimization can sometimes be separated from programming (i.e., when all you have to do to get speed is implement the correct indexes).
• Maturity and momentum, as reflected in the availability of:
  • People.
  • A broad variety of mature relational DBMS.
  • Vast amounts of packaged software that “talks” SQL.

Generally speaking, I find the reasons for sticking with relational technology compelling in cases such as: 

• You’re building a low-volume, medium-complexity suite of applications that will evolve over time. This is the use case for which relational DBMS were invented, and they’re still great for it.
• Your (duplicated) data volumes would be ridiculous if you didn’t do a reasonable amount of normalization. Once you need to normalize, you need to do joins — and if you’re doing joins, you’re in relational territory.
• You simply don’t see a cost/benefit advantage to moving away from proven legacy technology. If you’re looking for an off-the-shelf answer to your needs — or if you’re inventorying your own technological shelves — relational-oriented technology has overwhelming share.

For many enterprises, that third point alone should be decisive in a large fraction of cases.

But the advantages of relational technology are less clear when you’re doing serious engineering of path-breaking new applications, where by “serious engineering” I mean:

• The problem is big enough that you simply want the best solution, with only loose coupling needed to the rest of your technical environment.
• Long-lasting “strategic” or legacy technology is not a great concern; you’re willing to keep “rebuilding the 747 while it’s flying” if that’s what’s necessary to get the best possible result.
• You have access to sufficient quantities of sufficiently smart people.

For example:

• I recently suggested that innovative SaaS vendors could adopt object-oriented database technology.
• Major web applications are rarely very relational. Until recently, the default approach to scaling out web databases was memcached/sharded MySQL, hardly a whole-hearted adoption of relational technology. Now NoSQL DBMS are vigorous competitors.
• Analytic challenges that amount to teasing out signals from streams of data are sometimes handled non-relationally as well, although it’s often nice to be able to do a few joins to mix in information from more relationally-structured data.

Not coincidentally, in a lot of those cases, throwing performance concerns “over the wall” to the database administrator isn’t going to work.

*I do expect the pendulum to swing back a bit as high-performance/highly-scalable MySQL implementations mature, but there are relatively few supporting examples to date.

To look at it another way, it’s right to be skeptical about relational DBMS when you can defeat all of the reasons to favor them. For example:

• Data re-use may not arise when applications are self-contained and rapidly-changing.
• Sometimes you don’t need to normalize your data.
• It’s not obvious that the relational approach to separation of concerns is the best one. Perhaps you’d be better off with the people who understand a specific application best being responsible for all the decisions connected with it.
• As for that maturity and momentum:
  • People don’t actually learn much SQL in school.
  • Are any of the mature relational DBMS what you really want?
  • Is any of that packaged software out there really helpful for your specific problem?

I should probably stop there. But in an appeal to authority, I’ll close instead with a quote from Codd’s own OLAP paper:

IT should never forget that technology is a means to an end, and not an end in itself. Technologies must be evaluated individually in terms of their ability to satisfy the needs of their respective users. IT should never be reluctant to use the most appropriate interface to satisfy users’ requirements. Attempting to force one technology or tool to satisfy a particular need for which another tool is more effective and efficient is like attempting to drive a screw into a wall with a hammer when a screwdriver is at hand: the screw may eventually enter the wall but at what cost?

Related link

• My exchange with Mike Stonebraker highlighting our shared advocacy for database diversity

• Has been around as a company since 2006.
• Has developed memory-centric object-oriented DBMS technology that has been OEMed by a few application software companies (especially in bricks-and-mortar retailing and in online advertising).
• Is planning to actually launch an OODBMS product sometime this summer.
• Has 14 employees (most or all of whom are in Sweden, which is also where I think Starcounter’s current customers are centered).
• Is planning to shift emphasis soon to the US market.

Starcounter’s value propositions are programming ease (no object/relational impedance mismatch) and performance. Starcounter believes its DBMS has 100X the performance of conventional DBMS at short-request transaction processing, and 10X the performance of other memory-centric and/or object-oriented DBMS (e.g. Oracle TimesTen, or Versant). That said, Starcounter has not yet tested VoltDB. Starcounter does not claim performance much beyond that of disk-based DBMS on analytic tasks such as aggregations.

The key technical aspect to Starcounter is integration between the DBMS and the virtual machine, so that the same copy of the data is accessed by both the DBMS and the application program, without any movement or transformation being needed. (Starcounter isn’t aware of any other object-oriented DBMS that work this way.) Transient and persistent data are handled in the same way, seamlessly.

Other Starcounter technical highlights include: 

• Starcounter is focused on OLTP (OnLine Transaction Processing).
• The Starcounter OODBMS is ACID-compliant.
• The Starcounter DBMS is single-server, albeit multi-core. Starcounter thinks this is OK because the Starcounter DBMS can do millions of transactions per second on a server with 8 cores or less. (I neglected to ask how quickly Starcounter thinks RAM would fill up on a single server at that kind of update rate.)
• A Starcounter database sits in RAM. Logs go to disk, and Starcounter doesn’t commit a transaction in RAM until there’s been an acknowledgement that it’s been logged to disk.*
• Since Starcounter never wants to read from disk (except in the case of recovery), logs can be written pretty much at sequential/batch update speeds.
• You can do SQL queries against Starcounter objects, based on T-tree indexes. Otherwise, Starcounter data manipulation is done via what effectively is a proprietary object-oriented data manipulation language. (I neglected to ask whether there was any concept of join, or whether Starcounter SQL just does SELECTs. Starcounter did say that the only schema in a Starcounter database is the object model.)
• Naturally, Starcounter objects are compressed, so that the most possible data fits into the fastest possible tier of memory. Proxy objects also come into play here.
• Starcounter runs on Windows and .NET, supposedly for reasons of better virtual machine performance. Ports to Linux, Java, etc. are on the drawing boards, but won’t be particularly easy.

*I thought Starcounter said that the core that runs the operating system communicates the acknowledgement via Direct Memory Access to a core that runs the Starcounter DBMS, obviating the need for an interrupt (except to the core that runs the operating system). But upon reflection, that doesn’t really seem to make sense.

*Everybody seems to like “poly-structured” better when it has a hyphen in it — including me.

The big difference between the two is that “multi-” just means there are multiple structures, while “poly-” further means that the structures are subject to change. Upon reflection, I think the “subject to change” part is essential, so poly-structured it is.

The definitions I’m proposing are:

• A database is poly-structured to the extent that its structure is apt to be changed in the ordinary course of query, update, or programming.
• Data is poly-structured to the extent that it is best represented in a poly-structured database.
• A DBMS is poly-structured to the extent that it is oriented to managing poly-structured databases.

Please note:

• There are many different degrees of being poly-structured; that’s why I used the phrase “to the extent that”, instead of a simple “if”.
• And as always, no technology categorization is ever precise.

Examples of poly-structure include:

• XML or JSON documents/objects describe themselves. Add a new one to a database with a different structure than the others and — presto! — you have changed the overall structure. Thus:
  • XML and JSON data is apt to be poly-structured.
  • XML and JSON databases are apt to be poly-structured.
  • MarkLogic, MongoDB, et al. are poly-structured DBMS.
• A text document is inherently poly-structured. Some queries might look at it as a bag of words; others might group the words via stemming and synonyms; others might actually exploit the document’s grammatical structure. Text search engines are poly-structured because they support all those kinds of queries.
• A single log file can be somewhat poly-structured, in that different views of it might extract different kinds of name-value pair, or different temporal relationships.
• A database that seamlessly includes a variety of log files, each with its own structure(s), is quite poly-structured.
• A classic relational database is not very poly-structured, because DDL (Data Description Language) isn’t really in “the ordinary course” of programming or update.
• However, views add a bit of poly-structure to relational databases that is not present in, say, IMS databases.
• An object-oriented DBMS is highly poly-structured, as is Workday’s internal data store.

So what do you think? Do these definitions work?

So what other markets could MarkLogic pursue? Before Ken even started work, I sent over some thoughts. They included (but were not limited to): 

• Everybody now knows that not all problems require a relational DBMS. The NoSQL movement has seen to that.
• Not everybody agrees that “heavyweight” enterprise DBMS are needed for everything. The NoSQL movement has seen to that too.
• The “document”/”object” DBMS distinction has long been blurry. XML is full of documents, but they’re really objects. The same goes for the JSON/quasi-JSON objects of CouchDB/Couchbase and MongoDB. Object-oriented DBMS vendors have dabbled in XML on and off over the years because of technical similarity. Etc.
• MarkLogic has always focused on markets where the database truly was about documents in the conventional sense — especially long text documents — aka “content”. I always thought that focus was over-narrow.
• There are various cases where law, regulation, compliance etc. mandate the production of long text documents. I’m not sure MarkLogic has penetrated those as well as it could have.
• Graph DBMS technology is going nowhere fast, largely because nobody has solved the data distribution problem in cases big enough to need scale-out, and the technology isn’t obviously needed in single-server cases. (But see my post on Objectivity’s Infinite Graph.) Even so, graph-oriented apps are exploding, and MarkLogic should think about playing in the graph area, even if by acquisition.
• I think what I described in http://www.dbms2.com/2010/06/08/profile-of-revealed-preferences/ is non-relational and a very big market. Playing there with a “heavyweight” DBMS is of course a challenge.
• Coming over from Autodesk, Ken Bado hopefully knows more about the product data management business than I do.

It will be interesting to see what MarkLogic actually does.