• Sumo Logic does SaaS (Software as a Service) log management.
• Sumo Logic is text indexing/Lucene-based. Thus, it is reasonable to think of Sumo Logic as “Splunk-like”. (However, Sumo Logic seems to have a stricter security/trouble-shooting orientation than Splunk, which is trying to branch out.)
• Sumo Logic has hacked Lucene for faster indexing, and says 10-30 second latencies are typical.
• Sumo Logic’s main differentiation is automated classification of events.
• There’s some kind of streaming engine in the mix, to update counters and drive alerts.
• Sumo Logic has around 30 “customers,” free (mainly) or paying (around 5) as the case may be.
• A truly typical Sumo Logic customer has single to low double digits of gigabytes of log data per day. However, Sumo Logic seems highly confident in its ability to handle a terabyte per customer per day, give or take a factor of 2.
• When I asked about the implications of shipping that much data to a remote data center, Sumo Logic observed that log data compresses really well.
• Sumo Logic recently raised a bunch of venture capital.
• Sumo Logic’s founders are out of ArcSight, a log management company HP paid a bunch of money for.
• Sumo Logic coined a marketing term “LogReduce”, but it has nothing to do with “MapReduce”. Sumo Logic seems to find this amusing.

What interests me about Sumo Logic is that automated classification story. I thought I heard Sumo Logic say:

• It’s largely unsupervised machine learning.
• It’s specific to a particular user/data set.
• It can be up and running and classifying things effectively almost instantly (i.e., on seconds’ or minutes’ worth of data).
• It’s informed by what different users tag as false positives. (Or maybe that is planned for future versions.)

I have a little trouble seeing how all those points fit exactly together, so perhaps I got some details wrong.

The payoff is that machine learning directly informs the Sumo Logic user interface. In particular, large numbers of events are bundled into a small number of categories, hopefully making it much easier for network operations types to scan the UI and pick out what’s important.

In general, the idea of machine-learning informing analytic UIs via some sort of classification is common in text-oriented technologies, notably in:

• Good ol’ text search.
• Text mining vendors’ approaches to clustering hits on words or phrases that say substantially the same thing.

But otherwise it seems kind of rare, if we stipulate that ad-serving/general internet personalization isn’t really an analytic UI — but I’d love to hear of any interesting examples I’ve overlooked.

• Splunk is first and foremost an analytic DBMS …
• … used to manage logs and similar multistructured data.
• Splunk’s DML (Data Manipulation Language) is based on text search, not on SQL.
• Splunk has extended its DML in natural ways (e.g., you can use it to do calculations and even some statistics).
• Splunk bundles some (very) basic, Splunk-specific business intelligence capabilities.
• The paradigmatic use of Splunk is to monitor IT operations in real time. However:
  • There also are plenty of non-real-time uses for Splunk.
  • Splunk is proudest of its growth in non-IT quasi-real-time uses, such as the marketing side of web operations.

As in any release, a lot of Splunk 4.3 is about “Oh, you didn’t have that before?” features and Bottleneck Whack-A-Mole performance speed-up. One performance enhancement is Bloom filters, which are a very hot topic these days. More important is a switch from Flash to HTML5, so as to accommodate mobile devices with less server-side rendering. Splunk reports that its users — especially the non-IT ones — really want to get Splunk information on the tablet devices. While this somewhat contradicts what I wrote a few days ago pooh-poohing mobile BI, let me hasten to point out:

• Splunk is used for a lot of (quasi) real-time monitoring.
• Splunk’s desktop user interfaces are, by BI standards, quite primitive.

That’s pretty much the ideal scenario for mobile BI: Timeliness matters and prettiness doesn’t.

Hmm. Maybe StreamBase LiveView needs a mobile option as well …

Splunk’s basic use is to take the text string that is a log and make sense of it. But Splunk now also supports JSON structures. It does this via something called spath, which as you might guess from the name has XPath similarities. That probably bore more discussion than we found the time to have.

By the way: If you’re interested in BI over XML, that’s what my former clients at Skytide were founded to do, before they pivoted a bit. I don’t think those capabilities have disappeared from the product.

Splunk has graciously allowed me to post a slide deck. More stuff in there, including quotes from a customer — Expedia — that has 2700 Splunk users.

T-Mobile’s instructions tell you that you need to know the factory defaults for network name and password. That makes sense. They don’t also tell you that you need to know your SIM card number (included), IMEI number (included), or authorization number (not included).

That’s right — you need a number that T-Mobile doesn’t tell you you need. But the story gets a lot worse from there, because it’s almost impossible to get the number from them. I eventually talked with approximately 8 T-Mobile call center associates over the course of the evening before getting successfully connected.

One of the few redeeming features in this story is that T-Mobile call center folks pick up the phone quickly. One of the many non-redeeming ones is that they efficiently give you inaccurate information after they do. The one who finally got me the right answer was a young woman who put me on hold to call an internal resource approximately four times before finally handling the situation correctly.

At one point I found somewhat helpful information by searching on T-Mobile’s website for mobile hotspot activation. However, the same information did not surface on my earlier searches on strings like activate mobile hotspot. Stemming has been a basic feature of search engines since the mid 1990s, but evidently T-Mobile’s technological choices aren’t as current as that. Other inexcusable T-Mobile mistakes include:

• Not providing information about the need for an “activation number” in the product’s paper documentation.
• Taking the buyer to a sign-on screen that doesn’t lead to the call center reps responsible for the product being signed-on for.
• Not providing call center operators with the tools they need to get callers to the right place.

Perhaps Elbonian* contractors were involved.

Elbonia is a fictitious country of outsourcers in Scott Adams’ Dilbert comic strip. Elbonian work is not noted for its high quality.

In case you haven’t guessed yet, the missing T-Mobile  “activation number” was tantamount to a telephone number, complete with area code. T-Mobile was insisting on assigning a telephone number for a service that had nothing to do with making or receiving telephone calls. While I can believe there was some legitimate database/application design reason for having such inflexibility under the covers, it’s hard to see why T-Mobile didn’t get a composite application tool and hack a front-end that automatically generates the number without call-center intervention.

I did eventually get connected, and in my limited experience with T-Mobile’s Prepaid/Mobile Hotspot 4G “Broadband” offering, I get the impression it has good speed but conventionally flaky WiFi reliability. It may well be a T-Mobile service that is worthy of great success. Edit: That’s not true.* But it’s not going to experience such success as long as T-Mobile idiotically infuriates its users at the relationship start.

*Edit: It turns out that the T-Mobile Mobile Hotspot device has terrible range. We can use it to get online in my office, Linda’s office, or the living room/dining room area, but no 2 of the 3 at once.

My takeaways from this story include:

• Use competent documentation writers.
• Run usability testing on your entire customer-experience processes.
• Test your site search engine for usefulness.

Beyond that, there’s not a single part of this story for which there isn’t a straightforward fix, most of them alluded to above.

If you see anything of your organization in this story, it’s probably time to raise your standards. This is obviously a different kind of failure as, say, the one last year at Chase. Even so, “as awful as T-Mobile” would be a sad state to endure.

• More-of-the-same in line with MarkLogic’s core positioning.
• A new bi-directional Hadoop connector.
• A free MarkLogic Express edition, limited in license terms more than in actual features, as per Slide 27 of the deck MarkLogic graciously supplied for me to post.

Also, MarkLogic is early with a feature that most serious DBMS vendors will soon have – support for tiered storage, with writes going first to solid-state storage, then being flushed to disk via a caching-style algorithm.* And as befits a sometime search-engine-substitute, MarkLogic has finally licensed a large set of document filters, from an Australian company called Isys. Apparently, the special virtue of the Isys filters is that they’re good at extracting not only text, but metadata as well.

*If there’s a caching algorithm that doesn’t contain a major element of LRU (Least Recently Used), I don’t recall ever hearing about it.

MarkLogic seems to have settled on a positioning that, although distressingly buzzword-heavy, is at least partly based upon reality. The real part includes:

• MarkLogic is a serious, enterprise-class DBMS (see for example Slide 12 of the MarkLogic deck) …
• … which has been optimized from the getgo for poly-structured data.
• MarkLogic can and does scale out to handle large amounts of data.
• MarkLogic is a general-purpose DBMS, suitable for both short-request and analytic tasks.
• MarkLogic is particularly well suited for analyses with long chains of “progressive enhancement” (MarkLogic’s favorite term when talking about derived data).
• MarkLogic often plays the role of a content assembler and/or search engine, and the people who use MarkLogic in those ways are commonly doing things that can be described as research and analysis.

Based on that reality, MarkLogic talks a lot about Volume, Velocity, Variety, Big Data, unstructured data, semi-structured data, and big data analytics.

My November, 2010 overview of MarkLogic technology remains pretty relevant. One correction, however: Node heterogeneity configurations, in which “data” and “evaluation” nodes reside on separate servers, are the exception rather than the rule.

Like Vertica, MarkLogic has laudably said that true academic researchers can get MarkLogic for free without the severe license restrictions. Free MarkLogic should be of particular interest to researchers who:

• Are studying natural networks or graphs, such as social networks or biological pathways. (This might be a fit in the social or biological sciences.)
• Are managing metadata for, say, a variety of disparate kinds of experimental files. (This might be a fit anywhere in the natural sciences.)
• Are managing actual documents, images, videos, etc., or data about such things. (This might be a fit in the humanities or social sciences.)

MarkLogic provided some disclosable financial substance by email, which I shall quote verbatim:

• MarkLogic has 45% revenue growth and 55-60% license growth year over year.
• We expect to finish this year with over $85 million in revenue, up from $55 million last year.

Arithmetical purists might note that 85/55 is more than 145%, but I’m just going to settle for the information I got and move on.

Edit: I posted separately about the MarkLogic Hadoop connector. As for that Hadoop connector – stay tuned for a short follow-up post, as writing about it now would not be convenient. (My backup discipline isn’t what it should be, and the only copy of my notes about that product is on a heavy tower computer in a house that doesn’t have working power.)

I’ve gone on for two long posts about text data management already, but even so I’ve glossed over a major point:

Using text data commonly involves a long series of data enhancement steps.

Even before you do what we’d normally think of as “analysis”, text markup can include steps such as:

• Figure out where the words break.
• Figure out where the clauses and sentences break.
• Figure out where the paragraphs, sections, and chapters break.
• (Where necessary) map the words to similar ones — spelling correction, stemming, etc.
• Figure out which words are grammatically which parts of speech.
• Figure out which pronouns and so on refer to which other words. (Technical term: Anaphora resolution.)
• Figure out what was being said, one clause at a time.
• Figure out the emotion — or “sentiment” — associated with it.

Those processes can add up to dozens of steps. And maybe, six months down the road, you’ll think of more steps yet.

So when you manage text, it is convenient to assume dynamic schemas. That would be an argument for using MarkLogic, NoSQL document stores, and/or Hadoop, rather than strictly relational systems.

That said, text analytics can be done perfectly well in relational databases. Again, I point you to the example of Attensity, which will extract for you a large fraction of the information that can be gotten out of the text, put it into a convenient relational schema, and let you get to work. Once the principal extraction has been done, there’s no reason why your derived data issues need be any more complex than others you deal with relationally, especially on the analytic side of the house.

But what if you want to do your own text enhancement, rather than using a third party tool? The first thing to ask yourself is — why? With all due respect to the 10-20 internet-centric companies that are having fun reinventing large portions of the data processing wheel — if you’re not at one of those companies, you should probably be trying to use as much third-party software as you possibly can.

I can think of a couple of cases where rolling your own technology make sense, namely:

• The hard part of what you’re doing is extracting snippets of text from some data format proprietary to you.
• You’re trying to do very simple things across a variety of languages much broader than the 10-20 that the text analytics vendors currently do a halfway decent job of handling.

I can’t think of many others.

One thing I’d definitely be wary of is using Hadoop as a big bit bucket for individual documents in a variety of formats. I don’t know what you’d do with them once they’re there. Yes, Google invented MapReduce in part to do things like document indexing — but you’d probably prefer not to reinvent the Google stack. That’s quite apart from questions as to whether your document count exceeds Hadoop’s comfortable file-count limit. Solr is a different matter; but while Solr and Hadoop are both open source projects that can be traced back to Doug Cutting, otherwise they’re rather different things.

A useful way of looking at your choices may be to ask:

After text has run through the main pipeline of manipulation and information extraction:

• What will the output look like?
• Where do I want that output to end up?

If the output has to be something that fits into a structured/relational analytic system, then it should probably go into a relational DBMS. If you’re going to do social network analysis of the sort you’d ideally like to do in a graph database — well, unless you’re an intelligence agency with blank-check resources, you’ll probably still end up opting for a relational DBMS. If the output consists of simple, homogeneous text files, plus a few fields of metadata, and you’re not going to do much analysis of it, it can pretty much go anywhere; either SQL or NoSQL might suit your purposes. If you want maximum power and flexibility, MarkLogic may be the ideal destination.

From there, the next question is:

• What pipeline should the text run through to get to its final destination?

Often, as I’ve argued, the right answer is a third-party text analytic system. Those can generally consume text in almost any kind of file format. Other times — less often than you may think — it’s Hadoop. OK, then pass it through Hadoop. Other possibilities could come up as well (text search engines aren’t really as usually as I may have seemed to be suggesting).

Anyhow, when you’ve established where text starts out (that’s usually a given), what it passes through (please see above), and where its best parts need to end up (ditto), you’ve done the hardest parts. Figuring out the rest of your text management architecture should be relatively easy by comparison.

1. Confusion about text data management.
2. Choices for text data management (general and short-request).
3. Choices for text data management (analytic).

I’ve recently given widely varied advice about managing text (and similar files — images and so on), ranging from

Sure, just keep going with your old strategy of keeping .PDFs in the file system and pointing to them from the relational database. That’s an easy performance optimization vs. having the RDBMS manage them as BLOBs.

to

I suspect MongoDB isn’t heavyweight enough for your document management needs, let alone just dumping everything into Hadoop. Why don’t you take a look at MarkLogic?

Here are some reasons why.

There are three basic kinds of text management use case:

• Text as payload.
• Text as search parameter.
• Text as analytic input.

The simplest way to manage text electronically is to:

• Store it as whole documents — scanned images, .PDFs, word processing files, whatever.
• Find it only via fielded metadata, perhaps manually created — title, author, date, and so on.

For example, an application for college admission is accompanied by recommendation letters, transcripts, and so on; those are then moved around as dumb payloads, until such time as an admissions officer reads them. Most relational database management systems can manage BLOBs (Binary Large OBjects), but performance may be better if you leave the big objects outside the relational system. For text-as-payload, that way of managing documents can often suffice.

In other cases, the text may be so short that it naturally fits into a character field in a relational database. This is particularly likely when the text is typed in at the time of record creation, e.g. by a call center operator, a doctor, or a customer entering a support ticket. In such cases, leaving it under the management of an RDBMS makes perfect sense.

In many situations you actually want to search based on the context of the text. Unless you’re doing simple search on short text snippets in relational character fields, that generally calls for some kind of text index. Text indexes are generally found in text search engines. That said, however:

• Anything that can be done in a standalone text search engine can in principle also be integrated into relational DBMS and other data stores. (How well that works in practice is another matter.)
• Text search engines commonly index data in situ that they don’t actually manage.

In theory, then:

• You can store your text in your chosen DBMS or outside it, as pleases you.
• Your chosen DBMS can have a text search capability.
• This text search capability can be integrated with the query method used to get at the rest of the data managed by that DBMS.

That theory is commonly reflected in actual products, such as Oracle and DB2.

As so often, then, the choice of how to manage text comes down to issues such as performance, programming ease, and other components of total cost of ownership (or of some other general “goodness” metric, such as time-to-value). As a general rule, it seems:

• Text indexing inside relational DBMS has poorer performance than in, say, text search engines, often drastically so.
• BLOB management inside relational databases has poorer performance than leaving the files outside the DBMS’ purview.
• Relational DBMS do just fine at managing text strings up to, say, 2048 characters long.
• Tight integration between  text search and SQL is valuable in a few applications, but irrelevant to many others.

And so, to a first approximation:

• If you just have short text snippets, it can make sense to leave them in your relational database.
• If performance is not an issue, you can just leave your BLOBs in your relational database too.
• If performance is an issue, you probably want to have your larger text files outside your RDBMS’ control.

However, that phrasing assumes the default option is a relational DBMS, which may not be the case at all. Other choices include:

• Standalone text search engines. If you want the best available text search, get a search engine. But attempts to start with a search engine and wind up with an application platform have generally run into difficulty.
• Document-oriented (or other) NoSQL systems. The story here is surprisingly like that for relational DBMS. I’ve previously noted that document-oriented NoSQL systems manage objects, not “documents” in the ordinary sense of the word. Even so, conceptually they’re no less suited for management of true documents than relational DBMS are. I’d guess that the correlation between use cases involving true documents and use cases where document-oriented NoSQL is suitable is positive, but not very strong.
• MarkLogic. From one standpoint, MarkLogic is just a heavier-weight version of document-oriented NoSQL. But MarkLogic’s XML (and XQuery) orientation, tuned-for-years indexing, and built-in search engine put it on a different level for document management than the upstarts have reached.

I’ll cover the Hadoop option in the next, more analytically-focused post.

I hope I’ve demonstrated that there are appropriate use cases for each of:

• Letting documents be managed by the file system (and pointing to them from your preferred DBMS).
• Sticking documents straight into your preferred DBMS (SQL or non-SQL as the case may be).
• Using a specialty system such as MarkLogic (or of course, in some cases, an enterprise search engine).

And that’s even before we move on to analytically-oriented text data management.

1. Confusion about text data management.
2. Choices for text data management (general and short-request).
3. Choices for text data management (analytic).

There’s much confusion about the management of text data, among technology users, vendors, and investors alike. Reasons seems to include:

• The terminology around text data is inaccurate.
• Data volume estimates for text are misleading.
• Multiple different technologies are in the mix, including:
  • Enterprise text search.
  • Text analytics — text mining, sentiment analysis, etc.
  • Document stores — e.g. document-oriented NoSQL, or MarkLogic.
  • Log management and parsing — e.g. Splunk.
  • Text archiving — e.g., various specialty email archiving products I couldn’t even name.
  • Public web search — Google et al.
• Text search vendors have disappointed, especially technically.
• Text analytics vendors have disappointed, especially financially.
• Other analytic technology vendors ignore what the text analytic vendors actually have accomplished, and reinvent inferior wheels rather than OEM the state of the art.

Above all: The use cases for text data vary greatly, just as the use cases for simply-structured databases do.

There are probably fewer people now than there were six years ago who need to be told that text and relational database management are very different things. Other misconceptions, however, appear to be on the rise. Specific points that are commonly overlooked include:

• The terms “unstructured” or “semi-structured” data are inherently misleading. That’s why I favor “multi-structured” or “poly-structured” instead. (“Multi-structured” seems to be winning; e.g., it’s been adopted by Teradata and Teradata/Aster.)
• The “social media” text data any one enterprise brings in house isn’t all that much. For example, Attensity serves many different enterprises’ social media needs from a single 20-terabyte data store, and reports that no single enterprise has required as much as 1 terabyte of text yet. Text data may consume a lot of storage on spinning disks somewhere, but it’s not that big a factor in future DBMS industry growth. (That 20 terabyte figure does seem low.)
• Structured databases are typically worth a lot more per bit than other kinds. The most valuable electronic data, per-bit, is probably records of significant economic transactions — purchases, sales, money transfers, etc. The least valuable may be sensor log files, whose contents consist mainly of “Nothing going on here; ping you again in a minute.” Email logs, web interaction data and many other kinds fall somewhere in between. Highly valuable documents — such as signed contracts — generally persist in paper as well as electronic forms. Investors commonly overlook this point.
• The enterprise text search industry is screwed up.
  • FAST was a goofy company before it was acquired for far too much money by Microsoft.
  • Autonomy was a goofy company before it was acquired for far too much money by HP.
  • Google’s enterprise efforts are quiet.
  • The integration of text search and relational DBMS — e.g. at Oracle — has languished, with poor performance and evident lack of management attention.
  • Smaller text search vendors don’t seem to be getting a lot of traction — e.g., Coveo has a decent reputation, but when’s the last time you heard much about them? What has Attivio actually accomplished?
• Text analytics is a small business. Add up the revenue for Attensity, Clarabridge, Lexalytics, Temis, and all the others, and you might poke above $100 million, especially now that Attensity had a three-way merger. Then again, you might not.
• Even so, the text analytics vendors have developed sophisticated technology. In particular, you can use it to get a pretty good idea as to what people are writing about you, individually or as groups.

NoSQL is most easily defined by what it excludes: SQL, joins, strong analytic alternatives to those, and some forms of database integrity. If you leave all four out, and you have a strong scale-out story, you’re in the NoSQL mainstream.

• Thus, I’d say Cassandra, HBase, Mongo DB, and Couchbase are prime examples, in no particular order. Riak as well.
• I might have phrased that better if I’d used a different word than simply “strong” — but hey, there was a 140-character limit, and he was on deadline.

Using NoSQL can make sense when at least one of two things is paramount: low-cost scale-out or dynamic schemas.

• There are some seriously sensible use cases for dynamic schemas.
• “Low-cost” generally boils down to:
  • Performance.
  • Open source free-like-beer.
  • Not a lot of database administration.

I’ve generally given object-oriented DBMS vendors and also MarkLogic hard times whenever they consider saying they’re “NoSQL”. Reasons include:

• Closed source.
• Database administration overhead (even if you get good stuff for incurring that overhead, like MarkLogic’s comprehensive indexing).

Also, NoSQL started out being ACID-unfriendly.

What you give up are the query flexibility and the easily automatic data integrity of SQL-based systems. I should have added something about a mature ecosystem.

In the most recent live example, I influenced a client away from Cassandra and toward scale-out MySQL (dbShards and/or Schooner flavors, most likely). Part of the reason was the ability to do joins, which are useful in their application. Another part is that their development practices obviated any significant benefit from dynamic schemas. But perhaps the most important — or at least resonant — reason of all was that they really, really cared about .NET support.

There really are three major points to NoSQL.

• Dynamic schemas. This is the only one of the three that truly depends on NoSQL.
• Scale-out short-request processing. If you want to scale out efficiently at high request volumes, you’re best off not using all the flexibility SQL/relational DBMS offer. (In particular, you don’t want to do cross-node joins). Not coincidentally, a number of the best scale-out offerings were built to be NoSQL.
• Open source. Doing a relational DBMS is a big project. It seems easier to build NoSQL ones.

Oracle can address the latter two points as aggressively as it wishes via MySQL. It so happens I would generally recommend MySQL enhanced by dbShards, Schooner, and/or dbShards/Schooner, rather than Oracle-only MySQL … but that’s a detail. In some form or other, Oracle’s MySQL is a huge player in the scale-out, open source, short-request database management market.

So that leaves us with dynamic schemas. Oracle has at least four different sets of technology in that area:

• As Workday noticed years ago, MySQL can be used as a functional, basic key-value store.
• Oracle also has XML-based Berkeley DB/SleepyCat kicking around.*
• The XML extensions to Oracle’s core DBMS could be alleged to have a dynamic schema/NoSQL flavor. (Blech.)
• A dynamic schema argument could also be made for object-oriented DBMS technology. While Oracle doesn’t to my knowledge exactly sell that, it does have the Tangosol Coherence line of technology, with a potentially similar programming model.

If Oracle is now refreshing and rebranding one or more of these as “NoSQL”, there’s no reason to view that as a big deal at all.

*That’s Mike Olson’s former company, if you’re keeping score at home.

• A long-ago marketing decision to not give infrastructure details, so as to convey a “Don’t worry; we’ll take care of everything” message.
• Even so, a long-ago and perhaps now-regretted marketing decision to disclose and even exaggerate salesforce.com’s reliance on Oracle, as part of an early-days attempt to prove salesforce was using enterprise-class technology.
• A desire to hide the recipe for salesforce.com’s secret sauce.
• Force of habit — I’m not sure salesforce even knows how to tell its technical story with any clarity.

Actually, salesforce.com has moved some kinds of data out of Oracle that previously used to be stored there. Besides Oracle, salesforce uses at least a file system and a RAM-based data store about which I have no details. Even so, much of salesforce.com’s data is stored in Oracle — a single instance of Oracle, which it believes may be the largest instance of Oracle in the world.

Salesforce did spell out some of its database story in a 2008 force.com white paper, which is good stuff, but potentially misleading in one important way. The paper tells of a level of abstraction, whereby what the application sees as logical “columns” are stored in a very different schema than one might assume. However, it doesn’t spell out a second level of abstraction, whereby that logical schema also isn’t how the database is actually laid out.

Another flaw in the paper is that it spins “We had to do this, to support multitenancy, so we did.” issues as “Because we’re multitenant, we can do this, while single-tenant systems can’t.” One example is the query optimization step around “user visibility” in Figure 11. Welcome to marketing.

At the first level of abstraction, data seems to be kept mainly in a single wide table, with hundreds of columns. What’s more, many of those are “flex columns”; a flex column can hold data of many different kinds and even datatypes. Notwithstanding the second level of abstraction, I imagine the idea of stuffing different kinds of thing into the same column has something to do with the fact that Oracle’s physical limit on columns falls far short of the number of logical columns salesforce wants to use.

If we imagine that the different kinds of data in a flex column were each in their own column instead, the whole thing might sound like BigTable/Cassandra/HBase-style column-group NoSQL. Thus, much as Workday uses MySQL to simulate a key-value store, salesforce.com can be said to use Oracle to simulate a different kind of NoSQL. In both cases, what’s going on seems to be a kind of object/relational mapping, but with the relational aspect strongly deemphasized. Or, if you take a more relational view, we could say that salesforce.com’s tables are a lot wider than any one user organization’s, because each user sees only its own custom columns (plus the standard ones common to all users).

The second layer of abstraction has a lot to do with multitenancy. If you want to stick data for many different user organizations into the same huge table, then you have to label it in some way to show who is permitted to see or update each part. Logically, this leads to a join, between one table carrying data plus a simple key showing which users/roles are entitled to see it, and a second table showing who actually is that kind of user/has that kind of role. But that join makes a lot of sense to store in a denormalized way, all the more because data is partitioned across the computer cluster in line with which user organization it actually belongs to.

Multitenant security isn’t the only reason for this denormalization, but it appears to be the biggest one.

The whole thing is doing 550 million or so transactions per day. salesforce.com thinks that fact should be regarded as evidence that it works.