• Give or take minor tweaks, what I wrote in my August, 2010 Couchbase updates still applies.
• Couchbase now and for the foreseeable future has one product line, called Couchbase.
• Couchbase 2.0, the first version of Couchbase (the product) to use CouchDB for persistence, has slipped …
• … because more parts of CouchDB had to be rewritten for performance than Couchbase (the company) had hoped.
• Think mid-year or so for the release of Couchbase 2.0, hopefully sooner.
• In connection with the need to rewrite parts of CouchDB, Couchbase has:
  • Gotten out of the single-server CouchDB business.
  • Donated its proprietary single-sever CouchDB intellectual property to the Apache Foundation.
• The 150ish new customers in 2011 Couchbase brags about are real, subscription customers.
• Couchbase has 60ish people, headed to >100 over the next few months.

If you previously heard the brand names Couchbase Single or Couchbase Mobile, pay no further attention to them. Couchbase Single was CouchDB; Couchbase Mobile is part of Couchbase’s feature set.

The current product is Couchbase 1.8, which is a whole lot like what previously was called Membase. New features in Couchbase 1.8 (versus prior versions of Membase) were concentrated in client libraries/SDK (Software Development Kit). Not coincidentally, Couchbase has hired developer evangelists who are in charge of making Couchbase play nicely with various specific languages (e.g. C/C++)

Drilling down further into the CouchDB part of the story:

• Couchbase 2.0 will replace Couchbase 1.8/Membase’s SQLite back-end with CouchDB.
• Parts of CouchDB that do things like read, write, or compact data have been rewritten from Erlang to C.
• Couchbase still uses other Erlang parts of Apache CouchDB, and would be delighted if the community were to usefully enhance them.
• Couchbase’s heavy contributions to development of open source CouchDB will, for the most part, continue.
• CouchDB stuff donated to the Apache Foundation includes:
  • Documentation
  • Packaging
  • Performance enhancements

There’s at least one Couchbase user with >1000 nodes (at a guess, Zynga).  More typical might be 20 nodes or less. This led me to wonder how much data one puts on a Couchbase node anyway. The answer turns out to vary widely, in that you want your working set to be in RAM, and whether that’s your entire database or just a slice of it depends on the nature of the application.

James echoed a trend I’ve heard elsewhere as well, in which products one things of as being internet-specific are also sold in a few cases to conventional enterprises for — you guessed it! — their internet operations. I also asked him about competition, and he asserted:

• MongoDB is the big competition. He believes Couchbase has an excellent win rate vs. 10gen for actual paying accounts.
• DataStax/Cassandra wins over Couchbase only when multi-data-center capability is important. Naturally, multi-data-center capability is planned for Couchbase. (Indeed, that’s one of the benefits of swapping in CouchDB at the back end.)
• Redis has “dropped off the radar”, presumably because there’s no particular persistence strategy for it.
• Riak doesn’t show up much.

• Splunk is first and foremost an analytic DBMS …
• … used to manage logs and similar multistructured data.
• Splunk’s DML (Data Manipulation Language) is based on text search, not on SQL.
• Splunk has extended its DML in natural ways (e.g., you can use it to do calculations and even some statistics).
• Splunk bundles some (very) basic, Splunk-specific business intelligence capabilities.
• The paradigmatic use of Splunk is to monitor IT operations in real time. However:
  • There also are plenty of non-real-time uses for Splunk.
  • Splunk is proudest of its growth in non-IT quasi-real-time uses, such as the marketing side of web operations.

As in any release, a lot of Splunk 4.3 is about “Oh, you didn’t have that before?” features and Bottleneck Whack-A-Mole performance speed-up. One performance enhancement is Bloom filters, which are a very hot topic these days. More important is a switch from Flash to HTML5, so as to accommodate mobile devices with less server-side rendering. Splunk reports that its users — especially the non-IT ones — really want to get Splunk information on the tablet devices. While this somewhat contradicts what I wrote a few days ago pooh-poohing mobile BI, let me hasten to point out:

• Splunk is used for a lot of (quasi) real-time monitoring.
• Splunk’s desktop user interfaces are, by BI standards, quite primitive.

That’s pretty much the ideal scenario for mobile BI: Timeliness matters and prettiness doesn’t.

Hmm. Maybe StreamBase LiveView needs a mobile option as well …

Splunk’s basic use is to take the text string that is a log and make sense of it. But Splunk now also supports JSON structures. It does this via something called spath, which as you might guess from the name has XPath similarities. That probably bore more discussion than we found the time to have.

By the way: If you’re interested in BI over XML, that’s what my former clients at Skytide were founded to do, before they pivoted a bit. I don’t think those capabilities have disappeared from the product.

Splunk has graciously allowed me to post a slide deck. More stuff in there, including quotes from a customer — Expedia — that has 2700 Splunk users.

• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

• As one might expect, QlikView 11 contains fairly leading-edge stuff, but also some “better late than never” features.
• The leading-edge stuff is concentrated in the general area of “collaboration”.
• Additionally, QlikTech is always pushing the QlikView user interface ahead in various ways.
• The “Well, it’s about time!” feature list starts with the ability to load QlikView via third-party ETL tools (Informatica now, others coming).
• QlikTech is generally good at putting up pretty pictures of its product. You can find some in the “What’s New in QlikView 11″ document via a general QlikView resource page.*
• Stephen Swoyer wrote a good article summarizing QlikView 11.

*One confusing aspect to that paper:  non-standard uses of the terms “analytic app” and “document”.

As QlikTech tells it, QlikView 11 adds two kinds of collaboration features:

• Integration with social media, which QlikTech calls “asynchronous integration.”
• Direct sharing of the QlikView UI, which QlikTech calls “synchronous integration.”

I’d add a third kind, because QlikView 11 also takes some baby steps toward what I regard as a key aspect of BI collaboration — the ability to define and track your own metrics. It’s way, way short of what I called for in metric flexibility in a post last year, but at least it’s a small start.

That direct sharing of user interfaces is a cool feature, which every business intelligence vendor should offer. In an era of distributed workforces, when people can’t be assumed able to huddle around the same desk, it has value even for use among close coworkers. But it also should prove useful in a variety of more naturally remote use cases, multiple examples of which can be found in each of the areas of:

• Support (internal or external).
• Faceoffs — I mean collaborations — between two or more enterprise departments. Examples might include: manufacturing and purchasing, manufacturing and sales, or accounting and anybody else.

As for social media being used for BI collaboration — that’s generally in the air. For example:

• salesforce.com is pushing enterprise social media use broadly, and will surely increase its emphasis on the social media/BI intersection now that Dave Kellogg is there.
• Spotfire has announced similar features in its latest release.
• The more cumbersome side of the feature set (portal-based collaboration, emailing of individual reports) has been available from multiple vendors for years.
• eBay open-sourced a more dataset-centric version of the idea, just as Oliver Ratzesberger left the firm.*

*Umm — does anybody have a link to the project, or at least a name for it?

BI has been a communication tool since the first green paper report was dumped on the first desk. And there’s been collaboration in doing analysis at least since it’s been possible to email .XLS file attachments. Still, BI is too often used as bludgeon rather than binocular. Hopefully, the current generation of technology will finally serve to change that.

• I need to revise my figures for SAND’s customer count way downward.
• SAND finally has a reasonably clear positioning.
• SAND’s product actually seems to have a lot of features.

A few months ago, I wrote:

SAND Technology reported >600 total customers, including >100 direct.

Upon talking with the company, I need to revise that figure downward, from > 600 to 15.

One embarrassing point: SAND is a client, and I view it as part of my job to save clients from that kind of inadvertent misstatement.

It turns out that SAND has a very impressive customer — Dunnhumby, a data mart outsourcer with 200 terabytes of data in SAND, 30 or so incoming data streams, 400 or so nodes … and 600 or so end customers, all of which SAND was counting as OEM end customers for its DBMS. But I, other industry observers, and other vendors generally don’t count that way.

Besides Dunnhumby, SAND has 14 other customers on maintenance, with < 1 terabyte of data each. Until recently, SAND had a couple dozen more customers than that, but it sold its SAP-oriented archiving/near-line storage product line to Informatica.

I still don’t know where the “> 100 direct” part came from.

After the sale of its other product line, SAND is squarely in the market for analytic DBMS. SAND’s sales efforts seem to be focused on investigative analytics, although some of its existing users seem to be more focused on operational analytics. Most specifically, SAND is trying to focus on “people data” — customer loyalty, health care, etc . — rather than purely machine-generated data, with the paradigmatic target application being personalized marketing.

SAND technical highlights include:

• SAND sells a columnar analytic DBMS.
• The SAND DBMS operates on bitmaps, with heavy use of run-length encoding on the bitmaps. Bitmaps are used for everything except BLOBs (Binary Large OBjects).
• Actual data compression also comes into play, e.g. as result sets are being assembled. This is based on a true global dictionary — multiple columns are tokenized together.
• Indeed, SAND can decompose columns and tokenize their parts (e.g. time stamps).
• SAND’s workload management sees RAM and CPU, but not explicitly I/O.
• SAND lets you pin certain tables or even table segments in RAM if you want to.

SAND’s update story is straightforward — when data comes in, all the columns and bitmaps are updated as needed. Still, since SAND is columnar, you wouldn’t expect true updates in place, and you’d be right. Rather, there’s a story with MVCC (MultiVersion Concurrency Control) and garbage collection, lock-free. The MVCC is also exploited for a kind of time travel, and further for some kind of virtual data mart capability.

SAND’s parallelization story is a bit complicated.

• SAND has, or at least has the potential for, node specialization, with database and storage nodes being different.
• In principle, disks are specific to storage nodes, and it’s a configuration option as to whether a database node sees one, some, or all storage nodes.
• In practice, only Dunnhumby among SAND’s customers operates on other than a shared-disk basis. Dunnhumby’s configuration is mixed/matched among various SAND sharing options.

SAND is proud of its PMML (Predictive Modeling Markup Language) scoring capabilities, but otherwise hasn’t shipped much in the way of analytic platform capabilities. That said, work is underway on a user-defined table function capability that can also query external tables, fire off MapReduce jobs, and so on, under the code name UQL.

Top-level points included:

• Exasol’s technical philosophy is substantially the same as before, albeit not with as extreme a focus on fitting everything in RAM.
• Exasol believes its flagship DBMS EXASolution has great performance on a load-and-go basis.
• Exasol has 25 EXASolution customers, all in Germany.*
• 5 of those are “cloud” customers, at hosting providers engaged by Exasol.
• EXASolution database sizes now range from the low 100s of gigabytes up to 30 terabytes.
• Pretty much the whole company is in Nuremberg.

*That excludes some money from Hitachi. Exasol’s Hitachi partnership is still in limbo, an apparent casualty of the world economic crisis.

On the technical side:

• As noted in my 2008 post, EXASolution is a columnar, no-head-node MPP (Massively Parallel Processing) DBMS.
• The main way EXASolution compresses data is via dictionary/tokenization. 5:1 is a typical compression ratio before mirroring and so on, out of a 2-10:1 range.
• EXASolution writes data to blocks in memory that are smaller than what is otherwise its preferred size (1/2 to 5 megabytes). These are sent to disk, where merge eventually happens. Exasol insists that write performance has always been fully satisfactory to customers to date.
• EXASolution doesn’t have much in the way of performance tuning knobs. Exasol says they aren’t needed, and says that one really can start an EXASolution POC (Proof of Concept) in a day or so.
• EXASolution doesn’t have much in the way of workload management capabilities, except what’s automagic (e.g., short query bias). However, it does collect statistics you can query via your favorite BI tool.
• EXASolution doesn’t have much in the way of analytic platform capabilities, although there is some Lua-based scripting. However, there’s something NDA in the analytic platform area Coming Soon.*

In general, the whole thing sounds somewhat like ParAccel, at least at a high level.

*Exasol is not and never has been our client, but we can keep secrets for them even so.

Naturally, Exasol believes EXASolution has fine concurrency, with at least one customer routinely running 2000 concurrent users, 200 concurrent sessions (via connection pooling), and 5-10 concurrent queries. Another customer has 3500 Cognos users. 1-200 concurrent queries appears to be the record peak load. Anyhow, Exasol says that plans to offer real workload management could be accelerated if a need were discovered.

Exasol says it almost never loses POCs, but admits that it competes fairly rarely against Vertica and ParAccel, no doubt for reasons of geography. Exasol boasts one visible Sybase IQ replacement (Sony Music).

While Exasol’s sales to date have been in Germany, there are plans to change that soon. At least one sales cycle is well underway in Eastern Europe. Offices in other Germanic countries are planned. Existing customers are planning to deploy additional copies outside Germany. Discussions are underway regarding other geographies, e.g. English-speaking ones.

T-Mobile’s instructions tell you that you need to know the factory defaults for network name and password. That makes sense. They don’t also tell you that you need to know your SIM card number (included), IMEI number (included), or authorization number (not included).

That’s right — you need a number that T-Mobile doesn’t tell you you need. But the story gets a lot worse from there, because it’s almost impossible to get the number from them. I eventually talked with approximately 8 T-Mobile call center associates over the course of the evening before getting successfully connected.

One of the few redeeming features in this story is that T-Mobile call center folks pick up the phone quickly. One of the many non-redeeming ones is that they efficiently give you inaccurate information after they do. The one who finally got me the right answer was a young woman who put me on hold to call an internal resource approximately four times before finally handling the situation correctly.

At one point I found somewhat helpful information by searching on T-Mobile’s website for mobile hotspot activation. However, the same information did not surface on my earlier searches on strings like activate mobile hotspot. Stemming has been a basic feature of search engines since the mid 1990s, but evidently T-Mobile’s technological choices aren’t as current as that. Other inexcusable T-Mobile mistakes include:

• Not providing information about the need for an “activation number” in the product’s paper documentation.
• Taking the buyer to a sign-on screen that doesn’t lead to the call center reps responsible for the product being signed-on for.
• Not providing call center operators with the tools they need to get callers to the right place.

Perhaps Elbonian* contractors were involved.

Elbonia is a fictitious country of outsourcers in Scott Adams’ Dilbert comic strip. Elbonian work is not noted for its high quality.

In case you haven’t guessed yet, the missing T-Mobile  “activation number” was tantamount to a telephone number, complete with area code. T-Mobile was insisting on assigning a telephone number for a service that had nothing to do with making or receiving telephone calls. While I can believe there was some legitimate database/application design reason for having such inflexibility under the covers, it’s hard to see why T-Mobile didn’t get a composite application tool and hack a front-end that automatically generates the number without call-center intervention.

I did eventually get connected, and in my limited experience with T-Mobile’s Prepaid/Mobile Hotspot 4G “Broadband” offering, I get the impression it has good speed but conventionally flaky WiFi reliability. It may well be a T-Mobile service that is worthy of great success. Edit: That’s not true.* But it’s not going to experience such success as long as T-Mobile idiotically infuriates its users at the relationship start.

*Edit: It turns out that the T-Mobile Mobile Hotspot device has terrible range. We can use it to get online in my office, Linda’s office, or the living room/dining room area, but no 2 of the 3 at once.

My takeaways from this story include:

• Use competent documentation writers.
• Run usability testing on your entire customer-experience processes.
• Test your site search engine for usefulness.

Beyond that, there’s not a single part of this story for which there isn’t a straightforward fix, most of them alluded to above.

If you see anything of your organization in this story, it’s probably time to raise your standards. This is obviously a different kind of failure as, say, the one last year at Chase. Even so, “as awful as T-Mobile” would be a sad state to endure.

The explanation was led by Oliver Ratzesberger, late of eBay*and progenitor of eBay’s Singularity project. In simplest terms, one event can spawn a lot of event attribute information, perhaps in the form of name-value pairs, which it then makes sense to store together in some way. The example Oliver dwelled on was that, on any given web page, there can be 100+ pieces of information to record, including:

• All 50 search results you were shown, and their positions in the search rankings.
• Every ad, image, or graphical element.
• An ID as to which test you were participating in (every page you see on eBay has some element being tested).

*Oliver is leaving eBay for a still-secret large company. I would conjecture that Michael McIntire is on the move too, either to replace Oliver or to go with him, but Oliver did a very good job of not commenting on the matter.

There are several reasons why one might wish to store this information in ways that grieve relational purists. First, reconstructing all this information via joins would be brutally expensive. What’s more, reconstructing all this information via joins could be impractical. Some comes from third party ad servers, which might not reproduce the same ads upon demand. Other is in the form of rankings, which can’t always be reliably reproduced from one query to the next. (That’s just one of several reasons text search and relational DBMS are an awkward fit.)

Also, there’s a strong dynamic schema flavor to these databases. The list of attributes for one web click might be very different in kind from the list for the next page. Forcing that kind of variability into a fixed relational schema, while theoretically possible, doesn’t necessarily make a lot of sense.

2. That said, the Hadoop part of EMC ‘s story is based on MapR, which so far as I can tell is actually a pretty good Hadoop implementation. More precisely, MapR makes strong claims about performance and so on, and Apache Hadoop folks don’t reply “MapR is full of &#$!” Rather, they say “We’re going to close the gap with MapR a lot faster than the MapR folks like to think — and by the way, guys, thanks for the butt-kick.” A lot more precision about MapR may be found in this M. C. Srivas SlideShare.

3. On its latest earnings call, Oracle clearly said it would introduce a Hadoop appliance, versus just hinting at a Hadoop appliance the prior quarter. The money quote was: 

Finally, big data or the searching of large amounts of data using Hadoop. After Hadoop finishes filtering the data, the place you want to put that data is an Oracle Database, and that’s what a lot of our customers are doing. And we are exploiting the trend, the big data technology and the big data trend, if you prefer, by building a Hadoop appliance that attaches to the Oracle Exadata database or any Oracle Database for that matter. But you don’t have to buy our Hadoop appliance if you can use whatever servers you want running Hadoop, and we provide the interface between Hadoop and the Oracle Database.

In other words, Oracle is saying “We’d like to sell you a Hadoop appliance, but you can run Hadoop in some other way and we’ll coexist with it just fine.” That makes sense; refusing to coexist with Hadoop is not exactly a realistic option.

4. Back in June, I expressed great skepticism about the idea of a Hadoop appliance. There was at least partial pushback in the comment thread from both Amr Awadallah and Eric Baldeschwieler. Oops.

Their reasoning seems to be centered around matters of installation, administration, and general packaging.

5. A month ago I noted aggressive near-term plans for Apache Hadoop evolution. As noted above, one reason this is needed is competition from folks like MapR. Also, I note that:

• Three years ago, Oliver Ratzesberger’s group at eBay complained that CPU utilization running Hadoop was at 18%.
• Now Oliver uses a figure of 10-15%., and attributes an even lower figure to — I’m guessing here — Yahoo. (Another possibility might be Facebook.)
• In between eBay became one of the biggest and most prominent users of Hadoop.

The moral of eBay’s Hadoop adventures, as I see it, is neither “Hadoop sucks!” nor “Hadoop doesn’t suck!”; rather, it’s that there’s a lot of scope for Hadoop to operate differently in the future than it does today.

Similarly, whatever throughput Yahoo does or doesn’t get, it clearly has adopted Hadoop at the expense of the columnar-in-Postgres system it previously was so proud of.

Also, there has been a claim going around that — notwithstanding NameNode’s status as a single point of Hadoop failure —  no Hadoop installation has ever lost data due to a NameNode failure. The folks at MapR beg to differ, and sent over some links that sure seem to say the opposite.

6. Since we’ve just established that Hadoop will change, rapidly and pretty fundamentally, what exactly is the benefit of an appliance that is “balanced” for Hadoop usage today?

Pro tip: If you choose to announce at a conference where many other vendors will surely announce news also, you naturally run the risk of not garnering much attention.

For starters, it may help to realize or recall that:

• Cassandra was originally developed and revealed at Facebook, to much early NoSQL fanfare. Facebook later backed away from Cassandra use.
• Rackspace guys in Texas became Cassandra’s biggest backers. They eventually founded a company called Riptano to commercialize Cassandra.
• Texas company Riptano became the California company DataStax.
• DataStax came out with a Hadoop-on-Cassandra offering called Brisk. For a while, it sounded as if Hadoop was as big a focus for DataStax as Cassandra is.
• DataStax is now recommitted to being the Cassandra company, and has accordingly backed away from Hadoop and Brisk as a separate or coequal focus. However, it sees Hadoop capability as a nice, or even major, feature of its Cassandra-centric offering.
• To finalize its open source obligations with respect to Brisk, DataStax is in essence:
  • Donating a Hive driver for Cassandra straight into the main Apache Cassandra project.
  • Releasing the rest of Brisk as a separate open source project.
  • Disclaiming interest in further advancing open source Brisk.
• There’s also something called Solandra — evidently SOLR-on-Cassandra — whose status is similar to Brisk’s.
• There are three main ways that DataStax helps you to consume Cassandra.
  • DataStax is the principal sponsor of Apache Cassandra development, and presumably long will be. Apache Cassandra is both free-like-speech and free-like-beer.
  • DataStax is also introducing a paid-subscription version of Cassandra called DataStax Enterprise, which features proprietary code, support, and so on. DataStax Enterprise is neither free-like-speech nor free-like-beer.
  • There will also be something called DataStax Community Edition. DataStax Community Edition is free-like-beer, but not free-like-speech.

Various posts on the DataStax blog give DataStax’s explanation of what it’s doing. Ben Werther, the ex-Greenplum guy who briefly worked at DataStax and was most associated with telling the Hadoop/Brisk story, has moved on to his own startup Platfora.

DataStax Enterprise has three main aspects:

• DataStax Server, which is the actual database and analytics code. At this time, there is little closed-source code in DataStax Server, but DataStax reserves the right to widen that gap in the future.
• DataStax OpsCenter, which is management tools around DataStax Server. DataStax OpsCenter is entirely closed-source, even though DataStax gives a limited version away for free.
• Support.

To describe DataStax Community Edition, I’ll just quote the press release verbatim, which characterizes it as:

… a free platform based on Apache Cassandra that bundles the open source database with smart installers, drivers and connectors for popular development languages, demo apps, documentation, and a free version of DataStax OpsCenter for Apache Cassandra.

DataStax Community Edition is crippleware only in terms of feature set; there are no limitations on its database size, cluster size, or usage rights. A core mission of DataStax Community Edition is to create happy Cassandra users, who may then become customers for DataStax Enterprise.