• Give or take minor tweaks, what I wrote in my August, 2010 Couchbase updates still applies.
• Couchbase now and for the foreseeable future has one product line, called Couchbase.
• Couchbase 2.0, the first version of Couchbase (the product) to use CouchDB for persistence, has slipped …
• … because more parts of CouchDB had to be rewritten for performance than Couchbase (the company) had hoped.
• Think mid-year or so for the release of Couchbase 2.0, hopefully sooner.
• In connection with the need to rewrite parts of CouchDB, Couchbase has:
  • Gotten out of the single-server CouchDB business.
  • Donated its proprietary single-sever CouchDB intellectual property to the Apache Foundation.
• The 150ish new customers in 2011 Couchbase brags about are real, subscription customers.
• Couchbase has 60ish people, headed to >100 over the next few months.

If you previously heard the brand names Couchbase Single or Couchbase Mobile, pay no further attention to them. Couchbase Single was CouchDB; Couchbase Mobile is part of Couchbase’s feature set.

The current product is Couchbase 1.8, which is a whole lot like what previously was called Membase. New features in Couchbase 1.8 (versus prior versions of Membase) were concentrated in client libraries/SDK (Software Development Kit). Not coincidentally, Couchbase has hired developer evangelists who are in charge of making Couchbase play nicely with various specific languages (e.g. C/C++)

Drilling down further into the CouchDB part of the story:

• Couchbase 2.0 will replace Couchbase 1.8/Membase’s SQLite back-end with CouchDB.
• Parts of CouchDB that do things like read, write, or compact data have been rewritten from Erlang to C.
• Couchbase still uses other Erlang parts of Apache CouchDB, and would be delighted if the community were to usefully enhance them.
• Couchbase’s heavy contributions to development of open source CouchDB will, for the most part, continue.
• CouchDB stuff donated to the Apache Foundation includes:
  • Documentation
  • Packaging
  • Performance enhancements

There’s at least one Couchbase user with >1000 nodes (at a guess, Zynga).  More typical might be 20 nodes or less. This led me to wonder how much data one puts on a Couchbase node anyway. The answer turns out to vary widely, in that you want your working set to be in RAM, and whether that’s your entire database or just a slice of it depends on the nature of the application.

James echoed a trend I’ve heard elsewhere as well, in which products one things of as being internet-specific are also sold in a few cases to conventional enterprises for — you guessed it! — their internet operations. I also asked him about competition, and he asserted:

• MongoDB is the big competition. He believes Couchbase has an excellent win rate vs. 10gen for actual paying accounts.
• DataStax/Cassandra wins over Couchbase only when multi-data-center capability is important. Naturally, multi-data-center capability is planned for Couchbase. (Indeed, that’s one of the benefits of swapping in CouchDB at the back end.)
• Redis has “dropped off the radar”, presumably because there’s no particular persistence strategy for it.
• Riak doesn’t show up much.

• Splunk is first and foremost an analytic DBMS …
• … used to manage logs and similar multistructured data.
• Splunk’s DML (Data Manipulation Language) is based on text search, not on SQL.
• Splunk has extended its DML in natural ways (e.g., you can use it to do calculations and even some statistics).
• Splunk bundles some (very) basic, Splunk-specific business intelligence capabilities.
• The paradigmatic use of Splunk is to monitor IT operations in real time. However:
  • There also are plenty of non-real-time uses for Splunk.
  • Splunk is proudest of its growth in non-IT quasi-real-time uses, such as the marketing side of web operations.

As in any release, a lot of Splunk 4.3 is about “Oh, you didn’t have that before?” features and Bottleneck Whack-A-Mole performance speed-up. One performance enhancement is Bloom filters, which are a very hot topic these days. More important is a switch from Flash to HTML5, so as to accommodate mobile devices with less server-side rendering. Splunk reports that its users — especially the non-IT ones — really want to get Splunk information on the tablet devices. While this somewhat contradicts what I wrote a few days ago pooh-poohing mobile BI, let me hasten to point out:

• Splunk is used for a lot of (quasi) real-time monitoring.
• Splunk’s desktop user interfaces are, by BI standards, quite primitive.

That’s pretty much the ideal scenario for mobile BI: Timeliness matters and prettiness doesn’t.

Hmm. Maybe StreamBase LiveView needs a mobile option as well …

Splunk’s basic use is to take the text string that is a log and make sense of it. But Splunk now also supports JSON structures. It does this via something called spath, which as you might guess from the name has XPath similarities. That probably bore more discussion than we found the time to have.

By the way: If you’re interested in BI over XML, that’s what my former clients at Skytide were founded to do, before they pivoted a bit. I don’t think those capabilities have disappeared from the product.

Splunk has graciously allowed me to post a slide deck. More stuff in there, including quotes from a customer — Expedia — that has 2700 Splunk users.

• Data preparation, which is tedious unless you do a good job of automating it.
• Running the actual algorithms.

Numerous statistical software vendors (or open source projects) help you with the second part; some make strong claims in the first area as well (e.g., my clients at KXEN). Even so, large enterprises typically have statistical silos, commonly featuring expensive annual SAS licenses and seemingly slow-moving SAS programmers.

As I see it, the predictive analytics workflow goes something like this:

• Business-knowledgeable people develop a theory as to what kinds of information and segmentation could be valuable in making better business micro-decisions.
• Statistics-knowledgeable people determine a structure for modeling that reflects this theory.
• Statistics-knowledgeable people tweak the model over time, within a fixed general structure, as new data comes in.
• (Optional) Somebody sees to acquiring whatever data is needed that the organization doesn’t already have (and won’t get in the ordinary course of ongoing business).

The optional last part can be a purchase of third-party information (relatively fast and easy) or the development of a business process (and if necessary associated software) to capture the information (not always so easy). But even if that’s taken care of, or not present, we have at least two hand-offs where agility can be lost:

• Businesspeople may throw a request “over the wall” to the statisticians, who then work on it as their schedule permits.
• Once created, a model may be so set in stone that even small changes are as hard as building a new model from scratch.

The second problem can be solved by the statisticians themselves, without outside involvement. Model research and model refinement should be separate processes. You can recheck your clustering on one schedule, but recalibrate your regressions against each cluster more frequently. If that all sounds forbiddingly difficult, perhaps your model recalibration process needs another level of automation.

So I’ve finally gotten to the point of saying what may have been obvious from the start: The only excusable impediment to predictive analytic agility is the hand-off from the people who know the business to the people who know the math. So let’s examine ways that difficulty can be resolved.

At big internet companies, the usual answer is something like

Hey, it’s just data. From web logs. And network event logs. The data scientists know how to handle that.

In financial trading firms, the answer is more

The traders and analysts work closely together. Very closely. In fact, when the traders rip out their phones and throw them across the room, the analysts need to duck to avoid getting clobbered.

In credit card or telecom marketing or insurance actuarial organizations, the answer may be

Don’t worry; the stats geeks have been at this for a long time; they really do understand our business.

All three approaches work.

But what about conventional enterprises, where line-of-business people may not be as math-savvy as internet developers or financial traders, and where the math experts may not have the business issues down cold? My flippant answer is that businesspeople should know some math too.* My more serious answer is that the “business analyst” role should be expanded beyond BI and planning to include lightweight predictive analytics as well.

*I wasn’t being entirely flippant, of course. Statistics is even being taught in high school these days. And when I got a PhD in game theory, 2/3 of my thesis committee was at the Harvard Business School.

For example, at retailers:

• Market basket analysis is pretty simplistic (it only looks at small subsets of a basket at a time).
• Seasonality is tricky. (Weather and so on can skew it.)
• Each store or region can be its own universe.
• Some of the results of analytics are rather coarse-grained — e.g., merchandise adjacencies — so precision in statistical analysis may not matter much anyway.

And so truly rigorous statistical analysis may be both unfeasible and unnecessary; a lot of business-informed seat-of-the-pants reasoning needs to be mixed in. Consequently, there’s a lot to be said for pushing at least some retail predictive analytics pretty close to the merchandising department(s).

Similar stories could be told in many other industries and pursuits, including but emphatically not limited to:

• Event marketing.
• College admissions.
• Political campaigning.
• Field maintenance at utility companies.
• Price-setting (across many industries).

In each case, it’s easy to see how statistical and predictive analytic techniques could add real value to the business. But it’s hard to imagine how the enterprise could support the kind of large, experienced, business-knowledge analytic operation one might find in hedge fund investing or telecom churn analysis. And absent that, it’s tough to see why the only people doing predictive analytics for the organization should sit in some silo of statistical expertise.

• It’s called Odiago, and is one of my gratifyingly more numerous tiny clients.
• Odiago’s product line is called WibiData, after the justly popular We Be Sushi restaurants.
• We’ve agreed on a split exclusive de-stealthing launch. You can read about the company/founder/investor stuff on TechCrunch. But this is the place for — well, for the tech crunch.

WibiData is designed for management of, investigative analytics on, and operational analytics on consumer internet data, the main examples of which are web site traffic and personalization and their analogues for games and/or mobile devices. The core WibiData technology, built on HBase and Hadoop,* is a data management and analytic execution layer. That’s where the secret sauce resides. Also included are:

• REST APIs for interactive access.
• Import/export tools, including JDBC access.
• Management tools.
• Analytic libraries — data mining, predictive analytics, machine learning, and so on.

The whole thing is in beta, with about three (paying) beta customers.

*And Avro and so on.

The core ideas of WibiData include:

• ALL data pertaining to a single user (or mobile device) is kept in a single, possibly very long, HBase row.
• There are two primary operators in WibiData, Produce and Gather.
  • Produce operates on single rows. It can operate on one row at HBase speed (milliseconds) if you need to inform an interactive user response. Or it can operate on the whole database in batch via Hadoop MapReduce.
  • It is reasonable to think of Produce as mainly doing two things. One is the aforementioned serving of data out of WibiData into interactive applications. The other is scoring, classifying, recommending, etc. on individual users (i.e. rows), in line with an analytic model.
  • Gather typically operates on all your rows at once, and emits suitable input for a MapReduce Reduce step. It is reasonable to think of Gather as being a key cog in the training of analytic models.
• HBase schema management is done at the WibiData system level, not directly in applications. There’s a WibiData HBase data dictionary, powered by a set of system tables, that specifies cell data types/record types and, in effect, primitive schemas.

WibiData-enhanced HBase differs from relational DBMS in most of the ways you would imagine, both good and bad. In particular:

• Depending on how you look at it, WibiData-enhanced HBase either has no DML (Data Manipulation Language) at all, or else has one that ‘s a lot less rich than SQL.
• WibiData-enhanced HBase schemas are much more dynamic than SQL schemas.
• WibiData-enhanced HBase schemas can have nested or recursive data structures, such as array-valued cells.

To expand on each of those points in turn:

WibiData’s underlying one-giant-table philosophy notwithstanding, there are times you manage multiple tables with it. (For example, you ingest data into WibiData however you can, and then run transformations — typically batch — until the data is in the preferred structure.) While Wibidata does have ways to simulate joins, foreign keys, and so on, there’s nothing resembling referential integrity or foreign key constraints.

WibiData takes single-table schema flexibility to an extreme. Not only can different rows in the same table have different associated columns — something that relational systems can in effect also do via NULL values — but schemas can even change over the life of a column. If you have an array-valued cell storing the results of a marketing campaign, and you start recording more data partway through the campaign, then different rows in the table will, in the same column, hold different-sized arrays.

That nesting can also get pretty serious; where you’d have a single value in a relational table, you might have the equivalent of a whole relational table (or at least selection/view) in WibiData-enhanced HBase. For example, if a user visits the same web page ten times, and each time 50 attributes are recorded (including a timestamp), all 500 data – to use the word “data” in its original “plural of datum” sense – would likely be stored in the same WibiData cell.

That’s about all Odiago is disclosing about WibiData right now. Christophe will also be talking at Hadoop World next week, and presumably can be hit up with any burning questions then.

Oracle’s core DBMS is not well suited to track interactions (e.g. web clicks), even in cases where it’s the choice for transactions; it’s unnecessarily heavyweight. What’s worse, using the same database to store actions and interactions can lead to serious reliability problems. If a better architecture is to dump the clicks into some NoSQL store, massage the information, and eventually put some derived data into a relational DBMS, then Oracle will naturally try to own each step of the data pipeline.

Dynamic schemas are another area of Oracle weakness, leading in some cases to outright Oracle replacements. However, pure key-value stores go too far to the opposite extreme; you should at least be able to index and retrieve data one field at a time. Based on what I’ve seen of Oracle’s marketing literature, that feature will be missing from the first release of Oracle’s NoSQL.* Until it’s in there, and until it works well, I don’t see why anybody should use Oracle’s NoSQL product.

*Frankly, that choice makes no sense to me on any level. Yet it’s the way Oracle seems to have elected to go — or, if it isn’t, then there’s somebody writing Oracle marketing collateral who’s clearly in the wrong line of work.

The explanation was led by Oliver Ratzesberger, late of eBay*and progenitor of eBay’s Singularity project. In simplest terms, one event can spawn a lot of event attribute information, perhaps in the form of name-value pairs, which it then makes sense to store together in some way. The example Oliver dwelled on was that, on any given web page, there can be 100+ pieces of information to record, including:

• All 50 search results you were shown, and their positions in the search rankings.
• Every ad, image, or graphical element.
• An ID as to which test you were participating in (every page you see on eBay has some element being tested).

*Oliver is leaving eBay for a still-secret large company. I would conjecture that Michael McIntire is on the move too, either to replace Oliver or to go with him, but Oliver did a very good job of not commenting on the matter.

There are several reasons why one might wish to store this information in ways that grieve relational purists. First, reconstructing all this information via joins would be brutally expensive. What’s more, reconstructing all this information via joins could be impractical. Some comes from third party ad servers, which might not reproduce the same ads upon demand. Other is in the form of rankings, which can’t always be reliably reproduced from one query to the next. (That’s just one of several reasons text search and relational DBMS are an awkward fit.)

Also, there’s a strong dynamic schema flavor to these databases. The list of attributes for one web click might be very different in kind from the list for the next page. Forcing that kind of variability into a fixed relational schema, while theoretically possible, doesn’t necessarily make a lot of sense.

The total number of enterprises in the world paying subscription and license fees that they would regard as being for “Hadoop or something Hadoop-related” probably is not much over 100 right now, but I’d expect to see pretty rapid growth. Beyond that, let’s divide customers into three groups:

• Internet businesses.
• Traditional enterprises ‘ internet operations.
• Traditional enterprises’ other operations.

Hadoop vendors, in different mixes, claim to be doing well in all three segments. Even so, almost all use cases involve some kind of machine-generated data, with one exception being a credit card vendor crunching a large database of transaction details. Multiple kinds of machine-generated data come into play — web/network/mobile device logs, financial trade data, scientific/experimental data, and more. In particular, pharmaceutical research got some mentions, which makes sense, in that it’s one area of scientific research that actually enjoys fat for-profit research budgets.

On the partnering side, I heard things about a Hortonworks conference call that do not seem to have been contradicted by my visit to Hortonworks. Namely, Hortonworks promised prospective partners, such as analytic DBMS vendors, hardware vendors, or large system integrators, that it wouldn’t compete with them, in that Hortonworks pledges not to introduce its own products for at least two years. This is presumably targeted most directly at Cloudera, which has lots of partners, but also some proprietary code of its own. MapR, I’d think, would be the #2 target, but that’s just speculation.

The other big part of Hortonworks’ story is the claim that it holds the axe in Apache Hadoop development. Nobody doubts that a large fraction of the work on Hadoop’s core projects was done by Yahoo employees. Many of those indeed moved to Hortonworks; others left Yahoo earlier; Hadoop creator Doug Cutting is actually at Cloudera. So just how dominant Hortonworks really is in core Hadoop development is a bit unclear. Meanwhile, Cloudera people seem to be leading a number of Hadoop companion or sub-projects, including the first two I can think of that relate to Hadoop integration or connectivity, namely Sqoop and Flume. So I’m not persuaded that the “we know this stuff better” part of the Hortonworks partnering story really holds up.

What I am persuaded of is that the Hadoop platform competition is a good thing. Whichever vendors and projects win will be healthier from having had to outcompete worthy alternatives.

Also, Aster no longer focuses much on the general internet industry where it got its earliest sales, its continued success at LinkedIn and a recent win at an (NDA) fairly-big-name internet new account Razorfish notwithstanding. That said, the first target market Aster did share with me was “digital marketing optimization,” which includes “marketing optimization” (duh), search engine optimization (SEO), clickstream analysis, and the like. Also, Aster is going after “data scientists” in general, and that’s a term I’m still seeing used most frequently in the internet area.

I’m seeing ever more granularity as companies break down internet-related market segments. DataStax showed me a chart last week of 15 different market segments it had sold into, and at least 14 were in some way internet-related.

Rather, if Aster is to name three industries in which it has pleasingly strong sales traction, it would say manufacturing (which in Teradata lingo includes resource extraction), financial services (including insurance), and retail. A cynic might note that that breakdown, like many similar ones, adds up to fairly large swaths of the economy and the computer market, but never mind that part. (Other firms might have thrown in telecommunications and health care as well, to get even more coverage.

Two of Aster’s other favorite application areas are social network analysis/influencer identification and — which is analytically very similar — fraud detection/prevention. Taken together, that’s a whole lot of graph analysis. And I note with interest that the influencer identification stuff does NOT seem to be concentrated in telecom, which is the traditional sector one would imagine it being used in; all those call records are a lovely source of graph edges. Rather, the influencers seem to be identified from sources such as social media and credit card data .

Once again, this kind of thing gives me privacy jitters.

The match between Aster’s favorite industries and application areas is pretty much as you might expect — fraud in financial services, influencer analysis in retailing (and probably consumer financial services too), and digital marketing in both. As for manufacturing, the opportunities there seem to be focused on machine-generated data. That would be at least in high-tech manufacturing (I bet especially in flow-oriented stuff such as semiconductor fab) and oil/gas. Smart grid opportunities don’t seem to have arisen yet for Aster the way they have for a couple other vendors.

As for general Aster business trends, I think they’re good, while Aster would perhaps want to portray them as very good. Aster named a couple of impressive joint Teradata/Aster wins under NDA, but only a couple. Ramping up sales headcount is proving challenging, and some sales leadership turnover probably hasn’t helped. I do believe Aster’s spin that this is a matter of somebody being promoted quickly to a bigger job, and am optimistic about the current team — still, such moves tend to have at least short-term cost.

Context for any comments on customer traction includes:

• Membase went into limited production release in October, and full release in January. Similar things are true of CouchDB.
• Hence, most sales of Couchbase’s products have been made over the past 6 months.
• Couchbase (the merged product) is at this point only in a pre-production developer’s release.
• Couchbase has both a direct sales force and a classic open-source “funnel”-based online selling model. Naturally, Couchbase’s understanding of what its customers are doing is more solid with respect to the direct sales base.
• Most of Couchbase’s revenue to date seems to have come from a limited number of big-ticket “lighthouse” accounts (as opposed to, say, the larger number of smaller deals that come in through the online funnel).

That said,

• Most Membase purchases are for new applications, as opposed to memcached migrations. However, customers are the kinds of companies that probably also are using memcached elsewhere.
• Most other Membase purchases are replacements for the Membase/MySQL combination. Bob says those are easy sales with short sales cycles.
• Pure memcached support is a small but non-zero business for Couchbase, and a fine source of upsell opportunities.
• In the pipeline but not so much yet in the customer base are SaaS vendors and the like who use and may want to replace traditional DBMS such as Oracle. Other than among those, Couchbase doesn’t compete much yet with Oracle et al.
• Pure CouchDB isn’t all that much of a business, at least relative to community size, as CouchDB is a single-server product commonly used by people who are content not to pay for support.

Membase sales are concentrated in five kinds of internet-centric companies, which in declining order are:

• Social gaming
• Ad platforms
• Online retail
• Online business, including B2B  SaaS
• Social networking

Bob said that Couchbase often sees MongoDB competitively, but never Riak, HBase, or Redis. I got the impression Couchbase sees at least a little Cassandra. That would, of course, all pertain only to direct sales, rather than download/community kinds of usage.

Couchbase is also excited about the potential for the CouchDB-based Couchbase Mobile occasionally-connected offering. The hottest use cases, interestingly, seem to be non-consumer; Bob rattled off military, farming, and health care, and surely could have named more besides. However, the Couchbase Mobile sales effort still seems to be in early days, as is evidenced by the fact that Couchbase has not yet competitively encountered Sybase SQL Anywhere.

With all that said, I’ll go now to a separate post for a Couchbase technical update.

10gen recommends solid-state storage in many cases. In some cases solid-state lets you get away with fewer total nodes. 10gen also likes Flashcache (Facebook-developed technology to put a flash cache in front of hard disks). But the 100-node example mentioned above uses spinning disk.

Use cases 10gen is proud of include:

• Lots of user profile maintenance, including at online ad companies. This includes full user ad impression data. (I’ve argued for a while that user profile information belongs in something like a NoSQL database.)
• A big-name web company that wants to inspect every packet that enters their network, and replaced Splunk with MongoDB for performance reasons.
• A big-name photo/video site whose metadata is all in MongoDB. (That’s the kind of thing that often makes for good MarkLogic use cases.)

But actually, the reason we had the call was to review cases where MongoDB’s schemaless nature was significant. Examples of those included:

• A couple of top examples were of the kind “A bunch of apps, similar but not the same.” For MTV, it’s a single content management system for a bunch of websites. For Disney Playdom, it’s different schemas for every game.
• For a wireless telco, the issue was a product catalog in which devices and service plans called for very different schemas, and which the telco felt had thus become unmanageable in Oracle.
• For Craigslist, the issue wasn’t programming so much as performance — ALTER TABLE operations took months in MySQL, and that’s not a typo, although I’ll confess to not understanding why this was the case.

The 10gen guys went on to claim that schemalessness is helpful for incremental development in general, the point being that you don’t have a database-modification step. To some extent, changes can even be rolled back more easily than if you actually changed your schemas.