• People
• Financial trades
• Electronic networks
• Everything else

The most varied, interesting, and valuable of those four categories is the first one.

That may change some day, with the growing importance of machine-generated data, and of big-data science in particular. But I think it’s a fair assessment at the present, and for at least the next few years.

Some of the most interesting use cases are concentrated in the areas of identifying individuals, groups of people, or behaviors of (groups of) people. For example:

• comScore works hard to identify individual web surfers – i.e. to deanonymize them — even though they may have given incomplete or false personal information.
• Other companies at least try to figure out which information in a user’s profile is unreliable, so as to classify them better. (Yes, there are 62-year-old video-game-obsessed Lady Gaga fans, but that’s generally not the way to bet.)
• Multiple telecom vendors try to identify who their most influential customers are (to a first approximation, they’re the ones most often called by the most people, but it surely gets more sophisticated than that). This information is then used to reduce churn, either by working hard to retain those users, or – if they do churn – to move very fast to retain the business from their friends.
• Other kinds of companies do similar kinds of analysis, to the extent that they have enough of a social graph to do so. (This application is a case where the term “social graph” is not a misnomer.)
• Turing detectives (I just coined that phrase) try to determine whether users are humans or bots.
• Central to detecting insurance fraud is identifying suspiciously close connections between claimants, service providers, and so on.
• Identifying groups of people is also important in flagging insider trading. Even more important are other kinds of analysis, along the lines of “is this normal innocent trading behavior?”
• Intelligence agencies try to detect networks of terrorists and their sympathizers. They further try to identify unusual patterns of communication or meetings along those networks that might indicate terrorist acts are being planned. (Civilian law enforcement agencies can use similar techniques.)

In most cases, the analysis and/or run-time execution of the relevant models is done with the help of analytic DBMS. Other technologies that come into play include non-DBMS MapReduce (Hadoop), graph engines, and CEP (Complex Event Processing). The vendor most heavily represented on that list is probably Aster Data, because:

• Aster Data is focused on hard-core analytics.
• I talk a lot with Aster Data, and in particular had a long, detailed use-cases discussion with them last week.
• The comScore example happens to come from a speaker at an Aster event I also participated in.

And by the way, all this only scratches the surface of what will be possible down the road. It’s based mainly on where you live, what you purchase, how you behave on websites, and who you communicate with. Other kinds of data, which could be used to be yet more intrusive, generally aren’t involved.

I actually have two points in drawing up this list. One is golly-gee-whiz about how a lot of analytically sophisticated applications are actually getting into production. The other is to highlight the privacy and liberty threats If This Goes On Unchecked (which is why I didn’t include some other less-people-focused examples). There’s also a related danger that, to the extent we don’t get some smart regulations to keep us safe(r), we’ll get a bunch of stupid regulations instead.

The Analytic Era has only just begun.

• Aster Data showed me a lot of customer names and deal sizes, across a bunch of industries (mainly enterprise rather than web). Yes, Aster’s market success is for real. (But almost all those details are NDA.)
• Sybase’s product plans for IQ are pretty impressive. (But the most interesting parts are, you guessed it, NDA.)
• I’ve kissed and made up* with ParAccel, now that they’ve replaced their CEO, replaced their marketing chief, and stopped the worst of the marketing nonsense I used to complain about. ParAccel has some interesting plans for ParAccel 3.0 which are, naturally, NDA.
• The Peoplesoft guys are doing it over again at Workday. Only this time, their platform isn’t a relational DBMS. Rather, it’s an in-memory, completely object-oriented data model, with disk used only on a “Just in case the power ever goes out” basis. (Thankfully, nothing at all about our conversation was NDA.)
• I’m finally feeling good about Northscale’s memcached-compatible persistent store Membase. The main reason is that they showed me a near-term path to interfaces that are richer than key-value. Also, Todd Hoff reassured me that even pure persistent memcached has a place.
• Rumor says that even the one app for which Facebook was using Cassandra — in-box search — has been decommissioned. On the other hand, numerous other scale-0ut DBMS (SQL or otherwise) seem to have Facebook footholds. But details are — all together now! — NDA.

*If you know ParAccel’s new marketing chief Michael Weir, you surely guessed I mean that only in a figurative sense.

From elsewhere:

• Daniel Abadi offered his analysis of Kickfire’s demise. In general I agree, but Daniel neglected to mention one hugely important factor — the chicken-egg negative effect of Kickfire’s lack of market or marketing traction. Customers were extremely reluctant to buy from Kickfire because they perceived, correctly, that Kickfire’s survivability was far from assured.
• While the InfiniDB forums suggest that there are at least a couple of production users of Calpont’s free InfiniDB, Calpont seemingly has a long way to go to be even as successful as Kickfire. But Calpont does have a bit of money to spend on lead generation; maybe some day they’ll even have actual customers.
• In a response to a question I messaged over, XtremeData tells me they have actual customers now. Press releases to follow.
• The admiration for the job Mark Hurd did at HP is in my opinion overstated. Sure, the financial/operational management appeared to work, but HP did little on Hurd’s watch to strengthen its reputation or customers’ loyalty. In particular:
  • HP’s analytics efforts have accomplished little.
  • HP’s data warehouse appliance efforts have failed pathetically.
  • From what I hear, HP’s execution in its Exadata partnership was not good.
  • HP’s server business in general is distinguished mainly by HP being a big company.
  • HP’s EDS acquisition has been rocky, not that EDS was sailing so smoothly on its own beforehand.
  • HP’s success in PCs amounts to “arguably, HP sucks a little less than the other guys”.
  • HP’s elite reputation is long gone (admittedly, for the most part that predates Hurd).
• Doug Henschen evidently favors really strong intellectual property protection for software, even forbidding plug-compatible reverse engineering. I agree with Doug up to the point that it should be forbidden to copy proprietary software, but I don’t see why he (or a court) would view such behavior as copying.

But other highlights of his column include:

• Some of those accounts are rather big-name. However, I’m not at all sure whether they’re actual production references.
• Andy Mendelsohn characterizes the sweet spot of Exadata’s market as “virtual private cloud.” That matches what Juan Loaiza told me six months ago.
• Oracle claims numerous competitive wins for Exadata. Let me hasten to note that one vendor’s “competitive win” is another vendor’s “our salesman read the deal as an unfavorable one and chose not to compete,” or even sometimes “Huh? We never heard about that deal.” That said, what I’m hearing is that Exadata is indeed a much stronger competitor than it used to be.
• Oracle claims a near $1 billion sales run rate for Exadata. No doubt, a large majority of those are hardware upgrades for existing Oracle database customers, often from non-Sun/Oracle hardware. Even so, some of those are surely deals that would have migrated away from Oracle in the pre-Exadata past.

• Netezza (user conference)
• Aster Data (to steal some of Netezza’s thunder)
• Infobright (so far as I can tell, just because it was time for a product release, and also to get ahead of the summer doldrums)
• Northscale (ditto)

Time to play some catchup.

I’ll start with Aster Data, which added to the list of analytic packages it previously announced, and kindly gave me permission to post a partial slide deck from the briefing on same. Highlights of Aster’s analytic packages story include: 

• Statistics, statistics, and more statistics, including:
  • A deal with SAS
  • A lot of R packages
  • Deals with Fuzzy Logix and another partner
• Linear algebra/matrix manipulation, useful in:
  • You guessed it — statistics
  • Other machine learning
  • Optimization
  • and more
• Entity and pattern extraction, for example:
  • Market basket analysis, which is a great input for — I knew you’d keep up with this — statistics
  • Sessionization, ditto
  • Text tokenization (presumably pretty basic stuff)
  • The versatile nPath package
• Data mining/predictive analytics beyond straight statistics
• More stuff, some of it mentioned in my Aster-sponsored webinars late last year

Like Netezza, Aster is offering functions in two formats:

• Fully parallel — i.e., you can simply invoke them via SQL and they’ll execute in parallel
• Parallel-ready — i.e., you can invoke them on every node of the MPP cluster at once via Aster’s MapReduce framework

There are 30+ of the former and 1000+ of the latter, grouped into 40+ packages.

I will not repeat Aster’s current, confusing terminology for these two categories — hopefully Aster will do some renaming — but you can find same on Slide 6 of the deck linked above.

• Subsecond load latency is substantially impossible. Doing that amounts to OLTP.
• 5 seconds or so is doable with aggressive investment and tuning.
• Several minute load latency is pretty easy.
• 10-15 minute latency or longer is now very routine.

There’s generally a throughput/latency tradeoff, so if you want very low latency with good throughput, you may have to throw a lot of hardware at the problem.

I’d expect to hear similar things from any other vendor with reasonably mature analytic DBMS technology. Low-latency load is a problem for columnar systems, but both Vertica and ParAccel designed in workarounds from the getgo. Aster Data probably didn’t meet these criteria until Version 4.0, its old “frontline” positioning notwithstanding, but I think it does now.

Related link

• Just what is your need for speed anyway?

• Regarding Netezza, SAS Scoring Accelerator for Netezza currently supports Netezza Performance Data Server 4.6.  Support for Netezza TwinFin is slated for July 2010.
• Regarding the AsterData nCluster, I can understand your confusion on the distinction between Limited Availability (LA) and General Availability (GA). To help clarify, Limited Availability means that the technology is available for pre-qualified customers who are also active SAS Enterprise Miner users. In those cases, the product has been through QA and is available for purchase by those limited pre-qualified customers, and support for a limited number of customer sites is available. When the product becomes Generally Available, all qualifying customers are able to take advantage.
• Regarding your question of general parallelism/in-database capability, SAS currently is taking advantage of two technologies to provide scalability and performance. The first, SAS In-Database, takes the processing to where the data resides and parallelizes the analytical computations by leveraging the MPP architecture. The second technology, SAS Grid Manager, parallelizes computational steps or subparts of a process across different nodes and bring the results back together.
• To further expand upon Michelle’s answer to your question “What’s the big deal about in-database data mining scoring anyway?”, hopefully these additional bullets will help:
  • Manually converting the model scoring logic into SQL is difficult and time consuming in several instances. It would result in long hours and higher costs and would require testing and revalidation of the code. It will also restrict modelers to basic linear regression models. Automated model scoring (using SAS Scoring Accelerator) will allow modelers to use complex models like decision trees, neural networks, etc. (i.e. leverage full capabilities of SAS Enterprise Miner). (Note: Since I was comparing specialized in-database scoring to the in-database alternative of writing it all in SQL, this is actually the only one of the three bullets that address my question. )
  • Scoring is typically done on a periodic basis – either daily, weekly, monthly or on an event driven basis and many of our customers are scoring large numbers of models against enormous tables. Conventional scoring of SAS models requires connecting to the database server to extract rows to SAS for scoring. The scores are commonly bulk loaded back to the database. As the number of rows in the table grows over time, network latency grows because the amount of data that is fetched from the database to the SAS scoring process increases. SAS Scoring Accelerator reduces the unnecessary data movement and replication for analytic (i.e. model deployment) processing.
  • SAS Scoring Accelerator allows the scoring process to be linearly scalable, completely leveraging the power of the parallel shared-nothing architecture of the database or data warehouse in question.

Part of the time, I talked about things I’ve written about before. For example:

• Liberty and privacy. That’s a link to my most recent overview post on the liberty and privacy implications of modern analytic technology. The notes I spoke from were actually posted previously, after I spoke from them at the New England Database Summit at MIT in January. I’m gratified that, at both events, I got very positive feedback on liberty and privacy issues.
• Pick the right latency. That’s a link to a post (also based on a previous talk, in this case the one I traveled around the world giving for Netezza last September) in which I laid out the different levels of speed and latency an analytic application might require. I counted 9 orders of magnitude between the slowest and fastest, which is pretty much the difference between the speed of a turtle (at least a small, slow one) and the speed of light.
• On the more general point of operationalizing analytics, my best or at least most detailed writing to date may be in a 2004 whitepaper on analytic business processes which, sadly, is still fairly futuristic today.
• I offered a few ways to think about the different kinds of data that go into data warehouses.
  • Some of those were outlined in a post last January about three broad categories of data, distinguishing among human/tabular, human/non-tabular, and machine-generateddata.
  • That was a kind of sequel to a post last December about a three broad categories of data warehouse house growth drivers, namely more of the same vs. more detail vs. wholly new kinds of data.
  • I gave some examples of creating new data to analyze back in 2005 and 2006.
• Comments I made at various points were foreshadowed in a post on reinventing business intelligence.

I also raised a few points that I’m not finding good links for. I’ll try to cover those in future blog posts.

Related link

• Notes for my Boston Big Data Summit (no relation to Aster Data’s Big Data Summit series) talk in October, 2009

However, I chatted briefly Thursday with Michelle Wilkie, who is the SAS product manager overseeing all this (and also some other stuff, like SAS running on grids without being integrated into a DBMS). As best I understood, the story is:

• On Teradata, SAS is shipping in-database scoring today. SAS also is shipping a limited amount of in-database modeling on Teradata, the count recently having gone up from 4 “procs” to 10.
• On Netezza Twinfin(i), SAS is shipping in-database scoring, and this was recently announced. I can’t actually find much evidence of this announcement by searching the Web or the SAS website, but Michelle was pretty clear on the point even so.  Further confusing matters, SAS’ website seems to say in-database scoring is supported on Netezza’s old generation of products but not its latest one, even though SAS CTO Keith Collins told me exactly the opposite would be true.
• On Aster Data nCluster, SAS will ship in-database scoring by the end of 2010. If I understood correctly, this will be for “limited” rather than “general” availability, but Michelle framed that as a distinction without a difference. I.e., if you want to buy in-database SAS scoring on Aster nCluster, you’ll be able to.
• (More) in-database SAS modeling is expected on all of Teradata, Netezza Twinfin(i), and Aster Data nCluster in the vague future. (The concept of 2011/2012 came into play.)
• SAS/Teradata integration, developed first, involved more hand-coding. SAS has subsequently developed some kind of a more general parallelism/in-database capability, akin to what it has in the DBMS-less grid, that either is or isn’t a good match for DBMS vendors’ native way of supporting parallel processing. (Obviously, I’m still pretty unclear on this part.)
• SAS technology is a good fit for Aster Data’s MapReduce-centric way of doing parallelism.

I also took the opportunity to ask Michelle a question I’ve had a heck of a time getting answered: What’s the big-deal about in-database data mining scoring anyway? After all, the most common form of in-database data mining scoring is just to take a weighted sum of specific fields in a row, where the weights are the regression coefficients. You can do that in generic SQL, with performance that superficially should be at least as good as that for any alternative strategy. Michelle’s answers seemed to be twofold:

• There are other kinds of scoring too — neural networks, etc.
• Coding the scoring in SQL isn’t that easy. Michelle gave the example of a specific user (default Netezza reference account, with initials resembling mine) that spent 400 hours writing and testing something you now get for free with SAS/Netezza integration.

Edit: In response to this post, SAS wrote in with further clarification about in-database and/or MPP SAS.

On a rough-draft basis, my talk concept is:

Implications of New Analytic Technology in four areas:

• Liberty & privacy
• Data acquisition & retention
• Data exploration
• Operationalized analytics

I haven’t done any work yet on the talk besides coming up with that snippet, and probably won’t until the week before I give it. Suggestions are welcome.

If anybody actually has a link to a clear discussion of legislative and regulatory data retention requirements, that would be cool. I know they’ve exploded, but I don’t  have the details.

• A feed of MapReduce-related posts from several blogs, including this one.
• A calendar of MapReduce-related events, not necessarily Aster-specific, integrated with a feed combining …
  • … Aster MapReduce-related press releases and also …
  • … not necessarily Aster-specific MapReduce-related press articles.
• Links to a lot of Aster Data MapReduce-related collateral. Some of that stuff is quite good.*
• A sycophantic introduction from Colin White praising the value of the mapreduce.org “independent forum.”

*I did a couple of MapReduce-related webinars for Aster late last year. But seriously — Aster does a good job of writing clear and informative collateral.