• SAS faces a number of challenges, not unlike those faced by other high-priced legacy technology vendors.
  • It is used by organizations who have large budgets to pay for the product and to pay people to be expert on the product’s intricacies.
  • SAS has not integrated with scale-out analytic DBMS technologies as well or quickly as had been hoped, or as earlier marketing suggested was likely.
  • SAS has not been strong in helping its users do agile predictive analytics.
• SAS’ strengths are concentrated in product breadth:
  • Lots of statistical algorithms.
  • Various vertical products that make the modeling techniques more accessible in specific application domains.
  • Various approaches to engineering for scalability — no one of those has been a table-thumping success to date, but SAS has the resources to keep trying.
  • Some level of integration with its own business intelligence and text analytics products.
• For any particular use case, the burden of proof is on SAS alternatives to show that they have enough pieces in the toolkit to meet the needs.
  • SPSS (now owned by IBM) also has legacy issues.
  • KXEN is focused on marketing use cases.
  • Mahout has been one of the less successful Hadoop-related open source projects.
  • R-based technology is still maturing.
  • The modeling capabilities (as opposed to just scoring) bundled into RDBMS and well-parallelized tend to be pretty limited. Apparent exceptions tend to just be R repackaged.

• Data preparation, which is tedious unless you do a good job of automating it.
• Running the actual algorithms.

Numerous statistical software vendors (or open source projects) help you with the second part; some make strong claims in the first area as well (e.g., my clients at KXEN). Even so, large enterprises typically have statistical silos, commonly featuring expensive annual SAS licenses and seemingly slow-moving SAS programmers.

As I see it, the predictive analytics workflow goes something like this:

• Business-knowledgeable people develop a theory as to what kinds of information and segmentation could be valuable in making better business micro-decisions.
• Statistics-knowledgeable people determine a structure for modeling that reflects this theory.
• Statistics-knowledgeable people tweak the model over time, within a fixed general structure, as new data comes in.
• (Optional) Somebody sees to acquiring whatever data is needed that the organization doesn’t already have (and won’t get in the ordinary course of ongoing business).

The optional last part can be a purchase of third-party information (relatively fast and easy) or the development of a business process (and if necessary associated software) to capture the information (not always so easy). But even if that’s taken care of, or not present, we have at least two hand-offs where agility can be lost:

• Businesspeople may throw a request “over the wall” to the statisticians, who then work on it as their schedule permits.
• Once created, a model may be so set in stone that even small changes are as hard as building a new model from scratch.

The second problem can be solved by the statisticians themselves, without outside involvement. Model research and model refinement should be separate processes. You can recheck your clustering on one schedule, but recalibrate your regressions against each cluster more frequently. If that all sounds forbiddingly difficult, perhaps your model recalibration process needs another level of automation.

So I’ve finally gotten to the point of saying what may have been obvious from the start: The only excusable impediment to predictive analytic agility is the hand-off from the people who know the business to the people who know the math. So let’s examine ways that difficulty can be resolved.

At big internet companies, the usual answer is something like

Hey, it’s just data. From web logs. And network event logs. The data scientists know how to handle that.

In financial trading firms, the answer is more

The traders and analysts work closely together. Very closely. In fact, when the traders rip out their phones and throw them across the room, the analysts need to duck to avoid getting clobbered.

In credit card or telecom marketing or insurance actuarial organizations, the answer may be

Don’t worry; the stats geeks have been at this for a long time; they really do understand our business.

All three approaches work.

But what about conventional enterprises, where line-of-business people may not be as math-savvy as internet developers or financial traders, and where the math experts may not have the business issues down cold? My flippant answer is that businesspeople should know some math too.* My more serious answer is that the “business analyst” role should be expanded beyond BI and planning to include lightweight predictive analytics as well.

*I wasn’t being entirely flippant, of course. Statistics is even being taught in high school these days. And when I got a PhD in game theory, 2/3 of my thesis committee was at the Harvard Business School.

For example, at retailers:

• Market basket analysis is pretty simplistic (it only looks at small subsets of a basket at a time).
• Seasonality is tricky. (Weather and so on can skew it.)
• Each store or region can be its own universe.
• Some of the results of analytics are rather coarse-grained — e.g., merchandise adjacencies — so precision in statistical analysis may not matter much anyway.

And so truly rigorous statistical analysis may be both unfeasible and unnecessary; a lot of business-informed seat-of-the-pants reasoning needs to be mixed in. Consequently, there’s a lot to be said for pushing at least some retail predictive analytics pretty close to the merchandising department(s).

Similar stories could be told in many other industries and pursuits, including but emphatically not limited to:

• Event marketing.
• College admissions.
• Political campaigning.
• Field maintenance at utility companies.
• Price-setting (across many industries).

In each case, it’s easy to see how statistical and predictive analytic techniques could add real value to the business. But it’s hard to imagine how the enterprise could support the kind of large, experienced, business-knowledge analytic operation one might find in hedge fund investing or telecom churn analysis. And absent that, it’s tough to see why the only people doing predictive analytics for the organization should sit in some silo of statistical expertise.

• Consumer financial services. The idea here is to combine information about customers’ use of all kinds of services — banking, credit cards, loans, etc. SAS believes this is both for marketing and risk analysis purposes.
• Insurance. We didn’t go into detail.
• Mobile communications. SAS’ customers aren’t giving it details, but they’re excited about geocoding/geospatial data.

Meanwhile, in another interview I heard about, SAS emphasized retailers. Indeed, that’s what spawned my recent post about logistic regression.

The mobile communications one is a bit scary. Your cell phone — and hence your cellular company — know where you are, pretty much from moment to moment. Even without advanced analytic technology applied to it, that’s a pretty direct privacy threat. Throw in some analytics, and your cell company might know, for example, who you hang out with (in person), where you shop, and how those things predict your future behavior. And so the government — or just your employer — might know those things too.

• SAS no longer plans to go as far with in-database modeling as it previously intended.
• Rather, SAS plans to run in RAM on MPP DBMS appliances, exploiting MPI (Message Passing Interface).

The whole thing is called SAS HPA (High-Performance Analytics), in an obvious reference to HPC (High-Performance Computing). It will run initially on RAM-heavy appliances from Teradata and EMC Greenplum.

A lot of what’s going on here is that SAS found it annoyingly difficult to parallelize modeling within the framework of a massively parallel DBMS such as Teradata. Notes on that aspect include:

• SAS wasn’t exploiting the capabilities of individual DBMS to their fullest; rather, it was looking for an approach that would work across multiple brands of DBMS. Thus, for example, the fact that Aster’s analytic platform architecture is more flexible or powerful than Teradata’s didn’t help much with making SAS run within the Aster nCluster database.
• Notwithstanding everything else, SAS did make a certain set of modeling procedures run in-database.
• SAS’ previous plans to run in-database modeling in Aster and/or Netezza DBMS may never come to fruition.

SAS’ problems developing in-database modeling stem from, in essence, the limitations of UDFs (User Defined Functions). So why weren’t, for example, Teradata’s 2009 enhancements to its UDF capabilities enough? The clearest example SAS gave me is that, while database tables are commonly limited to something on the order of 1000 columns (their figure as well as mine), SAS might need 50-100,000 columns. One reason seems to be interactions between variables; SAS used the word “multiplied” a few times, but even so was coy about whether this could simply be regarded as quadratic terms in a regression. Another reason seems to be that in some cases, every value in a column spawns a new column in an intermediate table/array; indeed, this seems to be going on in the previously discussed case of logistic regression.

SAS code will be launched by the DBMS/data warehouse appliances, so potentially it can run under their native workload management. Teradata presumably has enough workload management richness to exploit that; EMC Greenplum, as of my August 2010 notes, probably did not.

SAS was gracious enough to let me post its slide deck, in both shorter and longer versions. Due to a technical glitch during the call, I neither looked at the slides nor took notes. I think the biggest loss from those difficulties is that I didn’t learn what the futures at the end of the longer deck were all about.

Related links

• Application areas for SAS HPA (April, 2011)
• SAS’ MPP story as of May, 2010
• SAS’ plans to run in-database on Teradata (October, 2007)

Revolution Analytics business and business model highlights include:

• Revolution Analytics is an open-core vendor built around the R language. That is, Revolution Analytics offers proprietary code and support, with subscription pricing, that help in the use of open source software.
• Unlike most open-core vendors I can think of, Revolution Analytics takes little responsibility for the actual open source part. Some “grants” for developing certain open source R pieces seem to be the main exception. While this has caused some hard feelings, I don’t have an accurate sense for their scope or severity.
• Revolution Analytics also sells a single-user/workstation version of its product, freely admitting that this is mainly a lead generation strategy or, in my lingo, a “break-even leader.”
• Revolution Analytics boasts around 100 customers, split about 70-30 between the workstation seeding stuff and the real server product.
• Revolution Analytics has “about” 37 employees. Headquarters are at 101 University Avenue (do I have to say in what city? ). There are also a development office in Seattle and a sales office in New York.
• Revolution Analytics’ pricing is by size of server. “Small” servers — i.e. up to 12 cores — start at $25K/year.
• Unsurprisingly, adoption is more alongside SAS et al. than rip-and-replace.

Revolution Analytics’ top market sector by far appears to be financial services, both in trading/investment banks/hedge funds and in credit cards/risk analysis. Pharma/life sciences is second, but sales cycles are slow. There’s also been at least a little activity each in a variety of internet/media/entertainment/gaming/telecom sectors.

When I asked Revolution Analytics why one would use R rather than, say, SAS, Revolution cited three reasons that seemed to be driving customer interest:

• You can do more with R. That may be debatable, but what’s harder to dispute is that there are a bunch of things you can do straightforwardly in R and its thousands of routines that would at best be more difficult in SAS.
• Students today are learning R, so you have access to (affordable?) talent. That’s pretty clearly correct, although I do note SPSS’ long history of academic social sciences use .
• R is cheaper. It’s hard to argue with that one.

Revolution Analytics’ parallelized-R story starts something like this:

• Although R is generally thought of as requiring all data to be in RAM, Revolution also offers external memory algorithms. (“External memory algorithms” seems to be the discipline-standard way of saying “Not all data has to be in RAM.”)
• In principle, Revolution is willing to parallelize external memory algorithms for you any which way — MapReduce, MPI (Message Passing Interface), and more.
• Revolution parallelized for multi-core last fall. Multi-server scale-out is coming this summer.
• Revolution is working on Netezza support. Revolution expects to use nzMatrix in the effort.
• Yes, logistic regression is one of the algorithms Revolution parallelizes.

Like Netezza with nzMatrix or Greenplum (now EMC) with its sparse vector routine, Revolution has some useful underpinnings to help with parallelization/scale-out as well. The main one seems to be a variance/covariance matrix, which can be arbitrarily large and can be computed in a very distributed way. Revolution notes that you can use this not just on data but also, for example, on parameters.

One analytic approach — if not meta-approach — that Revolution sees as hot is ensemble learning. Specifically mentioned was Max Kuhn’s caret package, which evidently automates ensemble techniques. Also specifically mentioned was the Netflix Prize, which I gather was won by an ensemble approach. The idea behind ensemble techniques is that, rather than pick a particular kind of model, you throw a bunch against the wall. The first benefit is that you get to see what works best. The second benefit is that you can combine results and hopefully outperform any one of the models.

Obviously, ensemble techniques can require vastly more performance than just running a single model. I wouldn’t be surprised if, going forward, they turned out to be one of analytics’ biggest performance challenges.

On the other hand, Aster Data said it had parallelized logistic regression a year ago. (Slides 6-7 from a mid-2010 Aster deck may be clearer.) I’m guessing Fuzzy Logix might make a similar claim, although I’m not really sure.

What gives?

• EMC Greenplum no longer uses my services.
• EMC Greenplum no longer briefs me.
• EMC Greenplum reneged on a commitment to fund an effort in the area of privacy.

The last one really hurt, because in trusting them, I put in quite a bit of effort, and discussed their promise with quite a few other people.

Yes, that five-word sentence really seems to have been the problem. I’ve heard that from more than one source.

I think the rest is overwrought too, and not just because I regret the loss of revenue, or of what seemed to be a warm, friendly, hug-laden, and sushi-intensive relationship with Scott Yara and some other folks. At various times, on the subject of its eBay installation:

• Greenplum overoptimistically told me that eBay’s Teradata installation would be replaced with Greenplum gear.
• Greenplum exaggerated the pace of its eBay installation; unfortunately, I believed them, and later had to publish a retraction.
• Greenplum neglected to tell me when eBay had its Greenplum equipment removed.

Now the same Scott Yara who hovered over me for months in marketing micromanagement before I broke the news of the Greenplum and Teradata eBay installations — he could do that because the whole discussion started out under NDA — doesn’t answer my email. Evidently, Greenplum thinks it’s OK to repeatedly be misleading, but doesn’t think it’s OK if my nuance is one they disagree with.

The most entertaining example I recall of Greenplum BS was when CTO Luke Lonergan told 50+ academics at the 2009 XLDB that Greenplum had 10 customers with half a petabyte each of data. I followed him out of the room and said “10 customers — half a petabyte each — I presume that’s for sufficiently small values of ‘one half’?” We eventually settled on a value of “one half” in the 0.2 range — which is actually a pretty impressive claim in itself.

Be all that as it may, EMC Greenplum has a couple of press releases out on which I’ve been asked to comment. One is a deal with SAS, less impressive than SAS’ deals with Teradata and Aster Data in that it offers no actual in-database modeling. Yes, it sounds like modeling on the same nodes where the data sits, but it sounds less desirable than true in-database modeling in that:

• You can only get great performance if the amount of data modeled is small enough to fit into RAM.
• Integration with other database processing, MapReduce, etc. may be limited.

Also, EMC Greenplum expanded its line of appliances, to include one that seems optimized for price-per-terabyte and one with solid-state drives. So far, that’s very standard stuff. There’s also a new data loading appliance, which seems to catch up with the Aster Data’s 2008 strategy of having separate nodes for bulk loading.

Ironically, when Aster moved away from a total reliance on that strategy, it was becoming more Greenplum-like. As is so often the case, it seems that different vendors’ feature sets are converging.

Meanwhile, the last I heard about Greenplum’s previously very strategic Chorus effort is that it’s being revamped. I don’t get the impression it’s nearly as central to Greenplum’s strategy as it used to be.

Evidently Mike Olson of Cloudera is still telling the machine-generated data story, exactly as he should be. The Information Arbitrage/IA Ventures folks said something similar, focusing specifically on “sensor data” …

… and, even better, went on to say: 

Privacy is dead.
What do we consider to be the boundaries of privacy, especially with respect to items like medical data? In a data privacy-free world, should we be regulating data usage instead? How do we deal with asymmetric access to our personal data, e.g., how is it that insurance companies claim the right to our personal information?

Obviously, my answer to the second question is Yes!!!!

Also from Hadoop World — Dave Menninger, now an analyst, reports on some Hadoop metrics:

How big is “big data”? In his opening remarks, Mike shared some statistics from a survey of attendees. The average Hadoop cluster among respondents was 66 nodes and 114 terabytes of data. However there is quite a range. The largest in the survey responses was a cluster of 1,300 nodes and more than 2 petabytes of data. (Presenters from eBay blew this away, describing their production cluster of  8,500 nodes and 16 petabytes of storage.) Over 60 percent of respondents had 10 terabytes or less, and half were running 10 nodes or less.

That eBay comment was particularly interesting.

A while back, Doug Henschen noted that Netezza flagship reference Catalina Marketing is now at 2.5 petabytes. Most of that is in one 600 billion row table. Oddly, the article talks of the Netezza/SAS partnership accelerating model-building via in-database scoring (not modeling) technology. Doug also wrote of a lot of analytic DBMS replacements, including:

• Microsoft by ParAccel
• Oracle by Aster Data, IBM, Oracle Exadata, probably Netezza, and probably Hadoop
• Netezza by Greenplum
• IBM by Teradata

Carl Olofson pointed out on Twitter that DataScaler was an in-memory database technology just bought by Oracle. This inspired me to google on them, and I found a sparse DataScaler CEO blog. I link it because of an amusing juxtaposition — the second-to-last post says, in effect, “We make appliances and we recommend all these awesome technology design partners who helped us design the hardware,” while the very last post says “Designing our own hardware was a mistake.”

Fred Holahan is now VP of Marketing at VoltDB, which is a lesson to me about giving free consulting … Anyhow, Fred tells me that VoltDB has about a dozen users on their way to production, some of whom are headed to being VoltDB paying customers, some of whom are not.

• Aster Data is generally an exception to this rule, as it should be, since that integration is at the core of its positioning. Even so, in the last paragraph of that link, I called Aster out for what at that time was some product description nonsense, which was specifically in an area that many vendors are confusing about explaining, namely …
• … the distinction between three kinds of parallelization.
  • If you do something entirely in SQL on an MPP system that parallelizes SQL — then it’s parallel!
  • If you have a parallelization framework such as SQL or MapReduce that can invoke the same function on every node — well, then that’s parallel!
  • Many algorithms — including almost every important statistical one — have to be explicitly coded to be parallel if they’re actually going to run in in parallel. The seminal paper on parallel data mining shows that such parallelization is, in many important cases, straightforward — but somebody still has to take the trouble to actually do it.
• Netezza TwinFin i-Class was renamed/repackaged/repriced before it ever shipped. Even so, when Tim Young or Phil Francisco tries to recall exactly the “i” stands for, comedy ensues. And the post I promised to write about Netezza TwinFin i-Class in June (as per the last sentence of this post) hasn’t happened yet, for reasons other than lack of interest on my part.
• SAS/DBMS integration tends to be a multi-year process, with in-database scoring coming long before in-database modeling. The drip-drip-drip of big-company PR over that time period can be quite bewildering …
• … especially since SAS partners in some cases are shipping home-grown in-database modeling long before SAS gives it to them.
  • See Slides 7-8 of a recent Aster Data slide deck.
  • The main point of Netezza’s nzMatrix is that Netezza needed it for in-database modeling algorithms it was building.
• After backing off from its early endorsement of MapReduce, Greenplum pretty much went to the other extreme and didn’t talk about its advanced analytics capabilities at all.

• Both on this blog and Reddit, there’s been considerable pushback against my idea that web usage types of user profile data shouldn’t be cluttering up an ACID-compliant database. But there’s also been considerable support, e.g. from Dan Weinreb, who knows quite a lot about huge OLTP systems.
• Meanwhile, RJP supplied details about the JP Morgan Chase Oracle outage that my actual source didn’t know.
• For obvious reasons, IBM wasn’t in a position to talk a lot of IBM/Netezza detail when we happened to chat post-merger-announcement. But they did want to set me straight on SAS being kicked out for SPSS, pointing out that SAS runs in the DB2 database today (scoring, not modeling).
• Product marketer Stephanie McReynolds added on to my post about Aster Data nCluster 4.6 in exactly the way I wish all vendors would. She added information I had been unsure about when I did the post — or had simply left out — and she was fast in doing so. I encourage all vendors I write about to follow her example.
• The comments on my post about Pentaho’s ETL-for-HDFS made the product sound more appealing than the post itself did.
• My August 18 NoSQL post was tailor-made for people to add-on pitches for their own favorite products, NoSQL-oriented websites, etc. A number of interesting such additions showed up accordingly.
• There were many thoughtful responses to my question about how somebody should teach themselves database programming skills. Indeed, whole other blog posts were written and linked back. That’s a great resource if you ever get the question asked by a friend or acquaintance of you.
• The flame war that erupted in response to my comments on vendor and analyst ethics spawned a number of more productive discussions elsewhere.
• Jeff Hammerbacher has made various comments to the effect “Yes indeedy! Hadoop does that too!” (My wording, not his. )
• Alin Dobra reported on some tests suggesting sequential reads remain far faster than random reads even on Flash SSDs.
• Calpont has an ever-slicker website and yet another new marketing VP, but no customers that are easy to detect.
• My July 4 privacy post engendered thoughtful discussion from three of the smartest guys who comment here — Chris Bird, Michael McIntire, and Dan Weinreb.
• IBM and Netezza both added crunchy details to my post about their data compression strategies.
• And for those of you who don’t read my other blogs — last night’s post was a long and optimistic rumination on the future of investigative reporting.