Analytic technology isn’t only for you. It’s also for your customers, citizens, and other stakeholders.

I am not referring here to what is well understood to be an important, fast-growing activity — providing data and its analysis to customers as your primary or only business — nor to the related business of taking people’s data, crunching it for them, and giving them results. That combined sector — which I am pretty alone in aggregating into one and calling data mart outsourcing — is one of the top several vertical markets for a lot of the analytic DBMS vendors I write about. Rather, I’m talking about enterprises that gather data for some primary purpose, and have discovered that a good secondary use of the data is to reflect it back to stakeholders, often the same ones who provided or created it in the first place.

For now I’ll call this category stakeholder-facing analytics, as the shorter phrase “stakeholder analytics” would be ambiguous.* I first picked up the idea early this decade from Information Builders, for whom it had become something of a specialty. I’ve been asking analytics vendors for examples of stakeholder-facing analytics ever since, and a number have been able to comply. But the whole thing is in its early days even so; almost any sufficiently large enterprise should be more active in stakeholder-facing analytics than it currently is.

*Comments as to what the category should be called are welcome below.

Examples of stakeholder-facing analytics include:

• Enterprises report back on the business customers do with them. For example:
  • Credit card companies provide reports on spending back to their credit card holders, especially small businesses.
  • So do office supply retailers.
  • Brokerage firms provide reporting back to their small-institution customers.
• Governments expose information to their citizens online.
  • In an early example, New York City restaurant ratings were put online.
  • Putting SEC filings online has has been a huge success.
  • The Obama Administration has committed to putting large amounts of information online.
• Regulated companies (such as utilities) could be required to put data online directly, without even using the government as an intermediary.
• Some part of Fox — perhaps MySpace Music? — offers free access to a PostgreSQL extract from its Greenplum database to each of its largest advertisers.
• Google Analytics offers some basic BI for free to website owners everywhere.
• Anybody from web hosting companies to public utilities could open their kimonos and allow their customers to track adherence to actual or implied SLAs (Service Level Agreements) in areas such as uptime, length of outage, responsiveness, and the like.

So what cool examples do you have of stakeholder-facing analytics?*

*Yes, this is an invitation to drop links to case studies into the comment thread below.

• Tom Foremski outlined the dire (at least in theory) privacy risks of geolocation services, going into a lot more detail on that point than I ever have. However, he topped that off with the odd claim that people pay toll (rather than using an electronic service) to cross the Bay Bridge because they fear being tracked, rather than for reasons of time or money.
• Oracle had an earnings conference call. Larry Dignan did a good job of covering the highlights; the gory details are on the Seeking Alpha transcript, especially pp. 3-5.  Oracle now claims to be getting lots of multi-system deals for Exadata. (But I still haven’t seen much in the way of production customers named.) ULAs, which I presume are Unlimited License Agreements, are important on the software side. Besides picking on IBM and SAP, Oracle even touted a competitive win vs. EMC, which not coincidentally seems to be working on partnering with almost every Oracle competitor it can find.
• Brian Prentice of Gartner basically accused open source of being Dotcom 2.0, in terms of dubious business models and the hype associated with same. I agree with many of his particulars, and indeed often steer vendor clients away from open source strategies. For marketing purposes, I do feel that sometimes free can be a real cool price; but open source is not the only way to be free.
• Akiban, which I wrote about a couple of days ago, seems to be building out its website. As of this writing the website is still pretty raw, with bewildering messaging, carelessly repeated paragraphs, and a notable lack of clues as to who’s in company leadership. Even so — unless I missed some of the current stuff before, the site has come a long way in a few days, so maybe there’s hope.
• Groovy Corporation, which introduced the Groovy SQL Switch just last summer, seems to be doing something different now. It’s merged into a company called uCirrus (where the u is really a mu), but uCirrus doesn’t have a meaningful website yet, whereas Groovy does. There’s stuff there about a “push data cloud,” stressing the importance of not being a DBMS, under the name Cortex, whatever that all means. Groovy seems to have an online gaming deal for Cortex with MySpace, or maybe Cortex is just the name of a specific Groovy/MySpace project.
• Mike Mooney offered a long rant on the problems with database (design) version control. He did concede that the most recent Microsoft Visual Studio might help, for those who are bought into (and can afford) the Microsoft stack. Frankly, I think that’s what views are for, updatable or otherwise. In many cases, they’ll let you build what you need, quickly and without breaking anything, and you can leave it to the DBAs to sort out database performance later.
• I just discovered Chad Stewart’s programming blog. While he’s evidently a game programmer, a lot of his comments have broader applicability.
• Chip Hazard offered a VC’s perspectives on the difficulties facing enterprise IT startups. (Hat tip to Miriam Tuerk for turning me on to him.) Although he didn’t phrase it this way, his bottom line (at least the part I agree with) is that the startup’s products have to be amazingly superior to the alternatives (big vendors or in-house).

Splunk’s technical stories include:

• Text search over log files.
• Business intelligence over text search. (That part sounds a lot like Attivio.)
• MapReduce with schema flexibility and smart multi-stage execution plans. (That part sounds a lot like Aster Data.)

More on those in a separate post.

Less technical Splunk highlights include:

• Splunk has ~1200 paying customers, and is adding a couple hundred more per quarter.
• Splunk has ~160 people.
• ~80% of Splunk sales are in North America.
• Typical Splunk sales prices are in the $10-50K range, with an average around $25K, or maybe that average is a bit over $30K. Some Splunk deals are six- or even seven-figure.
• Splunk is “quite profitable.”
• Splunk’s eponymous product is priced according to how much data is indexed per day. If you index half a gigabyte of logs per day or less, Splunk is completely free. So, while Splunk is closed-source, there’s something of an open-source-like Splunk adoption model.
• Splunk has been selling product for a couple of years. I gather Splunk 4 was recently released.
• Splunk’s biggest industry segments are, not too surprisingly,
  • Telco
  • Financial services
  • Government
  • “Online”
• Splunk’s paying customers seem to use it mainly for:
  • Web logs and associated network event logs (this seems to be the biggest area)
  • Security and perhaps other general IT log analysis
  • Physical security logs (mainly in the government)
  • Anti-fraud (I’m not sure how that works)
• One would think Splunk would be used to manage a lot of intelligence telemetry, but that wasn’t particularly hinted at.
• In general, the core problem Splunk is used for is log analysis for trouble-shooting purposes.
• Splunk’s nonpaying users are more diverse; examples mentioned included windmill operations and protein research.
• Splunk’s customers include Aster Data flagship accounts MySpace and LinkedIn. I bet many other top web companies are Splunk customers as well.

• In most cases, no.
• Most of these technologies are designed for simple, high-volume OLTP (OnLine Transaction Processing.) Most large enterprises have an established way of doing OLTP, probably via relational database management systems. Why change?
• MapReduce is an exception, in that it’s designed for analytics. MapReduce may be useful for enterprises. But where it is, it probably should be integrated into an analytic DBMS.
• There’s one big countervailing factor to all these generalities — schema flexibility.

As for the longer form, let me start by noting that there are two main kinds of reason for not liking SQL. First, you might be fine with the idea of a (somewhat) nonprocedural, schema-aware DML/DDL (Data Manipulation/Description Language), but just think another kind is better, or more suited to your use case. If your reason is like that, you might favor alternatives such as:

• OLAP-based languages such as MDX.
• XML-oriented languages.
• “True” relational languages, because SQL deviated from the path of relational virtue under the corrupt influence of IBM — aka “Blue Babylon” — and the IT world has been languishing in sin ever since.

The second class of reason for avoiding SQL is because you don’t like the idea of a separate schema-aware DML at all. Possible reasons for this orientation include:

• You just like to program, and want to manipulate stored data the same way you do anything else. Thus, you are bothered by an “impedance mismatch” between SQL and your favorite programming languages.  This is real. It also has been overcome by many, many enterprises around the world.
• You believe that more procedural alternatives are a better fit for cloud computing and extreme scale-out on failure-prone commodity hardware. Facebook made that case to me. However, I have trouble thinking of very many enterprise scenarios where it applies, especially when one considers electricity costs and the like.
• Your schemas change more quickly than your data architects can reasonably be expected to keep up with. Facebook made that case to me too. Enterprise examples might include marketing campaigns and M&A. I’ve long thought this to be a legitimate, looming concern. But I don’t know that stripped-down DBMS are the way to address it.
• You believe that SQL has severe processing overhead. In most enterprise use cases, that would just be bogus.
• You lack familiarity with SQL.

That last point is not a joke. One of the weirder database architectures I know of is the one underlying Guild Wars. Its developer — a brilliantly impressive guy — told me flat-out that he learned in college how to build a DBMS, but he didn’t learn how to develop for a conventional one. This was instrumental in his decision to build an unconventional data management architecture that uses SQL Server as little more than a smart file manager.

The questions of SQL performance and — often-unspecified — “overhead” are interesting to view through the lens of the H-Store/VoltDB project. Mike Stonebraker et al.:

• Are building a scale-out-oriented OLTP DBMS that is meant to run in RAM, preserving data through replication to other servers’ RAM more than through output to disk.
• Believe that 95% of what a typical SQL DBMS does to manage OLTP is wasteful overhead
• Originally planned to not use SQL, but wound up going with SQL because alternatives were insufficiently performant.

Mike himself, of course, has been all over the spectrum on SQL-like languages. First he favored QUEL vigorously over SQL for mainstream relational DBMS. Then he led the charge to extend SQL in PostgreSQL, Illustra, et al. Then he actually staked out a contrarian position in the area of complex event/stream processing by favoring a SQL-like language in an area where other alternatives were better established — but that was at what turned into StreamBase, which now emphasizes visual programming over any kind of coding language.

I need to write much more about schema flexibility, but tonight — which will be my third straight of <<8 hours sleep — is not the time for that.

• Aster announced today that it’s providing .NET support for SQL/MapReduce. Perhaps not coincidentally, Aster’s biggest customer is MySpace, which is apparently a big Microsoft shop.  (And MySpace parent Fox Interactive Media is a SQL/MapReduce fan, albeit running on Greenplum.)
• Aster generally puts more emphasis on MapReduce than SQL/MapReduce rival Greenplum.  That’s a non-trivial comparison, because Greenplum is making progress in SQL/MapReduce itself.
• When talking with Aster folks, I can’t get them to shut up hear a lot about SQL/MapReduce.

I was a big fan of SQL/MapReduce when it was first announced last August. Notwithstanding persuasive examples favoring pure DBMS or pure MapReduce over DBMS/MapReduce integration, I continue to think the SQL/MapReduce idea has great potential.  But I do wish more successful production examples would become visible …

DELIVERABLES: Our research analytics team uses Greenplum Database to conduct tens of thousands of real-time [statistical] tests against millions of users every day, analyzing each visitor’s reaction to ads against more than 2,000 variables. This analysis is turned into reports for the BI, database administrators, systems engineering and product teams.

VENDOR SUPPORT: We’ve had nothing but positive interactions with the Greenplum team, from first sales call through to implementation and support. The technical team has been able to answer all of our questions, and we have the utmost respect for the engineering minds behind Greenplum Database.

WEAKNESSES: The ability to prioritize and balance queries at run time is a weakness. This would allow more effective sharing of resources between power users and reporting tasks. Second, the query optimizer is still immature.

Also, when Greenplum briefed on its Enterprise Data Cloud vision, it gave me a couple of details about what Fox was doing in that area.  For starters, Brian Dolan has a team of analysts, single-digit in number, each with his/her private Greenplum sandbox, just like the vision says they should have.  Second, it so happens that Fox has some customer/partner-facing data marts fed by the Greenplum database, but those are run on PostgreSQL rather than Greenplum.

Also, Eric cites the Greenplum Fox Interactive Media database as being only 200 TB in size.  Surely there is some confusion somewhere, since Greenplum described it as being 400 TB back in August.

Now Greenplum has come out swinging, claiming “consistent” load speeds of 4 terabytes/hour at its Fox Interactive Media account, and armed with a customer quote saying just that.  Note however that load speeds tend to be proportional to the number of disks, and there are a LOT of disks at that installation.

One way to think about load speeds is — how long would it take to load the entire database? It seems as if the Fox database could be loaded, perhaps not in one week, but certainly in less than two. Flipping that around, the Fox site only has enough capacity to hold less than 2 weeks of detailed data. (This is not uncommon in network event kinds of databases.) And a corollary of that is — worldwide storage sales are still constrained by cost, not by absolute limits on the amounts of data enterprises would like to store.

The big aha moment occured for me during our panel discussion, which included Luke Lonergan from Greenplum, Roger Magoulas from O’Reilly, and Brian Dolan from Fox Interactive Media (which runs MySpace among other web properties).

Roger talked about using MapReduce to extract structured entities from text for doing tech trend analyses from billions of rows of online job postings.  Brian (who is a mathematician by training) was talking about implementing conjugate gradiant and Support Vector Machines in parallel SQL to support “hypertargeting” for advertisers.  I mentioned how Jonathan Goldman at LinkedIn was using SQL and MapReduce to do graph algorithms for social network analysis.

Incidentally: While it’s been some months since I asked, my sense is that the O’Reilly text extraction is home-grown, and primitive compared to what one could do via commercial products. That said, if the specific application is examining job postings, I’m not sure how much value more sophisticated products would add. After all, tech job listings are generally written in a style explicitly designed to ensure that most or all of their meaning is conveyed simply by a bag of keywords. And by the way, this effort has been underway for quite some time.

Related link

• Greenplum has a page on the O’Reilly relationship.  However, the part that isn’t behind a registration barrier is trivial — and I wouldn’t know one way or the other about the registration-required part.

Still, as Greenplum’s gushing customer video with Fox Interactive Media* illustrates, the Fox/Greenplum database is impressive on its own merits. Indeed, the Fox/Greenplum database seems to be larger than the one MySpace runs on Aster, even though it seems to be used less intensively. Perhaps data is thrown out from the Aster database but never from the Greenplum one, or perhaps the Greenplum database just ingests more data in the first place.

*The Greenplum video features completely different customer personnel than Aster’s. I’m sure of this because all the Fox people on the Greenplum video are male, while all the MySpace techies on Aster’s video are female.

In particular, Fox Interactive uses a database architectural idea I’m hearing ever more about. Within the data warehouse, it creates data mart “sandboxes,” that analysts can use however they wish. They can even create and delete tables without screwing up the underlying data. eBay famously uses a similar approach in its Teradata installation, and I believe Dell’s DATAllegro data warehouse is set up that way as well.

Related links

• MySpace’s multi-hundred terabyte database running on Aster Data
• Greenplum’s biggest customers

]]> http://www.dbms2.com/2009/03/05/fox-interactive-medias-multi-hundred-terabyte-database-running-on-greenplum/feed/ 3