• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

]]> http://www.dbms2.com/2009/09/30/facts-and-rumors/feed/ 2 http://www.dbms2.com/2009/09/11/xkoto-gridscale-highlights/ http://www.dbms2.com/2009/09/11/xkoto-gridscale-highlights/#comments Fri, 11 Sep 2009 18:36:03 +0000 Curt Monash http://www.dbms2.com/?p=881 I talked yesterday with cofounders Albert Lee and Ariff Kassam of Xkoto. Highlights included:

• Xkoto sells Gridscale, a clustering server for DB2 and, more recently, MS SQL Server.
• Xkoto Gridscale runs on a separate box, between the application and the database servers. This box is typically smaller and cheaper than the database server boxes.
• Xkoto most typically sells Gridscale into environments where there already are three database servers — one to do work, one for hot standby, and one for remote disaster recovery.
• In such environments, Gridscale’s big benefit is that you can distribute the query workload among all three servers. Xkoto believes this big performance increase is the reason customers don’t get much past 3 database servers under Xkoto (they didn’t seem quite sure as to whether the all-time record was 4 or 5). Note that even if a remote server is a little too far away for OLTP query response, it can work fine for reporting.
• Of course, if you don’t already have high/”continuous” availability and/or disaster recovery, then Xkoto would say those are core benefits of Gridscale as well.
• Gridscale sends transactions (or just SQL statements?) to all servers in the cluster. Once any of them responds affirmatively, that update is reflected in queries. Gridscale maintains a small query log to make sure it gets the other database copies in sync. It also tries to make sure that queries always go to the most current copy of the database. (I didn’t ask what happens if Server A executes Transaction T but not U, while Server B executes Transaction U and not T — but that does seem like something of an edge case.).
• Xkoto spun out of Halcyon Monitoring in 2006, starting with DB2 support. Microsoft SQL Server support was introduced in 2008.
• Xkoto likes its partnerships with IBM and Microsoft. For example, IBM provides Level 1 and 2 support for Gridscale itself. Due in large part to this partnership strategy, Xkoto says it has no plans to support DBMS beyond DB2 and SQL Server.
• Instead, Xkoto is pursuing partnerships with large application vendors and so on. (The figure “about 10″ was mentioned.) I gather the idea is to make sure that neither the application support folks nor the app itself freak out from the fact that the app isn’t exactly talking to the DBMS any more.
• Xkoto has done lab tests suggesting Gridscale offers near-linear scalability (in terms of SQL Server database throughput) on a query-only workload up to 10 servers.
• I gather that Xkoto and IBM have demos suggesting it’s a fine idea to have your disaster recovery server be in the Amazon cloud, but they haven’t yet made any sales based on that — er, based on that premise.
• Gridscale pricing is measured in the same metrics as DB2 or SQL Server pricing, and in each case is around 1/3 what database pricing would be on the same box (I’m guessing that’s for enterprise additions without add-ons, but I didn’t probe). Specifically, Gridscale charges $12K per 100 PVUs for the DB2 edition, and $12K per socket for running with Microsoft SQL Server.
• Gridscale typically runs on smaller boxes than the databases it talks to.
• Xkoto has about 35 revenue-recognized customers. Most are on DB2, the first environment Gridscale supported.
• Average Gridscale selling prices are $180K on DB2, and $40-50K in the early going for SQL Server.
• Xkoto has about 40 full-time employees, with engineering in Toronto and business operations in Waltham.

Microsoft Madison availability remains scheduled for H1 2010. Nothing new there. Tangible progress includes a few customer commitments of various sorts, including one outright planned purchase (due to some internal customer considerations around using up a budget). At the moment various Microsoft Madison technology “previews” are going on, which seem to amount to proofs-of-concept, that:

• Start with actual customer data (some from Microsoft, some from outside)
• Generate larger synthesized data sets based on those (database size seems to be 10-100 TB)
• Run in Microsoft data centers or “technology centers”, rather than on customer premises.

The basic Microsoft Madison product distribution strategy seems to be:

• Microsoft specifies configurations with different brands of hardware.
• The respective hardware vendors sell and deliver those configurations, with Madison software pre-installed.
• Actual software licenses are invoiced however makes sense, which in most cases may well be directly by Microsoft as part of a wider enterprise relationship.

Most of the usual-suspect big name hardware and storage vendors seem to be involved.

Microsoft Madison is focused on “high-end” data warehousing, with Stuart candidly saying that everybody at Microsoft seems to have a different definition of what “high-end” means. In practice, this “high end” probably will be whatever conventional SQL Server doesn’t do a good job on — e.g., >5 terabytes, or even smaller in table-scan-oriented workloads.

Microsoft Madison seems further focused on being the “hub” to SQL Server data marts, with Stuart citing a survey saying Microsoft SQL Server has 44% market share in data marts when counting by unit. When I pressed for technical strategy as to how the data would be moved and synchronized between the hub and spokes, details were vague. Obviously, Microsoft still has considerable work to do in this regard, whether in articulating strategy or in actual product development. The same goes double for ease-of-data-mart creation ala Greenplum’s Enterprise Data Cloud messaging.

While Madison is a future, Stuart says Microsoft SQL Server Fast Track is a “huge hit.” He knows of 20 sales, and estimates the total number is in the 100s. The disparity is explained by the fact that that Fast Track comprises a set of recommended hardware configurations, rather than an identifiable Microsoft software invoice line item. The distribution model for Microsoft SQL Server Fast Track seems to be similar to that planned for Madison, with hardware partners such as HP making many of the sales.

Sales of all these Microsoft SQL Server data warehousing products seem to be focused on the Microsoft SQL Server installed base. No surprise there either.

One of my favorite pages on the Monash Research website is the list of many current and a few notable past customers. (Another favorite page is the one for testimonials.) For a variety of reasons, I won’t undertake to be more precise about my current customer list than that. But I don’t think it would hurt anything to list the analytic/data warehouse DBMS/appliance specialists in the group. They are:

• Aster Data
• Greenplum
• Infobright
• Kickfire
• Kognitio
• Microsoft
• Netezza (my biggest client this year, probably, because of all the Enzee Universe appearances)
• Sybase
• Teradata
• Vertica
• Attivio, which may or may not be construed as being in the analytic DBMS business
• Clearpace, ditto

All of those are Monash Advantage members.

If you care about all this, you may also be interested in the rest of my standards and disclosures.

• Data marts aren’t just for performance (or price/performance). They also exist to give individual analysts or small teams control of their analytic destiny.
• Thus, it would be really cool if business users could have their own analytic “sandboxes” — virtual or physical analytic databases that they can manipulate without breaking anything else.
• In any case, business users want to analyze data when they want to analyze it. It is often unwise to ask business users to postpone analysis until after an enterprise data model can be extended to fully incorporate the new data they want to look at.
• Whether or not you agree with that, it’s an empirical fact that enterprises have many legacy data marts (or even, especially due to M&A, multiple legacy data warehouses). Similarly, it’s an empirical fact that many business users have the clout to order up new data marts as well.
• Consolidating data marts onto one common technological platform has important benefits.

In essence, Greenplum is pitching the story:

• Thesis: Enterprise Data Warehouses (EDWs)
• Antithesis: Data Warehouse Appliances
• Synthesis: Greenplum’s Enterprise Data Cloud vision

When put that starkly, it’s overstated, not least because

Specialized Analytic DBMS != Data Warehouse Appliance

But basically it makes sense, for two main reasons:

• Analysis is performed on all sorts of novel data, from sources far beyond an enterprise’s core transactions. This data neither has to fit nor particularly benefits from being tightly fitted into the core enterprise data model. Requiring it to do so is just an unnecessary and painful bureaucratic delay.
• On the other hand, consolidation can be a good idea even when systems don’t particularly interoperate. Data marts, which commonly do in part interoperate with central data stores, have all the more reason to be consolidated onto a central technology platform/stack.

Of course, the EDC vision isn’t quite as new or differentiated as Greenplum ideally would wish one to believe.

• To a first approximation, EDC sounds a lot like what eBay has already built on Teradata equipment.
• Greenplum’s EDC vision also sounds a lot like what Stuart Frost was talking about at DATAllegro, what Dell was planning to build on DATAllegro equipment, and what Stuart continues to talk about now that he’s been acquired into Microsoft.
• Something like EDC can also be presumed to be implicit in the strategies of the other one-size-fits-all vendors — i.e., Oracle and IBM.
• Greenplum has only implemented a little more of the EDC vision so far than have other firms, unless you give it credit for being cheap/fast/MPP/running on commodity hardware, but deny that credit to Teradata (specialized hardware, and not cheap in its most popular configurations), Oracle (ditto for Exadata), IBM (also not cheap), or Microsoft/DATAllegro (not released yet).
• Specifically: In Greenplum Release 3.3, which is being announced today, Greenplum is introducing the (enhanced?) ability for data marts to be spun out as a background operation, while the database otherwise remains functional. As of 3.3, spinning out a data mart is a command-line operation. But in Release 3.4, Greenplum plans to offer a web-based interface for same, at which point the “self-service data mart creation” discussion will become operative. Otherwise, EDC is a roadmap/vision/statement-of-direction much more than it is a fully-baked technical project.

One particular source of potential confusion is Greenplum’s emphasis on the buzzphrase self-service (data mart). This seems to be a conflation of two related concepts:

• End users should be able to create new data marts themselves. Strictly speaking, I view this ability as useless at most enterprises, and important at very few, because of logistical issues. (Who gives the permissions? Who decides which hardware is used?) That said, useless “end user” tools often wind up being important productivity aids for IT professionals, and this kind of “self-service” would surely be another example.  Edit: Hmm. Doug Henschen inspired me to think that over again, and I’m beginning to soften. Suppose users could order up the data mart they want, perhaps test it at a very low processing priority (if they choose), and then send the completed request to IT for approval and provisioning. That would have some value.
• End users should be able to manage data marts themselves, once created. That’s a great idea, full of agility and don’t-make-IT-a-roadblock goodness. Data miners and similar analytic professionals commonly have the technical ability to manage a simple database, and should be allowed to do so if it’s ensured that they don’t break anything for anybody else.

One thing that’s needed for this technology to come to full fruition is sophisticated data movement and synchronization. Ideally, some tables in a data mart could be virtual — views against a central database. But others would be physically recopied from the center, with all the ETL/ELT/ETLT/replication issues that entails. Meanwhile, it’s not obvious that the ideal architecture is a simpleminded hub-spoke — perhaps one should be able to spin data marts out of other marts, perhaps at least somewhat reducing the proliferation of tables and the recopying of data. And it should be easy for administrators to change deployment strategies, e.g. by starting a table out as a view and changing over to making it a physical copy as usage profiles change.

Oliver Ratzesberger of eBay also argues that workload management — not a current Greenplum strength — can be crucial. For example, if the CEO wants the CFO to get her an answer TODAY, the fastest approach may be to create an entirely virtual data mart, with very favorable SLAs (Service Level Agreements). More generally, if you’re setting up dozens of marts that contain views of the central database, sophisticated SLA management can be essential. There’s a big virtualization opportunity here — but virtualization requires a lot of system management infrastructure.

Related links

• My recent post on reinventing business intelligence
• Greenplum adviser Joe Hellerstein’s pitch for agile data warehousing
• Charlie Bachman’s “private database” idea, which never went anywhere (pp. 138-139)
• Greenplum’s EDC and Release 3.3 press releases
• An interview with some of Greenplum co-founder Scott Yara’s own words

Natural language and classic science fiction

Actually, there’s a pretty well-known example of BI near-perfection — the Star Trek computers, usually voiced by the late Majel Barrett Roddenberry. They didn’t have a big role in the recent movie, which was so fast-paced nobody had time to analyze very much, but were a big part of the Star Trek universe overall. Star Trek’s computers integrated analytics, operations, and authentication, all with a great natural language/voice interface and visual displays. That example is at the heart of a 1998 article on natural language recognition I just re-posted.

As for reality: For decades, dating back at least to Artificial Intelligence Corporation’s Intellect, there have been offerings that provided “natural language” command, control, and query against otherwise fairly ordinary analytic tools. Such efforts have generally fizzled, for reasons outlined at the link above. Wolfram Alpha is the latest try; fortunately for its prospects, natural language is really only a small part of the Wolfram Alpha story.

A second theme has more recently emerged — using text indexing to get at data more flexibly than a relational schema would normally allow, either by searching on data values themselves (stressed by Attivio) or more by searching on the definitions of pre-built reports (the Google OneBox story). SAP’s Explorer is the latest such view, but I find Doug Henschen’s skepticism about SAP Explorer more persuasive than Cindi Howson’s cautiously favorable view. Partly that’s because I know SAP (and Business Objects); partly it’s because of difficulties such as those I already noted.

Flexibility and data exploration

It’s a truism that each generation of dashboard-like technology fails because it’s too inflexible. Users are shown the information that will provide them with the most insight. They appreciate it at first. But eventually it’s old hat, and when they want to do something new, the baked-in data model doesn’t support it.

The latest attempts to overcome this problem lie in two overlapping trends — cool data exploration/visualization tools, and in-memory analytics. Tableau and Spotfire are known more for the former; hot BI vendor QlikTech is known for both. And many vendors — established or otherwise — are going to in-memory OLAP.

Collaboration and communication

The reason I’m finally buckling down and posting on this subject is the announcement of Google Wave, which I think foreshadows a revolution in communication and collaboration technology. Google Wave augurs two primary advances. First, it shows how to make email, instant messaging, microblogging, and so on much more useful. Second, Google Wave could evolve in a way that — finally — makes it truly practical for end-users to set up ad-hoc mini-portals that combine arbitrary URL-possessing resources, exposed to arbitrary workgroups of people.

If and when both of those promises are fulfilled, it will become vastly easier for people to reason together about analytic questions. That may take a little while, as Google Wave obviously wasn’t designed with business intelligence in mind. But whether from Google or from a frightened Microsoft redoubling its SharePoint efforts, there’s hope that we’ll see a leap forward in general collaboration technology. And since BI vendors are doing a generally decent job of exposing queries, charts and so on as portlets, it seems likely that business intelligence will benefit from the collaboration arms race.

That’s important. The first time I heard that reporting was as important for communication as for analytics was from Pilot Software a quarter-century or so ago, and it’s just as true now as it was then. In its first incarnations it probably will be a little too dumb for my tastes, focusing more on mindless reporting and same-old KPIs than on deeper analysis. Still, it’s a move in a good direction.

Other directions

As I said at the beginning, I find it too daunting to try to cover all facets of this subject in one post. So I’ll leave out, at a minimum:

• Data warehousing performance and TCO, which I of course write about extensively
• Complex event/stream processing, which I’ve written quite a bit about too
• Data mining and predictive analytics
• Operational BI

plus some hobby horses you probably don’t want to hear about anyway until I work out a better way of articulating my opinions.

But by all means please comment on what I’ve left out just as vigorously as on what I’ve included. This post is just the first of many to come.

]]> http://www.dbms2.com/2009/05/30/reinventing-business-intelligence/feed/ 13 http://www.dbms2.com/2009/05/21/notes-on-cep-application-development/ http://www.dbms2.com/2009/05/21/notes-on-cep-application-development/#comments Thu, 21 May 2009 08:21:49 +0000 Curt Monash http://www.dbms2.com/?p=788 While performance may not be all that great a source of CEP competitive differentiation, event processing vendors find plenty of other bases for technological competition, including application development, analytics, packaged applications, and data integration. In particular:

• Most independent CEP vendors have some kind of application story in the capital markets vertical, such as packaged applications, ISV partners with packaged applications, application frameworks, and so on.
• CEP vendors offer lots of connectors to specific financial industry price/quote/trade feeds, as well as the usual other kinds of database connectivity (SQL, XML, etc.)
• Aleri/Coral8 (separately and now together) like to call attention to their business intelligence/analytics offerings. Analytics is front-and-center on Truviso’s web site too, not that Truviso does much to call attention to itself, period. (Roman Bukary once said he’d outline Truviso’s new strategy to me in 6-8 weeks or so … it’s now 14 months and counting.)

So far as I can tell, the areas of applications and analytics are fairly uncontroversial. Different CEP vendors have implemented different kinds of things, no doubt focusing on those they thought they would find easiest to build and then sell. But these seem to be choices in business execution, not in core technical philosophy.

In CEP application development, however, real philosophical differences do seem to arise. There are at least three different CEP application development paradigms:

• DML (Data Manipulation Language) extensions to handle time windows, etc., which are then embodied into a conventional application development stack. Microsoft exemplifies an extreme form of this strategy, but most vendors offer it at least as an option.
• Visual event-oriented programming language. This is StreamBase’s claim to fame; StreamBase says that 95% of its customers do 100% of their development in its Eclipse-based visual tool, which truly creates executable programs without any kind of code generation step. (Other “this isn’t just a toy” buzzwords StreamBase offered were “modularity, ” “parametrizability,” and, best of all, “visual debugger.”)
• Rules engine. When Progress Apama first told me how its preferred tool (I think a precursor of what is now Apama Event Modeler) worked, it sounded a lot like a RETE-based expert system shell. The Apama folks assured me that this really wasn’t RETE, and I see that there’s definitely more than just a rules language in Apama MonitorScript. Still, the paradigm seems to basically be that you write a bunch of rules, which are then executed by a suitable engine.

Even more fundamental, however, is the question of event-driven programming. That’s a term which first was popular in the 1990s, signifying programs that — rather than being wholly procedural — listened for and responded to events, often in user interfaces. More generally, event-driven programming is inherent in almost any kind of loosely-coupled architecture, be it client-server, service-oriented, or whatever. But to hear some CEP proponents tell it, all that isn’t event-driven enough.

In particular, StreamBase recommended to me Gregor Hohpe’s paper Programming Without a Call Stack – Event-driven Architectures, which argues that if there’s any concept of procedure or method invocation at all, the whole thing is too “command-and-control” (Boo!!) and insufficiently event-driven to be well-suited for CEP. Rather, everything should be done on a fine-grained publish-subscribe basis. It’s an interesting argument. Usually, the problem with extremist programming paradigms is that the thing you write has to interface with the rest of the world, and by the time it accommodates itself to them, the paradigm is violated anyway. But if the essence of the paradigm is loose coupling to begin with, maybe that pitfall can for once be avoided.

]]> http://www.dbms2.com/2009/05/21/notes-on-cep-application-development/feed/ 5 http://www.dbms2.com/2009/05/13/microsoft-announced-cep-this-week-too/ http://www.dbms2.com/2009/05/13/microsoft-announced-cep-this-week-too/#comments Thu, 14 May 2009 03:58:53 +0000 Curt Monash http://www.dbms2.com/?p=781 Microsoft still hasn’t worked out all the kinks regarding when and how intensely to brief me. So most of what I know about their announcement earlier this week of a CEP/stream processing product* is what I garnered on a consulting call in March. That said, I sent Microsoft my notes from that call, they responded quickly and clearly to my question as to what remained under NDA, and for good measure they included a couple of clarifying comments that I’ll copy below.

*”in the SQL Server 2008 R2 timeframe,” about which Microsoft wrote “the first Community Technology Preview (CTP) of SQL Server 2008 R2 will be available for download in the second half of 2009 and the release is on track to ship in the first half of calendar year 2010. “

Perhaps it is more than coincidence that IBM rushed out its own announcement of an immature CEP technology — due to be more mature in a 2010 release — immediately after Microsoft revealed its plans. Anyhow, taken together, these announcements support my theory that the small independent CEP/stream processing vendors are more or less ceding broad parts of the potential stream processing market.

The main use cases Microsoft talks about for CEP are in the area of sensor data. For example, Microsoft has prospects or customers who have many pieces of manufacturing or resource extraction equipment each. Each may spawn only a few messages per second, but overall there can be 1000s of messages/second, or indeed terabytes of data/day. (The orders of magnitude don’t quite match up there, but we were speaking pretty vaguely anyway.)

Microsoft called out four reasons to me why CEP might be needed in addition to ordinary database processing. Two are the standard reasons for data reduction:

1. Without CEP, you can’t bang the data into the database fast enough.

2. You don’t want to keep most of the data past a short time window anyway.

The other two are also fairly standard reasons for using CEP:

3. Standard SQL isn’t all that great for time series anyway.

4. CEP use cases often call for incremental processing and/or parameterization of queries, something CEP engines are commonly better designed for than are DBMS.

However, Microsoft seems to be taking a somewhat different approach to time-based SQL extensions than some other vendors. To quote email Microsoft sent today:

Microsoft Research (MSR) introduced the temporal extensions to relational algebra based upon a notion of application time that is independent of system time. It matters when the event originated instead of when they arrived at the processing system. Further it treats each event as being associated with an interval of time as opposed to a point in time. This helps in modeling certain real life phenomenon naturally. [StreamBase et al.] also reason about multiple streams. Both the approaches are extensions to relational algebra. The MSR approach took the algebra as the starting point while StreamBase took an existing language over the algebra – SQL as the starting point. The MSR approach consequently avoids having to rework other elements of the SQL surface. The primary language extensions through which this algebra will be exposed initially is LINQ.

Microsoft’s CEP capability is “fully integrated” with Visual Studio. There will be lots of adapters, both for inputs and outputs, with perhaps the most interesting non-obvious one being Excel charts. I definitely like the idea of CEP engines doing a good job of integrating both with dashboards/BI and with operational apps, because if you get value from one of those integrations, you’re apt to quickly want the other as well.

Microsoft told me that its CEP is written in a combination of “managed” and “native” code, where “managed” code, in Microsoft lingo, is more an issue of memory management than code. Noticing that I was confused on this point, Microsoft elucidated by email:

The implementation is built around getting the technologies available in the CLR and native code together to build the best implementation possible. We use the ability to do JIT code generation to efficiently evaluate expressions and back it up with very effective native memory management techniques.

]]> http://www.dbms2.com/2009/05/13/microsoft-announced-cep-this-week-too/feed/ 7