This post is a moderately edited form of an actual interview. Two other posts cover analytic trends to watch (planned) and analytic vendor execution challenges to watch (already up).

Question: What do you think will happen next year with the Tableaus of the world?

Answer:

• I think adoption of flexible-visualization business intelligence tools will continue to be rapid.
• I think enterprise-friendly features will be increasingly important as a basis of competition.

Question: What do you mean by “enterprise-friendly”?

Answer: An example would be QlikTech no longer forcing you to use their native ETL, but rather working with Informatica and soon other third-party products. Also important can be:

• Database size.
• Concurrency.
• A full-featured development cycle for analytic applications.

Question: What does HP have to do to be relevant in analytics/data warehousing?

Answer: Avoid stupidity. HP Vertica is already relevant.

Question: OK. But what can HP do to build on Vertica?

Answer: HP — which botched Exadata 1 hardware — could do a good job with SAP HANA or other kinds of appliance products.

However:

• I don’t think trying to force Vertica beyond its natural growth — the way EMC is with Greenplum — is necessarily a good idea. Natural growth in Vertica’s case is plenty fast anyway.
• Obviously, making good Vertica hardware would be nice. But being hardware-independent is crucial to Vertica, not least because of cloud deployment, an option many buyers want to at least have in their hip pockets.

Question: You expressed some skepticism toward mobile BI/use cases. Why so?

Answer: The form factor hurts functionality a lot, so it’s only worthwhile in cases where timeliness is key.

And without more refined alert-setting functionality, it’s hard to think of that many cases.

Note: My views on mobile BI haven’t changed much since July, 2010.

Question: What about the idea of an enterprise being able to pay-per-drink to run jobs on an analytic cluster. Do you expect that concept to have any legs in 2012?

Answer: While other kinds of SaaS (Software as a Service) BI might make sense, remote computing BI that focuses on hardware cost sharing is problematic. Moving data in and out of the cluster is a big part of the overall cost, at least if you plan to process it only occasionally once it gets there. I haven’t seen a plan yet that gets around that point.

• 100s of gigabytes of data at first, growing to >1 terabyte over time.
• High peak loads.
• Public cloud portability (but they have private data centers they can use today).
• Simple database design — not a lot of tables, not a lot of columns, not a lot of joins, and everything can be distributed on the same customer_ID key.
• Stream the data to a data warehouse, that will grow to a few terabytes. (Keeping only one year of OLTP data online actually makes sense in this application, but of course everything should go into the DW.)

So I’m leaning to saying:  

• They should go with a scalable, MySQL-based solution.
  • Lots of third-party software works with MySQL, in case that’s helpful.
  • Yes, any one vendor is small and not yet firmly established, but there are numerous vendors around with interesting MySQL scaling stories.
  • In a vendor emergency, just going with Oracle’s MySQL stuff would probably work …
  • … especially because there are these lovely things in the world called solid-state drives.
  • There’s also good escapability if one wants to move away from MySQL, because everybody knows how to handle MySQL data.
• The first product to look at is dbShards, because it meets all the topology needs:
  • Local scale-out (transparent sharding).
  • Local high availability.
  • Remote disaster recovery (details of that are underway).
• The first analytic DBMS to look at is Infobright.
  • Yes, I know Infobright is focused more on machine-generated data these days, but this client’s analytic needs are so straightforward Infobright should pass with flying colors.
  • The MySQL-to-MySQL aspect should make ETL dead simple.
  • Again, there’s escapability.

Mainly, this is all fine. But I’m getting pushback on the solid-state aspect, for fear that it will compromise public cloud portability.

Am I missing something here? As far as I’m concerned, if you’re planning an OLTP system with a many-year lifespan today, of course you should assume solid-state storage. Maybe you scale out just as far as you would with disk, striping indexes or entire databases across the RAM of multiple servers. It that case, having solid-state backing reduces the risk of bottlenecks. Maybe you don’t scale out as far as you would with disk. In that case, solid-state backing saves you money.

As for public-cloud support for solid-state storage, that’s coming fast, right? (Actually, I have data points in support of that theory, but they’re a bit tenuous.) A large fraction of web businesses with private data centers seem to be using solid-state storage — from Facebook on down — or so the NoSQL/NewSQL/short-request DBMS guys tell me. Surely a number of public cloud vendors are close behind.

• Log-stream machine-generated data, when what you’re looking at — at least initially — is the entire output of verbose logging systems.
• Remote machine-generated data.

Here’s what I’m thinking of for the second category. I rather frequently hear of cases in which data is generated by large numbers of remote machines, which occasionally send messages home. For example: 

• I heard yesterday about a case with 10s of millions of machines, phoning home every 5 minutes, and another case with 10s of 1000s calling in every 5 seconds, both of them sending data initially to MySQL. (Application details weren’t given.)
• I heard not long ago about a set-top box case that the vendor hoped would also grow to 10s of millions of machines, which I guessed might send a small number of messages per hour each.
• I also heard recently about a remote security monitoring case whose data was destined for (probably) Netezza, although in that case I’m not sure about the “occasionally” aspect of the communication.
• The last time I visited Splunk, I got the sense that energy-sensor use cases (especially in the electric grid) had finally emerged. I believe these sensors are periodic message senders — they wake up, take their temperature (figuratively or literally as the case may be), send a message, snooze, repeat.
• I would guess that the energy use cases Infobright talked about in 2009 were of a similar kind.
• An April, 2010 comment on the post linked above talks about many kinds of sensor data.
• Back in 2007, Coral8 talked of a truck phone-home use case (on-board GPS data and also, e.g., refrigeration level, sending messages once a minute or so). Truviso seemed to have one similar deal before one of its frequent changes in strategic direction, and not coincidentally cites UPS as an investor.
• In principle, there are a lot of RFID use cases out there, even if I rarely seem to hear of any. (That would be a shorter “phone call” home than most of the other examples, of course, but might be otherwise technically similar.)

Many technologies can be used to collect and manage remote machine-generated data, but a few common points are worth nothing.

• If a device takes the trouble to send a message across a wide-area network, that message may be somewhat more valuable than the average piece of log-vomit. Perhaps such information doesn’t need to be stored in the cheapest possible way.
• Similarly, a message that is sent occasionally over time, or upon a specified event, may be more structured than a random log entry. Perhaps such data is suitable for sending straight to a relational database.
• If there’s no central place the data originates, there may also be no favored place for the data to end up. It may make great sense to collect and analyze remote machine-generated data in the cloud. (Exceptions may of course arise if you want to use the data in connection with other information, and you hence want to bring it to that other information’s location.)
• In a number of use cases, the whole point is to identify anomalies, and respond to them rapidly. I.e., remote machine-generated data use cases commonly raise challenges in low-latency integration of short-request and analytic processing.

Bit bucket

• Kinds of data likely to be included: Logs, other technical/external
• Likely use styles: Staging/ETL, investigative
• Canonical example: Log files in a Hadoop cluster
• Stresses: TCO, scale-out, transform/big-query performance, ETL functionality

With the explosion of machine-generated data has come the need for a place to put it all, sometimes called the big bit bucket. This is like the investigative data mart for big databases, but more poly-structured. In some cases it is focused on data staging and transformation; but it can also be used for analysis in place.

The list of candidate technologies to run your bit bucket starts with Hadoop and Splunk.

Archival data store

• Kinds of data likely to be included: Operational, CDR (call detail record), security log
• Likely use styles: Archival, reporting (for compliance), possibly also investigative
• Examples: Any long-term detailed historical store
• Stresses: TCO, compression, scale-out, performance (if multi-use)

Analytic DBMS vendors have been insulting each other with the claim “that’s just an archival data store,” dating back at least to the first time Greenplum was deployed on an underpowered Sun Thumper system. Perhaps only Rainstor truly embraces the archival positioning, and I’ve become pretty dubious about their technical claims and their company alike.

Still, there’s a legitimate need for data stores — especially relational analytic DBMS that:

• Store data cheaply, with high rates of compression.
• Have decent performance if you do want to query the data.
• May have archiving/compliance-specific features as well.

Along with Rainstor, SAND and SenSage have at least partially targeted that use case. In addition, appliance vendors such as Teradata and Netezza try to have an archive-oriented product version in their lineups.

Outsourced data mart

• Kinds of data likely to be included: All
• Likely use styles: Traditional BI, investigative analytics, staging/ETL
• Examples: Advertising tracking, SaaS CRM
• Stresses: Performance, TCO, reliability, concurrency

Much of what happens in analytic database management can also be outsourced. Some applications that run via SaaS (Software as a Service) are analytic. I’ve had three different clients whose main business is picking marketing targets in various vertical segments; others who wanted to add analytics to what were historically OLTP applications; and others yet who just offered online business intelligence. Also, if your fundamental business is gathering data and reselling it to a variety of user organizations, that’s an analytic data management challenge. The possibilities expand from there.

Data outsourcers are in the IT business, and so their IT development is — hopefully! — more serious and less politically encumbered than at many conventional enterprises. Thus, legacy systems and master data management issues are commonly less prevalent, or at least more aggressively disposed of. The same, up to a point, goes for vendor politics.*  Multitenancy is commonly an issue, as is running in the cloud.

*Even so, there’s often That Guy who doesn’t want to migrate away from Oracle, no matter what.

Vertica gets the nod in a number of these cases; it’s cloud-friendly, and often the problem is naturally columnar. Other columnar products can be good choices too, with added brownie points for Infobright if the shop is MySQL-oriented anyway. Running Netezza or other appliances makes sense mainly if you’re pretty sure you want to keep operating your own data centers, but some data outsourcers are just fine with that assumption.

Operational analytic(s) server

• Kinds of data likely to be included: Customer-centric, log, financial trade
• Likely use styles: Advanced operational analytics
• Examples:
  • Lower latency: Web or call-center personalization, anti-fraud
  • Higher latency: Customer profiling, Basel 3 risk analysis
• Stresses: Performance, reliability, analytic functionality, perhaps concurrency

Even with eight different choices, I need a “catch-all” category; this is it.

Suppose you want to do reasonably sophisticated analytics, then use the results in operations. This is the classical challenge in integrating short-request and analytic processing. There are multiple ways to tackle it, embodying different trade-offs in cost, convenience, or analytic accuracy. If the platform on which you want to run your investigative analytics also has the reliability and concurrency appropriate for mission-critical operations, you’re set. Otherwise, you may want to pipe derived data into a more “industrial-strength” DBMS, ideally the one that runs your operational apps anyway

Another option is to integrate a limited amount of analytics immediately into your short-request processing system. For example, as bad as they are at the kinds of queries that require joins, NoSQL systems are often fast at simple aggregations. As MapReduce/NoSQL integrations mature, that option may not require pumping the data anywhere else for deeper analytics; even if it does, at least you’re starting out with the data in a convenient bit bucket.

Streaming/CEP-centric architectures could come into play as well. And it goes on from there. The possibilities in this last category are just too varied to generalize about.

So did I get them all? Or are there yet other analytic data management use cases that I don’t fit into my eight categories?

In almost any IT decision, there are a number of environmental constraints that need to be acknowledged. Organizations may have standard vendors, favored vendors, or simply vendors who give them particularly deep discounts. Legacy systems are in place, application and system alike, and may or may not be open to replacement. Enterprises may have on-premise or off-premise preferences; SaaS (Software as a Service) vendors probably have multitenancy concerns. Your organization can determine which aspects of your system you’d ideally like to see be tightly integrated with each other, and which you’d prefer to keep only loosely coupled. You may have biases for or against open-source software. You may be pro- or anti-appliance. Some applications have a substantial need for elastic scaling. And some kinds of issues cut across multiple areas, such as budget, timeframe, security, or trained personnel.

Multitenancy is particularly interesting, because it has numerous implications. If you’re a SaaS vendor supporting multiple customers, you must keep each customer’s data inaccessible to other users* — even if you offer high levels of flexibility or customization. You probably also want to keep data logically partitioned by user, in a way that the DBMS recognizes; you may also want that partition to hunt as a pack for caching purposes, especially if no one customer occupies a large part of your database. Administratively, you need a way to measure customer-specific metrics of the sort that might go into SLAs (Service-Level Agreements).

*Of course, there are exceptions. One of my clients is a SaaS vendor facilitating commerce; the whole point of their app is to let two different customers see and update the same records.

Getting more specific now, I’m usually called upon to advise users in two categories — those that already know they want to upgrade analytic functionality, and those that quickly realize they do once I remind them of it. Even so, many organizations struggle with the question “What do you want to do analytically?” It’s tough to blame them, for the question is distressingly circular; a big part of analytics is figuring out which kinds of analytics are worth doing. Also, SaaS vendors often struggle with the same question for a different reason, responding “Well, we know we’ve only been giving them basic stuff to date. What else do you think they would like?”

There’s no perfect solution to those difficulties, but a good way to start the evaluation is by assessing:

• The nature and value of your decisions that analytics could reasonably affect.
• Your realistic scope for automation of analytic decisions.
• The number and training of your “full-time analysts” — statisticians, SQL jocks who can program, SQL jocks who can’t really program, full-time users of BI tools, whatever.
• The number and training of your “part-time analysts” — normal business users who can get something out of a dashboard, and perhaps even drill down into it.

That should at least tell you which broad categories of analytics you want to engage in, and roughly how advanced in those areas you should try to be.

Basic business intelligence/dashboarding? Surely. Visualization-centric BI? If nothing else, it demos well. Basic predictive modeling? Hmm, are you sure nobody will want that? Advanced predictive modeling? Um, are you sure your users can handle that, or that the results will be worth the investment?

When I talk with users, there’s usually a data management problem in the mix too. In such cases, I quickly ask about data-related metrics, starting with database size, ingest volumes (batch, if relevant, but especially continuous), and simultaneous query load /concurrent user count. Similarly important are requirements for various kinds of latency, the big two being query response time and how long it takes for data to first be available for query. Less numeric questions in a similar vein boil down to “What kinds of requests will you make against the database, in what volume?”

And this loops back to the analytic-user inventory. Suppose you had a near-real-time dashboard — would anybody actually look at it minute to minute?

Specialized metrics I request when considering analytic DBMS include “How many columns are there in your widest table?” and “How many joins — or lines of SQL — are there in your most complex query?”, both of which are tools for assessing “Is your use case naturally columnar?”. Another, more general “natural structure of data” kind of consideration is what structure the data is in before it gets to the database being discussed; candidates include relational batch, XML stream, log file, and many more.

Also crucial are requirements for consistency, availability, and data integrity. Those tell you your needs in high availability and disaster recovery, and perhaps even how picky you have to be about your brands of hardware, software, or cloud/hosting provider. They also indicate how much you should care about relational or ACID properties, and where you should come down on CAP Theorem trade-offs.

I could go on even longer, but those seem like a pretty good set of initial questions with which to start discussions of data management, data integration, and analytic tools and architectures. What do you think I left out? And what do you think I could make substantially clearer by just adding a few more words? Any comments will be much appreciated.

My quick thoughts start:

• Mainly, this isn’t for production usage. But exceptions might arise when:
  • An application, from creation to abandonment, is only expected to have a short lifespan, in support of a specific project.
  • There is an extreme internal-politics bias to operating versus capital expenses, or something like that, forcing a user department to cloud production deployment even when it doesn’t make much rational sense.
  • An application is small enough, or the situation is sufficiently desperate, that any inefficiencies are outweighed by convenience.
• There is non-production appeal. In particular:
  • Spinning up a quick cloud instance can make a lot of sense for a developer.
  • The same goes if you want to sell an Oracle-based application and need to offer demo/test capabilities.
  • The same might go for off-site replication/disaster recovery.

Of course, those are all standard observations every time something that’s basically on-premises software is offered in the cloud. They’re only reinforced by the fact that the only Oracle software Amazon can actually license you is a particularly low-end edition.

And Oracle is indeed on-premises software. In particular, Oracle is hard enough to manage when it’s on your premises, with a known hardware configuration; who would want to try to manage a production instance of Oracle in the cloud?

• SnapLogic has raised about $18 million from Gaurav Dillon and Andreessen Horowitz.
• SnapLogic has almost 60 people.
• SnapLogic has around 150 customers.
• Based in San Mateo, SnapLogic has an office in the UK and is growing its European business.
• SnapLogic has both SaaS (Software as a Service) and on-premise availability, but either way you pay on a subscription basis.
• Typical SnapLogic deal size is under $20K/year. Accordingly, SnapLogic sells over the telephone.
• SnapReduce is in beta with about a dozen customers, and slated for release by year-end.

SnapLogic’s core/hub product is called SnapCenter. In addition, for any particular kind of data one might want to connect, there are “snaps” which connect to — i.e. snap into — SnapCenter.

SnapLogic’s market position(ing) sounds like Cast Iron’s, by which I mean:

• As a practical matter, clients usually first buy SnapCenter to connect on-premise and SaaS applications.
• SnapCenter supports cloud-to-cloud* and on-premise-to-on-premise integration as well.
• SnapCenter itself runs either on-premise on in the cloud. (Larger customers at the moment tend to prefer on-premise deployment.)
• SnapLogic suggests its products are simpler than many ETL alternatives.

Not atypically, SnapLogic believes that SnapCenter is higher-end than Cast Iron (which is now an IBM company), and that SnapCenter’s real top competitor is in-house/hand-coded integration.

*When discussing data integration, “SaaS” and “in the cloud” are close to synonymous.

What SnapLogic said about its use cases seemed to boil down to:

• The base SnapLogic use case is when a client wants to push all data from one application to another application.
• The source and target applications can be any combination of on-premise, SaaS (e.g. salesforce.com), and so on.
• ETL purchases often start when somebody purchases a database (by which I presume SnapLogic also meant data feed).
• SnapLogic sees three main kinds of use case:
  • One-time batch (a big move of historical data).
  • Classic repeating batch.
  • “Trigger-based” (I don’t think SnapLogic was using the term “trigger” just in its technical DBMS sense).
• SnapLogic sees a big future in providing a scalable integration layer for Twitter, RSS feeds, and so on, especially straight into websites, but I get the impression there are only a few pioneering users of such capabilities right now.

The main technical sizzle in the SnapLogic story is the SnapStore, with lets you download free snaps and buy unfree ones.* SnapLogic says there are 100 or so snaps in the SnapStore more, with a couple more being added weekly. That claim started making sense to me when SnapLogic said most snaps are offered by system integrators (as byproducts of specific integration contracts?) or software vendors (to connect to their own offerings?).

*I was expecting snap pricing to be subscription-based also, but when I went to the SnapStore this didn’t seem to be the case.

At least, I think that’s the main sizzle. I’ll confess to not having come away with much understanding of other nuances of SnapLogic technology. In particular, I don’t know what the core data interchange format is that allows all the “simplification” and “normalization” needed for this approach to be possible. So in particular I didn’t drill down far enough to uncover any limitations (functionality or performance) in that aspect of the architecture, and the same goes for SnapCenter’s RESTfulness. That’s all probably my fault; SnapLogic did put a bunch of good people on the phone, and we did at least lay the groundwork for future understanding.

• Two years ago, Attensity merged with two other companies in somewhat related businesses, thus expanding 4X or so in size.
• Due to the merger, Attensity now has two core lines of business:
  • Text analytics.
  • Driving actions, such as call center or social media response, based on text analytics.
• The combined Attensity is part American, part German.
• Attensity’s German part compels it to do some public financial reporting. Attensity will do $50-60 million in 2011 revenue.
• Attensity crunches text in 17 languages. English is preeminent. #2 is — you guessed it! — German.
• A big part of Attensity’s business (or at least of its value proposition) is analyzing the text in social media. Attensity boasts coverage of 75 million social media sources, such as blogs, forums, or review sites.

The four most interesting technical points were probably:

• Attensity has changed how it does exhaustive extraction. I’m having some trouble writing that part up, so for now I’ll just refer you to Attensity’s own description of the new way of doing things.
• Attensity has development work underway meant to address some of the problems in text analytics/other analytics integration. I don’t feel I got enough detail to want to talk about that yet.
• Attensity runs its own data centers, with approximately 60 Hadoop/HBase nodes and 30 nodes of Apache Solr (open source text search). More on that below.
• Attensity now OEMs Vertica. More on that below too.

Some more specific notes include: 

• Attensity has long had customers who use text analytics as an input into churn analysis, for example Charles Schwab.
• At least one customer, who may or may not wish to be named, uses Attensity technology to help de-anonymize social media posters. (I didn’t ask how that worked, actually.)
• Attensity’s founding CTO David Bean has been gone for a while.
• Social media analyzers generally require less sophisticated analytics than Attensity’s older kinds of customers.
• Social media has, in part, a vocabulary all its own.

Attensity and relational DBMS

Notes on Attensity’s choice of DBMS to OEM include:

• Attensity uses Hadoop/HBase itself, but didn’t consider it realistic to try to persuade OEM customers to go that way.
• I get the impression that Attensity’s two finalists were Vertica and Sybase IQ.
• Attensity seems to have considered only query, and not more general analytic platform capabilities, which makes sense given that the evaluation was conducted (starting) in 2009.
• One reason Vertica won was that it required very little tuning.
• Another reason Vertica won was true MPP scale-out, notwithstanding that the largest known installation is capable of running on two nodes (although Attensity recommended that the customer get four just to be on the safe side).
• Sybase IQ’s load speed was even better than Vertica’s.
• Database max-size-to-date metrics include:
  • Under 1 terabyte.
  • 50 million documents (not rows), growing by 1million documents per day..
  • Several hundred million sentences (I guess the documents are short, but it makes sense that they would be).
  • Several billion rows.

It seems there are two parts to the Attensity schema. The raw output of “exhaustive extraction” sounds as if it has rather narrow rows. But Attensity then builds something more star-schema-like to feed into BI tools. Perhaps the latter is the reason for preferring columnar DBMS. There don’t seem to be a lot of auxiliary tables; the only ones Ian cited were:

• Category tables have ontology up to a couple thousand rows
• Tables of terms
• Structured fields that provide metadata for the triples

Previous Attensity database targets (partner, not OEM) included Teradata, SQL Server, Oracle, and MySQL. Hibernate layers were in the mix somewhere too. SQL Server actually had the best performance. I don’t think that’s counting a more recent Sybase IQ partnership, which only racked up a couple of sales.

Attensity, Hadoop, and other non-relational technologies

But that’s OEM. Attensity runs its own data centers, with approximately 60 Hadoop/HBase nodes and 30 nodes of Apache Solr (open source text search).* One reason for moving out of Amazon EC2 was that Solr cried out for solid-state drives; another was just cost.

*But those are just rough figures, from Ian’s memory.

Attensity uses HBase to store full-text documents. However, it doesn’t seem that this is a classic low-latency update HBase use case; Attensity reports doing 3 loads a day, 50 gigabytes of documents total. Apparently that works out to 1 billion documents/month; I gather Attensity just keeps them for 6 months. HBase has been nicely stable for Attensity.

Attensity uses Solr to build distributed search indexes. Solr has not been nicely stable.

What Attensity does in Hadoop seems to be rather simple NLP (Natural Language Processing), plus things one might do in a relational DBMS instead. Examples include:

• Named entity extraction.
• Scoring for sentiment.
• Influence scores, in whatever ways Attensity can calculate them.

There surely also is some basic preprocessing, ingesting text (and document metadata) in various forms and normalizing it into a more standard format. Some real-time ingesting is done outside of Hadoop, in more of a queuing system, the most obvious example of that being the Twitter firehose. Ian suggested that in the future this system will get more uses, in the form of a UIMA-like pipeline.

I further get the impression that Attensity uses Hadoop to do on a SaaS (Software As A Service) basis what its customers do in Vertica. The old idea that Attensity provides hosted services for about half its customers still seems to apply, at least on the new-customer front. However, I’m not sure exactly which product lines Attensity was referring to when they said that.

• Direct improvement in user interface or functionality. Examples (again overlapping) include:
  • True UI enhancements.
  • Application functionality that just lets you do more.
  • Most modern mobile, web, and/or social software efforts, in which a relatively small amount of coding effort produces features that may or may not lead to rapid viral adoption.
  • Ease or functionality not just for end users, but also for administrators. In particular, SaaS, cloud, private cloud and/or appliance benefits are commonly concentrated in this area.
  • Languages and other programmer aids too.
• Performance/efficiency improvement. Overlapping examples include:
  • Anything that directly purports to improve response time, hardware cost or utilization, or power/floor space consumption.
  • Anything to do with parallelization or scale-out.
  • Many, many under-the-covers enhancements to make data more protected (against theft or loss alike), user features snazzier, and so on. With a few exceptions – which are generally regarded as unsolved artificial intelligence problems – almost anything can be hacked together quickly in some high-level programming tool, assuming performance is of no concern. It’s getting the performance remotely right that can often slow market introduction.
• New or enhanced logical data model. Examples of innovation via data model – either truly new or else just newly implemented in a performant way — include:
  • A huge fraction of application innovation, in “traditional” functionality and workflow alike. In several technological eras, just about everything about applications has been a commodity except the data model, but the data model alone was enough to provide long-lasting product differentiation. Indeed, it probably is true today, although that may finally change as business intelligence integration becomes a large part of application software technology.
  • Most things that are called knowledge representation.
  • Many things that are described by terms like “unstructured” or “semi-structured” data.
  • Most innovations described by terms such as metadata management.

To check that I’m not being too glib here, let’s consider a few categories of software technology.

• MPP analytic DBMS are all about performance/efficiency improvement (whether of SQL queries or other analytics), except when they’re about ease of administration and the like.
• Hadoop is about scaling out cheap machines in a way that is (for some purposes) easy to program.
• The core of NoSQL is about efficient scale-out; easier programming also plays a big role.
• Disruptive small vendor business intelligence innovation has a lot to do with better and more useful user experiences, except when it’s about ease of programming and/or administration. The BI industry is also moving to in-memory analytics, which harnesses better performance to provide more interactive user experiences.
• SAS, which has long competed on the basis of superior functionality for statistical programmers, is now also on a big performance kick via MPP analytic DBMS partnerships.
• Oracle’s DBMS efforts have long been focused on performance and administrative usability.
• As noted above, enterprise application functionality is usually all about the data model. Exceptions arise when there is a major generation of UI functionality, such as interactivity (long ago), GUIs (ditto), or BI integration (in its early days now). SaaS is also pitched as an ease-of-everything play.
• Administrative tools are usually about making administration easier. In a few cases (e.g., backups), they’re more about performance.

I’d say my proposed trichotomy is holding up pretty well.

So what set me off on this line of reasoning? Well, Stephen O’Grady wrote

The reason I am against software patents is … very simple. … I am against software patents because it is not reasonable to expect that the current patent system, nor even one designed to improve or replace it, will ever be able to accurately determine what might be considered legitimately patentable from the overwhelming volume of innovations in software. Even the most trivial of software applications involves hundreds, potentially thousands of design decisions which might be considered by those aggressively seeking patents as potentially protectable inventions. If even the most basic elements of these are patentable, as they are currently, the patent system will be fundamentally unable to scale to meet that demand. As it is today.

In addition to questions of volume are issues of expertise; for some of the proposed inventions, there may only be a handful of people in the world qualified to actually make a judgment on whether a development is sufficiently innovative so as to justify a patent. None of those people, presumably, will be employed by the patent office. … Nor will two developers always come to the same conclusions as to the degree to which a given invention is unique.

In considering whether I agreed, I realized that the analysis is different for each of my three categories of innovation mentioned above.

• In the case of a logical data model, O’Grady is almost surely right. Many of those are just copied from the real world anyway, and hence don’t meet any kind of “novel and non-obvious” test. The rest are so general and abstract it’s really hard to say what – if anything – is new and non-obvious about them vs. well-established, often academic prior art.
• In the case of performance enhancements, the core ideas can usually also be found in well-established computer science publications. What’s more, the true innovations may be such simple algorithms that they’re not patentable. What’s left over is incremental enhancement. Once again, O’Grady is right.
• But the case of user interface/experience enhancements is not so clear. Inventor comes up with a useful idea for something that hasn’t been built before. Inventor builds and patents it. I’m not sure how that’s different from the case of building physical devices of various kinds, which have been patented for centuries. Determining what’s novel or non-obvious doesn’t seem to require specialized technical knowledge, at least not above and beyond that required in other disciplines.

Bottom line: There are many other reasons to oppose software patents, but Stephen O’Grady’s “It’s impossible to adjudicate them fairly” argument remains unproven, at least when it is applied to software enhancements whose essence is better designs for user experiences.

Related links:

• My negative comments about patents in the areas of MapReduce and columnar DBMS
• Three standpoints from which to view a software product strategy

• The most important issue in database and analytic technology, in my opinion, isn’t technological at all – rather, it’s the legal and political steps needed to preserve liberty in the face of advancing, intrusive technology.
• Another important issue for society – and this one does involve a lot of technology – is scientific number crunching. In particular, database technology for scientific computing needs to be developed much further. I’ll have more to say on all this soon.
• More generally, technology needs to keep advancing for parallel analytics. Fortunately, it is. Watch this space over the next few weeks.
• Oracle has said, in effect, that its most important technological challenge of the decade is getting solid-state memory right. I agree.
• Data volumes will keep going up, up, up. Technology needs to keep evolving accordingly. Much of what I write is on that subject.
• Data needs to be processed and analyzed at very different latencies. And there’s much further to go in integrating disparate latencies.
• Analytic database management in the cloud hasn’t been solved yet, especially for Big Data. Among the reasons are the difficulty of moving data into the cloud (unless it originated there), the slowness of moving it from node to node in shared-nothing architectures (which reduces the elasticity benefit), and above all the long and unpredictable latencies of interprocessor communication while queries are running (a key subject of discussion at the Boston Big Data Summit).
• Better business intelligence user interfaces are increasingly available. I’m thinking particularly of approaches with buzzwords like visualization/interactive exploration or faceted. But they aren’t well-integrated into the overall analytic stack, as big BI vendors are trailing the smaller ones in this regards. (Part of the problem relates to my previous point.)
• Application development over text search isn’t in the same league as application development over relational DBMS. The choices are mainly XML (e.g., MarkLogic), SQL for text integrated into RDBMS (limited by the weakness of those integrations), and something like Attivio’s Java SDK. There’s a major conceptual barrier in building those apps, namely the unpredictability of query results. Still, it should be possible to do better.
• Similarly, text analytics and conventional analytics exist well side by side. They can even be in the same database and/or dashboard, although in practice that is limited by the strong SaaS focus of text mining vendors and users. But analytic integration of them is really hard. Linguistic imprecision is, in my opinion, only the #2 reason for this difficulty. The #1 reason is that trends detected by text analytics are much less precise than trends on tabular data – e.g., a 50% increase in a certain kind of complaint may be no more significant than a 5% change in a revenue variable.
• I’m increasingly persuaded that graph analytics can be handled without a graph-centric data model. But right now, it isn’t being handled well at all. Lots more needs to be done – although when it is, it will just exacerbate the privacy/liberty dangers that so concern me.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Interesting trends in database and analytic technology