• PostgreSQL is good technology.
• MySQL is narrowing the gap, but PostgreSQL is still ahead of MySQL in some ways.  (Database extensibility if nothing else.)
• PostgreSQL has a lot of users. (Many of them in academia and/or Russia.)
• Neither EnterpriseDB (which now calls itself “The enterprise PostgreSQL company”) nor the PostgreSQL community leadership have covered themselves with stewardship glory.
• A significant number of interesting DBMS products can be regarded as PostgreSQL forks (e.g. Greenplum, Aster Data nCluster, Netezza if you squint, and Vertica if you stand on your head*).
• PostgreSQL advancement is not dead. For example, Hadapt beta users are running actual PostgreSQL on many nodes each.
• There’s no assurance that Oracle will be a benevolent MySQL steward forever. (Specifically, Oracle’s “Play nicely with others” antitrust commitments expire in 2014.)

So I think it would be cool if one or the other big company put significant wood behind the PostgreSQL arrow.

*While Vertica was originally released using little or no PostgreSQL code — reports varied — it featured high degrees of PostgreSQL compatibility.

• My most important post of the week was a general guide to IT vendor strategy. That one has already spawned discussion at many companies, from the tiny to the multi-billion-dollar.
• The best comment thread of the week was probably on my post about scale-out relational OLTP choices, in which people discussed the merits of various particular alternatives.
• I recommended that people strongly consider attending XLDB 5 in Menlo Park on October 18-19.

Most of the posts, however, were reactions to news events. In particular:

• Teradata announced that Teradata 14 will be hybrid-columnar, more in Vertica’s way than in Greenplum’s or Aster Data’s. (Pay no attention to the Wall Street Journal’s apparent belief that no other analytic DBMS is hybrid-columnar at all.)
• Aster announced the unsurprising news that there will be a Teradata Aster appliance. Also, Aster talked about greater analytic flexibility in the forthcoming Aster 5.0.
• With Oracle OpenWorld coming up, Oracle decided to get some of its announcing out of the way early. In particular, it announced the Oracle Database Appliance, which is small-business-friendly hardware for running the Oracle DBMS. However, the Oracle Database Appliance doesn’t seem to do much about the complexity of running the Oracle DBMS software.
• In a catch-all Hadoop post, I noted that:
  • Oracle has now clearly said it has a Hadoop appliance coming, no doubt next week at OpenWorld.
  • I still can’t see why Hadoop appliances would succeed, but a lot of smart folks seem to disagree with me.
  • Greenplum announced what looks like a nice but unimportant little product upgrade.
  • It’s a really good thing that previously reported plans to revamp Hadoop are underway.
• DataStax announced that it really is a Cassandra company after all. Pay no attention to previous marketing that seemed to put DataStax in the same Hadoop-alternative category as, say, MapR.
• Ingres has changed its name to Actian. The announcement seems like a confession that Ingres and VectorWise are going nowhere.

• Teradata is proud that Teradata 14′s workload management now explicitly manages I/O, to go with Teradata’s long-standing management of CPU. Teradata’s WLM still does not explicitly manage RAM.
• Aster is proud that Aster 5′s workload management now explicitly manages RAM, to go along with the WLM capabilities Aster has had for a while managing CPU and I/O. Aster’s Tasso Argyros believes this is an important capability, at least in some edge cases.
• Mike Pilcher of SAND emailed me that SAND’s WLM capabilities to explicitly manage CPU, I/O, and RAM are very well-received by the marketplace.

One would think that Teradata’s workload management is more sophisticated and powerful than Aster Data’s.* So I asked Scott Gnau what gives (he was pretty much the ideal guy to comment, since he runs development for Teradata and oversees Teradata’s Aster acquisition as well).

*Except, of course, that Aster was a pioneer in having workload management cover all kinds of analytic processes, rather than just traditional database requests.

Scott’s main response was that Aster’s system was much more consumptive of RAM than Teradata’s; indeed, he reminded me that in the very old days, Teradata could make do with as little as 4 megabytes. Scott also did not argue when I suggested that Aster’s not-just-database analytic processes might require large amounts of RAM as well.

• Oracle does not actually offer columnar I/O; the other three systems do. But see the “I won’t be surprised” part in yesterday’s Teradata post.
• Aster does not offer columnar compression; the other three do.
• EMC Greenplum and Teradata offer different kinds of ways to mix column and row storage in the same table; each has its advantages.
• Teradata generally has a more mature and capable offering than EMC Greenplum, for most purposes, whichever way you choose to organize your tables.

Edit: The Wall Street Journal got this wrong, writing that Teradata was the first-ever hybrid columnar system. Specifically, they wrote

While columnar technology has been around for years, Teradata says its product is unique because it allows users to include both columns and rows in the same database.

Googling on “Teradata To Unveil New Analytics Product To Speed Business Adoption” might get you around the paywall to see the offending piece.

Along with the announcements comes updated positioning such as:

• Better SQL than the MapReduce alternatives have.
• Better MapReduce than the SQL alternatives have.
• Easy(ier) way to do complex analytics on multi-structured data. (Aster has embraced that term.)

and of course

• Now also with Teradata’s beautifully engineered hardware and system management software!

As might also be expected, the announcements are accompanied by pictures along the lines of “There are your various data sources; there’s Teradata; there’s Aster; there’s Hadoop; look at all the nice arrows connecting them!”

Teradata Aster further decided it was time for a 5.0 DBMS release. Highlights include:

• Aster’s SQL-MapReduce has more flexible inputs. Specifically, if you view SQL/ MapReduce as steroid-enhanced table functions, those functions can now each have multiple tables as input. Aster is rightly positioning this as the key feature of the Aster 5.0 release.
• Workload management now explicitly manages not only CPU and I/O, but also RAM. That surely makes it safer to use algorithms which aggressively create temporary data structures. And the allocation is dynamic, in that it can be throttled back if workloads require.
• There’s more SQL functionality — I think this is minor, as Aster seems to have had pretty good SQL coverage already.
• Performance has been improved; i.e., Bottleneck Whack-A-Mole has progressed in multiple ways. One improvement Aster thinks is cutting-edge is a hybrid kind of join that tries to be a hash, then reverts to a merge if it has to spill out of memory. (E.g., if the available RAM is throttled back.)

Also, Aster is always expanding its library of prebuilt analytic functions/packages — often in connection with specific customer engagements — and took this opportunity to mention numerous recent or near-future additions to the list.

Part of Aster’s motivation in making multiple input tables available to its parallel analytic functions seems to be to allow the use of intermediate result sets alongside raw data. In some ways, this seems to be an alternative to the MPI-based approach favored by SAS, and highlights limitations of the vanilla MapReduce paradigm. The specific examples given were k-means clustering and — which I’d never heard of before — SAX pattern matching.

For an example of two true data tables being used as inputs, Aster offered a case of advertising attribution, with the data being about impressions and also conversions. Frankly, I suspect a “join them all and let MapReduce sort them out” strategy would also work for that application; if you join on something like Customer_ID, just how big would the result set really be? Even so, we can imagine other cases in which messy boundaries for graphs or time series makes that strategy unappealing, and — you read it here first! — Aster’s target use cases are focused on time series and graphs.

And finally: Whenever I ask the Aster folks “So, how big are Aster databases that are actually in production?”, they try to convince me that this is the wrong thing to ask. But — without actually answering the question — they did say:

• The new Teradata Aster appliance has been tested to a couple hundred terabytes.
• They are very confident about scaling Aster to a few hundred terabytes.
• They don’t have much in the way of proof in the 1 petabyte range.

Also, Aster no longer focuses much on the general internet industry where it got its earliest sales, its continued success at LinkedIn and a recent win at an (NDA) fairly-big-name internet new account Razorfish notwithstanding. That said, the first target market Aster did share with me was “digital marketing optimization,” which includes “marketing optimization” (duh), search engine optimization (SEO), clickstream analysis, and the like. Also, Aster is going after “data scientists” in general, and that’s a term I’m still seeing used most frequently in the internet area.

I’m seeing ever more granularity as companies break down internet-related market segments. DataStax showed me a chart last week of 15 different market segments it had sold into, and at least 14 were in some way internet-related.

Rather, if Aster is to name three industries in which it has pleasingly strong sales traction, it would say manufacturing (which in Teradata lingo includes resource extraction), financial services (including insurance), and retail. A cynic might note that that breakdown, like many similar ones, adds up to fairly large swaths of the economy and the computer market, but never mind that part. (Other firms might have thrown in telecommunications and health care as well, to get even more coverage.

Two of Aster’s other favorite application areas are social network analysis/influencer identification and — which is analytically very similar — fraud detection/prevention. Taken together, that’s a whole lot of graph analysis. And I note with interest that the influencer identification stuff does NOT seem to be concentrated in telecom, which is the traditional sector one would imagine it being used in; all those call records are a lovely source of graph edges. Rather, the influencers seem to be identified from sources such as social media and credit card data .

Once again, this kind of thing gives me privacy jitters.

The match between Aster’s favorite industries and application areas is pretty much as you might expect — fraud in financial services, influencer analysis in retailing (and probably consumer financial services too), and digital marketing in both. As for manufacturing, the opportunities there seem to be focused on machine-generated data. That would be at least in high-tech manufacturing (I bet especially in flow-oriented stuff such as semiconductor fab) and oil/gas. Smart grid opportunities don’t seem to have arisen yet for Aster the way they have for a couple other vendors.

As for general Aster business trends, I think they’re good, while Aster would perhaps want to portray them as very good. Aster named a couple of impressive joint Teradata/Aster wins under NDA, but only a couple. Ramping up sales headcount is proving challenging, and some sales leadership turnover probably hasn’t helped. I do believe Aster’s spin that this is a matter of somebody being promoted quickly to a bigger job, and am optimistic about the current team — still, such moves tend to have at least short-term cost.

It’s blindingly obvious that Zynga is one of Vertica’s petabyte-scale customers, given that Zynga sends 5 TB/day of data into Vertica, and keeps that data for about a year. (Zynga may retain even more data going forward; in particular, Zynga regrets ever having thrown out the first month of data for any game it’s tried to launch.) This is game actions, for the most part, rather than log files; true logs generally go into Splunk.

I don’t know whether the missing data is completely thrown away, or just stashed on inaccessible tapes somewhere.

I found two aspects of the Zynga story particularly interesting. First, those 5 TB/day are going straight into Vertica (from, I presume, memcached/Membase/Couchbase), as Zynga decided that sending the data to some kind of log first was more trouble than it’s worth. Second, there’s Zynga’s approach to analytic database design. Highlights of that include:

• Data is divided into two parts. One part has a  pretty ordinary schema; the other is just stored as a huge list of name-value pairs. (This is much like eBay‘s approach with its Teradata-based Singularity, except that eBay puts the name-value pairs into long character strings.) About half the data is in each part, but I don’t think that’s by deliberate choice.
• Zynga adds data into the real schema when it’s clear it will be needed for a while. This isn’t a matter of query volumes, for the most part; rather, it’s when Zynga’s tests (e.g. of new games?) have determined that the data will keep being collected and used for a while.
• Zynga only adds columns to its analytic  database; it never goes through the more complex process of deleting them.

Just as Zynga is one of Vertica’s flagship accounts, LinkedIn is one of Aster Data’s. Specifically, before leaving LinkedIn for Aster, Jonathan Goldman built LinkedIn’s People You May Know feature in Aster nCluster. This was long ago, and I’m not sure how sophisticated his use of SQL and MapReduce would be in today’s terms; for example, I was told he didn’t use “nPath or anything like that.” (Edit: See the comments below for clarifications from Jonathan.) Anyhow, LinkedIn has replaced Aster for PYMK with Hadoop, and in my opinion is getting much better results.

That, from an Aster standpoint, is the bad news. The good news is that LinkedIn is happily using Aster nCluster for several other applications; LinkedIn folks doesn’t seem to regret throwing out* Greenplum for Aster; and they also seem to have a very high opinion of Jonathan and his work while he was there.

*And this time that is indeed the phrase that was used.

One thing that astonished me is that LinkedIn PYMK is based only on data innate to LinkedIn (as opposed to imported email addresses, the results of web crawls, and so on). Given that, I am at a loss to explain how it suggested a couple of old friends, to whom I have no discernable chain of connection. Yes, we were at Harvard at the same time, but if that’s all it was, there would be a huge number of false positives I’m not actually seeing.

Let’s try eight categories instead. While no categorization is ever perfect, these each have at least some degree of technical homogeneity. Figuring out which types of analytic database you have or need — and in most cases you’ll need several — is a great early step in your analytic technology planning. 

Enterprise data warehouse (Full or partial)

• Kinds of data likely to be included: All, but especially operational
• Likely use styles: All
• Canonical example: Central EDW for a big enterprise
• Stresses: Concurrency, reliability, workload management

The enterprise data warehouse (EDW) ideal says that you copy all your data into one place, and drive all decision-making from there. Full EDWs are pipedreams. Still, a partial EDW makes sense for most large enterprises, and many indeed already have one. The first product lines to consider for classical EDWs are Teradata, DB2, Exadata, and maybe Microsoft SQL Server, especially if you’re going to stress concurrency and/or operational use cases.

Traditional data mart

• Kinds of data likely to be included: All
• Likely use styles: Business intelligence, budgeting/consolidation, investigative
• Examples: Reporting servers, planning/consolidation servers, anything MOLAP, etc.
• Stresses: Performance, concurrency, TCO

Whether or not you have something like an enterprise data warehouse, it’s common to have lighter-weight data marts as well. A traditional data mart might drive reports and dashboards. Or it might be specialized for budgeting, planning, and/or consolidation.  Some investigative analytics may be in the mix as well.

Any DBMS that can support an EDW can also support a data mart, but it may not be the most cost-effective way to do so. Columnar DBMS might have more attractive performance and TCO (Total Cost of Ownership); the same goes for Netezza. Some of them — e.g. Sybase IQ and Vertica — have excellent track records in concurrent usage as well. Ted Codd pushed what amounts to MOLAP (Multidimensional OnLine Analytic Processing) systems for these use cases. But relational DBMS commonly do a better job, which is one reason most major MOLAP products have wound up at RDBMS companies.

Investigative data mart — agile

• Kinds of data likely to be included: All, especially customer-centric
• Likely use styles: Investigative
• Canonical example: A few analysts getting a few TB to examine
• Stresses: Ease of setup/load, ease of admin, price/performance

Besides the traditional data mart, there are at least two other kinds. Both are focused on investigative analytics, but they’re differentiated by database size.

If you have just a few analysts,* looking at no more than a few terabytes of data (perhaps even just some gigabytes) — and if that data is “single-subject” and fairly homogenous — your watchwords should be “cheap”, “easy”, and “fast”. You don’t need to invest in much hardware, in expensive software, in much administrative effort (the analysts can be their own DBAs),  nor should you endure much set-up time. Just grab a product, grab some data, and start running queries (or extracts into the statistical tool of your choice).

*If you have dozens or even hundreds of analysts hitting the same database, you’re probably back to the more concurrency-oriented scenarios outlined above.

Infobright is often cost-effective among columnar analytic DBMS. Other vendors might cut you a price break as well. If you have multiple terabytes of data, don’t rule out Netezza’s lowest-end products (even if they’d really rather sell you something bigger). Or, if you’re in the sub-terabyte range, maybe you can get by with an in-memory BI tool such as QlikView, and not do anything special on the DBMS side at all.

Investigative data mart — big

• Kinds of data likely to be included: All, especially customer-centric, logs, financial trade, scientific
• Likely use styles: Investigative
• Canonical example: Single-subject 20 TB – 20 PB relational database
• Stresses: Performance, scale-out, analytic functionality

But if you’re looking at tens of terabytes of relational data, or even more, you really do have a “big data” problem. Performance and scalability are major challenges, usually best addressed by MPP (Massively Parallel Processing) systems, such as Netezza, Vertica, Aster Data, ParAccel, Teradata, or Greenplum. Performance POCs (Proofs Of Concept) are a big part of the buying process. Vendor price negotiations are crucial too.

Actually, in the low tens of terabytes you might be able to get away with a shared-disk system that has excellent compression — e.g., columnar products like Sybase IQ, Infobright, or SAND, rather than just Vertica and ParAccel.

Assuming you have affordable, scalable query performance, the competitive differentiator can switch to additional analytic functionality. Aster, Netezza, ParAccel, Vertica, and Greenplum either offer full analytic platforms, or seem to be on the path to doing so. Teradata, which now owns Aster Data, offers substantial built-in analytic capability in its traditional products as well, and the same goes for Sybase IQ.

Continued in Part 2, where we cover some of the more difficult use cases.

Back in April, Steve Miller suggested that approximate BI could be a growing trend, gaining speed at the expense of (often false anyway) precision. That idea of course goes well with Infobright’s recent released Rough Query feature, and also with Datameer’s year-earlier pitch.

Aster Data (now a Teradata company) is positioning itself as analyzing multi-structured data — which is my second-choice term, behind the more precise but odder-sounding “poly-structured.” I hope “poly-structured” wins, and plan to keep using it myself; but I recognize that “multi-structured” may actually be the one that prevails.

Barbara Darrow wrote a great piece on Oracle Exadata pricing. Highlights include:

• Routine Oracle software discounts are high.
• Exadata discounts are higher.
• Big/referenceable customers get the best Exadata discounts. The term “extremely deep” was used. (I’ve also heard that from Oracle competitors, with the term “free” even coming up, hyperbolically or otherwise.)
• Oracle’s hardware maintenance pricing is forcing users to trash Sun gear, even when it’s working. One guy told the story of literally crying as the Sun boxes were taken away.
• Oracle’s 22% of license maintenance fee goes up to 27% after two years. I didn’t know that.

Oracle has been making considerable messaging fuss around a win in Japan, where Softbank replaced years-old Teradata systems with vastly less new Exadata gear. I blogged that this is hardly an apples-to-apples comparison. During my visit last April, Oracle pushed back, in particular pointing out that the Softbank division that awarded the deal was very separate from the one that was an Oracle reseller. But Monday Teradata shared with me a counter-pushback, asserting that during the recent worldwide recession, Softbank assigned its underemployed systems integration division to do internal projects — including the data warehouse upgrade. I.e., Teradata stands by its claim that this replacement was strongly influenced by the Softbank/Oracle partnership.

If you’re analytically inclined, Kx Systems has some interesting ideas, manifested in kdb+ and so on. A 2009 ACM article seems as good a starting point as any, the company’s website probably aside. Confusingly, Kx is small company that evidently does most of its selling through a couple of much larger partners. Also, 1010data happens to be built on an older version of Kx’s technology.

Known HDFS and Hadoop data storage and management issues include but are not limited to:

• Hadoop is run by a master node, and specifically a namenode, that’s a single point of failure.
• HDFS compression could be better.
• HDFS likes to store three copies of everything, whereas many DBMS and file systems are satisfied with two.
• Hive (the canonical way to do SQL joins and so on in Hadoop) is slow.

Different entities have different ideas about how such deficiencies should be addressed. 

For most practical purposes, Yahoo’s and IBM’s views about Hadoop have converged. Yahoo and IBM both believe that Hadoop data storage should be advanced solely through the Apache Hadoop open source process. In particular:

• IBM and Yahoo both talk of the great undesirability of Hadoop “forking” like Unix did.
• Yahoo appeared on stage at IBM’s analyst event this week to reinforce the meeting-of-the-minds, even though there’s no IBM/Yahoo customer relationship involved.
• IBM has disclaimed any intention of providing its own Hadoop distribution, but even so is committed to selling lots of IBM InfoSphere BigInsights, which incorporates Apache Hadoop.*
• Yahoo has stopped offering its own Hadoop distribution, period.

*IBM is emphatic about ruling out marketing terms whose connotation it doesn’t like. IBM’s Hadoop distribution isn’t a “distribution,” because that might make it sound too proprietary; IBM’s Oracle emulation offering isn’t an “emulation” offering, because that might make it sound too slow; and IBM’s CEP product InfoSphere Streams isn’t a “CEP” product, because that might make it sound too non-functional.

Cloudera can probably be regarded as part of the Yahoo/IBM camp, some stern looks from IBM in Cloudera’s direction notwithstanding. Cloudera Enterprise — also an embrace-and-extend offering — remains the obvious choice for enterprises Hadoop users; meanwhile, nobody has convinced me of any bogosity in the “no forking” claim Cloudera makes for its free/open source Hadoop distribution. Indeed, when I visited Cloudera a couple of weeks ago, Mike Olson showed me a slide demonstrating that Cloudera might be supplanting Yahoo as the biggest ongoing contributor to Apache Hadoop.

EMC’s Data Computing Division, nee’ Greenplum, made a lot of Hadoop noise this week. Unlike Yahoo, IBM, and Cloudera, EMC really is forking Hadoop. I’m not talking with the EMC/Greenplum folks these days, but the whole thing was covered from various angles by Lucas Mearian, Doug Henschen, Derrick Harris, and Dave Menninger.

Another option is to entirely replace HDFS with a DBMS, whether distributed or just instanced at each node. DataStax is doing that with Cassandra-based Brisk; Hadapt plans to do that with PostgreSQL and VectorWise (edit: As per the comment below, Hadapt only plans a partial replacement of HDFS); and Netezza’s analytic platform has a Hadoop-over-Netezza option as well. Mike Olson objects to such implementations being called “Hadoop”; but trademark issues aside, those vendors plan to support a broad variety of Hadoop-compatible tools. Aster Data has long taken that approach one step further, by offering an enhanced version of MapReduce — aka SQL/MapReduce — over its nCluster DBMS. And 10gen offers a more primitive form of MapReduce with MongoDB, but probably wouldn’t position it as addressing a “MapReduce market” at all.