The basic idea of Teradata Columnar is:

• Each table can be stored in Teradata in row format, column format, or a mix.
• You can do almost anything with a Teradata columnar table that you can do with a row-based one.
• If you choose column storage, you also get some new compression choices.

The “mix” option is like Vertica’s FlexStore, in that different columns (e.g. different components of a street address) can be grouped into a mini-row, even if you otherwise choose to store that table in a columnar way. Teradata does not at this time offer the Greenplum or Aster way of mixing rows and columns, whereby some of the rows in a table can be stored in a column-store way, while other rows are stored in entire-row row-store solidarity

Thus, Teradata Columnar gives you many of the basic I/O and compression benefits of columnar DBMS, along with all the usual Teradata goodness of concurrency, workload management, system management, concurrency, SQL support, and so on. By way of comparison:

• Similar things are true of Greenplum’s offering (except for the parts about concurrency, advanced workload management, and so on).
• Aster doesn’t have columnar compression.
• Oracle has columnar compression but no true columnar storage.*

Also, as I noted above, Teradata mixes rows and columns in a different way than Aster or EMC Greenplum do.

*However, I won’t be surprised if Oracle soon announces true hybrid-columnar as well. I originally heard about Teradata Columnar and Oracle’s efforts to develop true hybrid-columnar storage the same week, 23 months ago.

Going hybrid-columnar is a big deal. Aster Data, for example, told me that a considerable fraction of all its workloads ran faster with columnar than row-based storage.* And it’s of extra importance to a vendor that, like Teradata, needs to play catch-up in the compression derby.

*Anything in which the queries eliminated more than half or so of the columns (60%, if I recall correctly, but it was definitely an approximate figure). That pretty much means any query except full and near-full table scans.

Teradata’s columnar compression story is pretty complicated. To quote from a forthcoming press release:

Teradata automatically chooses from among six types of compression: run length, dictionary, trim, delta on mean, null and UTF8. based on the column demographics.

The trickiest words in that are “automatic” and “dictionary”. Teradata divides column-store data into “column containers” of, say, 8 KB. (Current thinking is 8 KB default, 65 KB maximum, but that could change by the time of product release.) By default, Teradata software decides separately for each column container which compression algorithm(s) to use. It can even change its mind dynamically over time, as the contents of the container change.

What I find weird about Teradata’s columnar dictionary compression is that the dictionary is container-specific. One benefit versus having a more global dictionary is that, since you compress fewer items, compression tokens can each be shorter. (The length of a typical token is a lot like the log of the cardinality of the dictionary.) Another benefit is that smaller dictionaries are faster to search. The obvious offsetting drawback is that a larger and more global dictionary has the potential to compress various items that wind up being left uncompressed in this smaller-scale scheme.

Other notes about Teradata compression include:

• Teradata has for a while had a more manual form of dictionary compression.
• Teradata also has block-level compression.
• You can do block-level compression even on top of the columnar compression described above.
• The Teradata/Rainstor partnership for archiving-level compression that Rainstor made so much fuss about doesn’t seem to actually be happening; Teradata seems content with the other compression choices it offers.

And finally, Teradata 14 extends Teradata Virtual Storage with a feature called Compress on Cold. The idea is that “cold” data can safely get (extra) compression — that block-level stuff — automatically. If the data heats up again (e.g. by becoming relevant for a while to the latest year-over-year comparisons) it can be just as automatically removed from compression. Teradata thinks this is significantly better than the alternative of making manual compression choices based on not-so-granular range partitions.

Unsurprisingly, Teradata lacks some features and benefits found in certain columnar-first analytic DBMS. One biggie is that, absent clever workarounds such as Vertica’s in-memory write-optimized store, columnar DBMS have a single-row-update performance problem, because you are putting the information in many places on disk rather than just one. I generally take it for granted that a columnar-first vendor has such a workaround. Row-based vendors gone columnar, however, are a different story. Teradata et al. are also likely to decompress data and reassemble it into full rows as soon as it hits RAM, which obviates the potential benefit that you have less data per row clogging up cache.* (Edit: As per Todd Walter’s comments below, this is not accurate — and that’s a potentially important feature.)

*Late decompression actually depends on columnar compression, not columnar storage, and hence can also be enjoyed by row-based DBMS such as DB2.

To use Teradata Columnar, you need to be using round-robin data distribution rather than, say, hash. Teradata jargon for this is NoPI, where the “PI” stands for Primary Index.* Drawbacks to that include:

• You don’t get the hash distribution benefit of saving a data redistribution step on joins whose join key happens to be the same as the hash key.
• In Teradata-land, NoPI implies append-only, so you get the garbage collection/compactification that implies.

However, that’s a physical append-only; you can still do logical updates.

*PI is not to be confused with PPI, which stands for Primary Partition Index, and is Teradata’s name for range (or case-statement-based) partitioning. PPI works just fine with Teradata Columnar. As of Teradata 14, you can do PPI up to 62 levels deep.

The Teradata folks also sent along a slide deck laying out parts of the Teradata Columnar story. But it’s not one of the better Teradata decks I’ve ever posted.

Bit bucket

• Kinds of data likely to be included: Logs, other technical/external
• Likely use styles: Staging/ETL, investigative
• Canonical example: Log files in a Hadoop cluster
• Stresses: TCO, scale-out, transform/big-query performance, ETL functionality

With the explosion of machine-generated data has come the need for a place to put it all, sometimes called the big bit bucket. This is like the investigative data mart for big databases, but more poly-structured. In some cases it is focused on data staging and transformation; but it can also be used for analysis in place.

The list of candidate technologies to run your bit bucket starts with Hadoop and Splunk.

Archival data store

• Kinds of data likely to be included: Operational, CDR (call detail record), security log
• Likely use styles: Archival, reporting (for compliance), possibly also investigative
• Examples: Any long-term detailed historical store
• Stresses: TCO, compression, scale-out, performance (if multi-use)

Analytic DBMS vendors have been insulting each other with the claim “that’s just an archival data store,” dating back at least to the first time Greenplum was deployed on an underpowered Sun Thumper system. Perhaps only Rainstor truly embraces the archival positioning, and I’ve become pretty dubious about their technical claims and their company alike.

Still, there’s a legitimate need for data stores — especially relational analytic DBMS that:

• Store data cheaply, with high rates of compression.
• Have decent performance if you do want to query the data.
• May have archiving/compliance-specific features as well.

Along with Rainstor, SAND and SenSage have at least partially targeted that use case. In addition, appliance vendors such as Teradata and Netezza try to have an archive-oriented product version in their lineups.

Outsourced data mart

• Kinds of data likely to be included: All
• Likely use styles: Traditional BI, investigative analytics, staging/ETL
• Examples: Advertising tracking, SaaS CRM
• Stresses: Performance, TCO, reliability, concurrency

Much of what happens in analytic database management can also be outsourced. Some applications that run via SaaS (Software as a Service) are analytic. I’ve had three different clients whose main business is picking marketing targets in various vertical segments; others who wanted to add analytics to what were historically OLTP applications; and others yet who just offered online business intelligence. Also, if your fundamental business is gathering data and reselling it to a variety of user organizations, that’s an analytic data management challenge. The possibilities expand from there.

Data outsourcers are in the IT business, and so their IT development is — hopefully! — more serious and less politically encumbered than at many conventional enterprises. Thus, legacy systems and master data management issues are commonly less prevalent, or at least more aggressively disposed of. The same, up to a point, goes for vendor politics.*  Multitenancy is commonly an issue, as is running in the cloud.

*Even so, there’s often That Guy who doesn’t want to migrate away from Oracle, no matter what.

Vertica gets the nod in a number of these cases; it’s cloud-friendly, and often the problem is naturally columnar. Other columnar products can be good choices too, with added brownie points for Infobright if the shop is MySQL-oriented anyway. Running Netezza or other appliances makes sense mainly if you’re pretty sure you want to keep operating your own data centers, but some data outsourcers are just fine with that assumption.

Operational analytic(s) server

• Kinds of data likely to be included: Customer-centric, log, financial trade
• Likely use styles: Advanced operational analytics
• Examples:
  • Lower latency: Web or call-center personalization, anti-fraud
  • Higher latency: Customer profiling, Basel 3 risk analysis
• Stresses: Performance, reliability, analytic functionality, perhaps concurrency

Even with eight different choices, I need a “catch-all” category; this is it.

Suppose you want to do reasonably sophisticated analytics, then use the results in operations. This is the classical challenge in integrating short-request and analytic processing. There are multiple ways to tackle it, embodying different trade-offs in cost, convenience, or analytic accuracy. If the platform on which you want to run your investigative analytics also has the reliability and concurrency appropriate for mission-critical operations, you’re set. Otherwise, you may want to pipe derived data into a more “industrial-strength” DBMS, ideally the one that runs your operational apps anyway

Another option is to integrate a limited amount of analytics immediately into your short-request processing system. For example, as bad as they are at the kinds of queries that require joins, NoSQL systems are often fast at simple aggregations. As MapReduce/NoSQL integrations mature, that option may not require pumping the data anywhere else for deeper analytics; even if it does, at least you’re starting out with the data in a convenient bit bucket.

Streaming/CEP-centric architectures could come into play as well. And it goes on from there. The possibilities in this last category are just too varied to generalize about.

So did I get them all? Or are there yet other analytic data management use cases that I don’t fit into my eight categories?

• Compliance-specific features.
• Bottleneck Whack-A-Mole.

The point is that Rainstor is focusing its efforts on enterprises that: 

• Have a compliance mandate to keep detailed information, either now or coming down the pike.
• Would like to query the information, either as part of the compliance mandate or for the usual business reasons one does analysis (or for that matter pinpoint lookup of historical information).
• Might want to delete the information as soon as the compliance mandate runs out. (That’s a new feature. Frankly, I think the clients demanding it are being foolish. Information is valuable and should never be thrown away if one can afford to keep it.)
• Might want to annotate the information, even though it is being preserved immutably. (Also a new feature. I think that one is smart.)

“Application retirement” was mentioned only in the context of Rainstor’s flagship Informatica partnership, and even then mainly for clients who had a compliance reason to keep old application data around. “Cloud” and “private cloud” get mentioned, but they don’t seem to be as central as Rainstor was previously hoping they would be. (This is one area we could and probably should have touched on more had I been more awake.)

One thing that hasn’t changed:  “Information preservation,” which I coined for Rainstor at our first meeting, is still the company catchphrase.

So far as I could tell, the big point on Rainstor 4 Bottleneck Whack-A-Mole is this: When you load data into Rainstor (bulk or otherwise), it likes to do some metadata analysis first. (I imagine this is related to the sophisticated Rainstor compression scheme.) Well, that isn’t much of a performance hit for schemas with small numbers of tables, but is a bigger deal for more complex schemas. The Rainstor 4 fix is to remember/persist some of that analysis from one time the database is updated until the next time. Sounds obvious, but so do a lot of bottleneck fixes once they are made.

Robert Hodges of Continuent offers a great outline of Continuent’s clustering story, with a lot of “Now we got right what we previously didn’t know/admit we got wrong.” Continuent now claims to have a strong clustering offering, both paid and free/open-source, for both MySQL and PostgreSQL, with Oracle support perhaps coming really soon.

Merv Adrian, who has overrated the importance of TPC benchmarks in the past, seems to have become more skeptical.

Interim CEO Mark Burton laid out Infobright’s focus pretty clearly when he took over:

… the focus must be in building products that fit market segments where ease-of-use and easily attainable performance are valued.  This doesn’t sound like the high end of Data Warehousing to me where highly complex MPP architectures and teams of DBAs spend their time.  It sounds like the realm of Departmental IT and SMB where business leaders are in a hurry to gain access to data and answers without the lead time and pain of complex architectures and high costs.

I’m hearing about a SaaS focus from a lot of companies. The Continuent link above mentions one. So does RainStor’s latest blog post. Gooddata, a SaaS vendor itself, seems focused on analyzing data that was originally created via SaaS. I haven’t talked with Cast Iron or Pervasive for a while, but when I did, their ETL market targeting was all about SaaS. And of course, I hear dumber SaaS-focus ideas as well. I think the biggest substantive reason for this trend is — if you don’t have the broadest feature set, and fear large enterprises therefore won’t want your stuff, going after SMBs makes sense. And SMBs are presumed to be going SaaS. Also in the mix, of course, are a single platform to support, a small number of large SaaS vendors to sell to or partner with, and/or general trendiness.

• RainStor expects to finish the year with > 50 users (overwhelmingly via partners)
• A big market for RainStor (at least in terms of signed partnerships and large deal activity) is retention of telecom records, for compliance purposes, typically for a 1-3 year period. This includes:
  • CDRs (Call Detail Records)
  • Mobile phone records including CDRs and missed calls
  • SMS (Short Message Service), including the complete text of same
• RainStor thinks a number of larger telcos have the need to store a billion records per day each. (I’m not sure how many subscribers such a telco would have to have).
• John further thinks that, for the same query performance, RainStor can handle such a database on 4 blades. More precisely, he says that’s what happened at a test conducted by a major technology firm. In the same test case, SenSage required 40 blades, and Oracle required 80 or more cores on a pair of big SMP machines. John further says that the Oracle solution required a new table and new tablespace every day, while RainStor’s took 3 days for initial installation and required no DBA afterwards. However, I’m in no position to verify this report independently.
• In a different kind of proof point, so extreme it gives even the RainStor folks pause, a user has retired 300 different applications and put their databases onto a single 2-core box. (Presumably, this is via RainStor’s OEM relationship with Informatica.)
• Coming Very Soon are some services tying RainStor’s DBMS to obvious-suspect SaaS offerings. The core positioning is “SaaS data escrow”.i.e., RainStor will help you ensure that, in a worst-case scenario, there’s a nice safe copy of your data you can get at. RainStor also encourages you to do basic reporting and BI against the RainStor copy of the data, if you choose.
• The idea I’ve been pushing lately of taking a heterogeneous replication offering like Continuent’s and having it feed an archiving store like RainStor’s has hit a rather basic snag. RainStor doesn’t actually consume change data capture kinds of information directly, at least as of yet, because of difficulties fitting such a stream into its guaranteed-data-immutability model.

*I coined that category description for John in the tea room of the Park Lane Hotel. He’s subsequently embraced it enthusiastically, and I kind of like it myself.

Related links

• RainStor’s approach to compression, as described by me and by RainStor itself

To the extent there’s a Clearpace secret sauce, it seems to lie in NParchive’s unusual data access method.  NParchive doesn’t just tokenize the values in individual columns; it tokenizes multi-column fragments of rows.  Which particular columns to group together in that way seems to be decided automagically; the obvious guess is that this is based on estimates of the cardinality of their Cartesian products.

Of the top of my head, examples for which this strategy might be particularly successful include:

• Denormalized databases
• Message stores with lots of header information
• Addresses

• Fully functional relational DBMS.
• Claims of fast query performance, but that’s not how they’re sold.
• Huge compression.
• Careful attention to time-stamping and auditability.

*Actually, SAND has two products, one of which really is sold as a DBMS, competing with Sybase IQ or Netezza. But I’m talking about the other one, which is the current main focus of SAND’s sales efforts.

When Clearpace CEO John Bantleman and I chatted last week, he spoke of such uses as:

• Cheap compliance with data-retention regulations
• Keeping data accessible even though the application that created it has been decommissioned
• Cheap duplication for disaster recovery

He also invoked the buzzphrase “information lifecycle management” (ILM).

When I pointed out that all of this could be construed as being aspects of “information preservation,” John enthusiastically agreed. Yesterday I bounced that phrase off SAND’s marketing chief Linda Arens, and she liked it too.

And that makes perfect sense. What do “archives” and “archivists” do in the classical senses of the terms? First and foremost, they preserve information. They don’t feel they’ve done their job well if it’s too too difficult to access, but utter ease-of-use is not their top concern.

Digression: I actually spent a day once with a university archivist (retired). She came to my house to check out a portrait of one of my Monasch ancestors and to rummage through my 19^th Century family photos. Australian readers — and WW1 history buffs — will have little trouble guessing which university she was from.

So far, so good. But why use a specialty product for the purpose of information preservation, when you can instead just dump everything into your data warehouse environment? Well, the vast majority of large enterprises do just that, getting by without specialized technology from SAND, Clearpace, or any close competitor. And of course data warehouse technology is getting cheaper very quickly. So not all enterprises will ever need what SAND and Clearpace have to offer.

But every enterprise does need to think about a comprehensive information preservation strategy. Too often ILM puts the cart before the horse, focusing on throwing stuff away more than on keeping it. Notwithstanding the excessive popularity of some inherently shady legal tricks — “Let’s make sure to destroy the evidence before somebody can think of ordering us to preserve it” — and also notwithstanding some legitimate rules about privacy — preserving information is almost always better than losing it, whether accidentally or on purpose.

So I’d like to propose a deceptively simple exercise for any enterprise, really of any size. Inventory all the sources of potentially valuable information that are already being tracked in your enterprise. Then make a matching list of the preservation strategies for each. Some of those strategies will be very good. Others will fall into that ever-popular category “not ideal, but also not bad enough to bother fixing.” Then see which kinds of information are covered neither by a good preservation strategy, nor one that’s good enough. And think about whether you should move all those into one or two* information preservation environments of last resort.

*Two = one for tabular data + one for documents and media

• NParchive takes a multi-version concurrency control approach. Data is never updated in place; new information is just appended. Clearpace is careful to “time-proof” the data, keeping track and allowing the unwinding of, for example, changes in schema table structure.
• Data is stored in very large blocks – the default is 1 million rows. Currently any change to actual data values – as opposed to just database design changes – requires rewriting a whole block, but a redo log is on the roadmap.
• NParchive has four different approaches to compression, which can be used in series. Clearpace says that if any two of the four work well on a particular data set, 20X compression is realistically. If all four work well, 50-100X can be achieved. Presumably, not all have to be turned on for any particular database.
• Three of NParchive’s approaches to compression are pretty standard – tokenization, a “collection of cheap, standard compression algorithms” (including delta, which often works well), and EDLIB.
• The fourth part of the NParchive compression story has something to do with representing records as trees, and noticing when patterns are repeated and deduping them. I’m still fuzzy on how that all works.  (Edit: I subsequently posted an explanation of that part.)
• Clearpace believes NParchive’s query performance is competitive with Oracle’s but not, say, Netezza’s. (And yes, that’s a meaningful assertion, even if you believe that all Oracle performance problems are solely due to poor implementation practices.)
• Clearpace says that no database administration is ever needed. Everything happens automagically – or as they say nowadays, “autonomically.”

According to Clearpace CEO John Bantleman, NParchive use cases include:

• Archiving data warehouses
• Archiving log files and similar kinds of data that never made it into a data warehouse
• Storing – and making available for query – data from decommissioned old applications

If I understood a couple of actual OEM stories correctly, we can also add to the list the archiving of transaction processing databases. Buzzphrases mentioned included information lifecycle management (ILM) and disaster recovery.

And then I coined a database archiving buzzphrase of my own …