I mean for the term “data mustering” to encompass at least three scenarios:

• Integrated (relational) data warehouse.
• Big bit bucket.
• Big bit stream.

Let me explain what I mean by each. 

“Integrated data warehouse” is a phrase Teradata has started using for enterprise data warehouses that, like approximately every other EDW in the entire history of data warehousing, aren’t truly enterprise-wide. In other words, it means “not just a data mart”. No category name is perfect, but I think that one works reasonably well.

I previously described the big bit bucket use case as

Users take a whole lot of data, often machine-generated data in logs of different kinds, and dump it into one place, managed by Hadoop, at open-source pricing.

and quickly added

Of course, there are various outfits who’d like to sell you not-so-cheap bit buckets. Contending technologies include Hadoop appliances (which I don’t believe in), Splunk (which in many use cases I do), and MarkLogic (ditto, but often the cases are different from Splunk’s). Cloudera and IBM, among other vendors, would also like to sell you some proprietary software to go with your standard Apache Hadoop code.

I think I’ll stand pat on that explanation.

By analogy, a big bit stream is various streams of data, assembled in the custody of a streaming engine. Sybase told me Wednesday that this scenario appears in both of the traditional markets for CEP/streaming — national intelligence, where it is a major use of streaming, and capital markets in some use cases as well. And it’s consistent with what I’ve heard from other CEP/streaming vendors as well.

As for where I got the word “mustering” — it’s a military term, for when you assemble your troops and their gear either for inspection or for actual use. The main modern usage I know of the word is as part of the phrase “pass muster”, which originally referred to the concept that the person being paid to put a regiment together should from time to time demonstrate that the regiment physically existed in the form that regimental records seemed to show.

The core mission of Teradata Unity is asynchronous, near-real-time replication across Teradata systems. The point of “asynchronous” is performance. The point of “near-real-time” is that it Teradata Unity can be used for high availability and disaster recovery, and further can be used to allow real work on HA and DR database copies. Teradata Unity works request-at-a-time, which limits performance somewhat;* Unity has a lock manager that makes sure updates are applied in the same order on all copies, in cases where locks are needed at all.

*Other options, more suitable for bulk loading and so on, are on the Teradata Unity roadmap.

The idea of doing real work on your high availability or disaster recovery database copies is an important one. Teradata systems are often used for the kinds of mission-critical purposes that call for such extra 2- or 3-way mirroring; so the ability to use all the systems for real work offers, if not exactly 2-3X price/performance savings, at least something significant. Teradata reports low but non-zero penetration in its customer base for active-active replication today. But I’m hopeful that number will increase, as Teradata Unity looks to be a big improvement over the possibilities that existed before.

In theory, the whole workload could be split among mirror-copy systems, although I’m sure we could construct various edge-case scenarios in which doing so would be a Bad Idea. In practice, I’d normally think of using second/third copies of a data warehouse for specific workloads, such as:

• Long-running queries or other analytic exercises.
• Virtual data marts.
• Backups, exports, and so on.

Another possibility to consider is only mirroring part of your database for HA or DR, since not all missions are equally critical. Yet another possibility is to mirror the whole thing, but on systems with different performance characteristics; in case of failover, you might only keep the most crucial applications up, while turning the others off until you can again run on a system powerful enough to handle them.

As Teradata tells it, Teradata Unity has two key aspects:

• Multi-System Synchronization. DDL, DML and DCL (Data Description/Manipulation/Control Language) all are replicated. I.e., data gets copied around, and so does everything else.
• Query Management. Queries get shipped around to appropriate systems based on:
  • Which systems manage all the data needed to execute the query.
  • Which is up and running at the moment.
  • Which is backlogged at the moment. (I get the impression Teradata Unity load balancing is fairly basic in the first release, but there is some.)

Further details may be seen in the slide deck Teradata graciously sent over for posting.

And finally, here’s some Teradata product name housekeeping:

• The first release of Teradata Unity will be numbered 13.10. Not coincidentally, that’s the version number of Teradata’s latest database software. Teradata Unity 13.10 will support Teradata 13 and 13.10.
• Teradata Unity 14 will ship soon after Teradata 14. It will support Teradata 13, 13.10, and 14.
• Teradata Unity runs on a “Managed Server”. “Managed Servers” are nodes inside Teradata’s cabinets, managed by Teradata’s system software and so on, but which do not run Teradata database software.
• In the 14.10 release, Teradata Unity:
  • Will scale out across multiple Managed Servers.
  • Will be able to serve as a general load facility for Teradata.
  • Teradata Unity replaces Teradata Query Direct.
  • While I suspect Teradata Unity may replace Teradata Data Mover (bulk data copy) in the future, it surely doesn’t yet.
  • Teradata Unity works with Teradata Multi-System Manager, which does things like end-to-end job management).

• My most important post of the week was a general guide to IT vendor strategy. That one has already spawned discussion at many companies, from the tiny to the multi-billion-dollar.
• The best comment thread of the week was probably on my post about scale-out relational OLTP choices, in which people discussed the merits of various particular alternatives.
• I recommended that people strongly consider attending XLDB 5 in Menlo Park on October 18-19.

Most of the posts, however, were reactions to news events. In particular:

• Teradata announced that Teradata 14 will be hybrid-columnar, more in Vertica’s way than in Greenplum’s or Aster Data’s. (Pay no attention to the Wall Street Journal’s apparent belief that no other analytic DBMS is hybrid-columnar at all.)
• Aster announced the unsurprising news that there will be a Teradata Aster appliance. Also, Aster talked about greater analytic flexibility in the forthcoming Aster 5.0.
• With Oracle OpenWorld coming up, Oracle decided to get some of its announcing out of the way early. In particular, it announced the Oracle Database Appliance, which is small-business-friendly hardware for running the Oracle DBMS. However, the Oracle Database Appliance doesn’t seem to do much about the complexity of running the Oracle DBMS software.
• In a catch-all Hadoop post, I noted that:
  • Oracle has now clearly said it has a Hadoop appliance coming, no doubt next week at OpenWorld.
  • I still can’t see why Hadoop appliances would succeed, but a lot of smart folks seem to disagree with me.
  • Greenplum announced what looks like a nice but unimportant little product upgrade.
  • It’s a really good thing that previously reported plans to revamp Hadoop are underway.
• DataStax announced that it really is a Cassandra company after all. Pay no attention to previous marketing that seemed to put DataStax in the same Hadoop-alternative category as, say, MapR.
• Ingres has changed its name to Actian. The announcement seems like a confession that Ingres and VectorWise are going nowhere.

• Teradata is proud that Teradata 14′s workload management now explicitly manages I/O, to go with Teradata’s long-standing management of CPU. Teradata’s WLM still does not explicitly manage RAM.
• Aster is proud that Aster 5′s workload management now explicitly manages RAM, to go along with the WLM capabilities Aster has had for a while managing CPU and I/O. Aster’s Tasso Argyros believes this is an important capability, at least in some edge cases.
• Mike Pilcher of SAND emailed me that SAND’s WLM capabilities to explicitly manage CPU, I/O, and RAM are very well-received by the marketplace.

One would think that Teradata’s workload management is more sophisticated and powerful than Aster Data’s.* So I asked Scott Gnau what gives (he was pretty much the ideal guy to comment, since he runs development for Teradata and oversees Teradata’s Aster acquisition as well).

*Except, of course, that Aster was a pioneer in having workload management cover all kinds of analytic processes, rather than just traditional database requests.

Scott’s main response was that Aster’s system was much more consumptive of RAM than Teradata’s; indeed, he reminded me that in the very old days, Teradata could make do with as little as 4 megabytes. Scott also did not argue when I suggested that Aster’s not-just-database analytic processes might require large amounts of RAM as well.

• Teradata emailed me a couple of months ago saying something like that at that point they could count 16 petabyte-level customers. In response to my repeated requests for clarification, Teradata has explicitly refused to identify the metric used in reaching that conclusion.
• At some point Teradata did something — as per a tweet of his — to convince Neil Raden that they have 20 petabyte-class users.
• That tweet was made around the time that Teradata apparently showed a slide naming big users at the Strata conference (last week).
• If Teradata is counting the way they did three years ago, that count of 16 or 20 or whatever is probably inflated compared to, say, Vertica’s figure of 7 a few months back.
• Even so, it’s obvious — and not just from the eBay example — that Teradata has one of the most scalable analytic DBMS offerings around.

• Oracle does not actually offer columnar I/O; the other three systems do. But see the “I won’t be surprised” part in yesterday’s Teradata post.
• Aster does not offer columnar compression; the other three do.
• EMC Greenplum and Teradata offer different kinds of ways to mix column and row storage in the same table; each has its advantages.
• Teradata generally has a more mature and capable offering than EMC Greenplum, for most purposes, whichever way you choose to organize your tables.

Edit: The Wall Street Journal got this wrong, writing that Teradata was the first-ever hybrid columnar system. Specifically, they wrote

While columnar technology has been around for years, Teradata says its product is unique because it allows users to include both columns and rows in the same database.

Googling on “Teradata To Unveil New Analytics Product To Speed Business Adoption” might get you around the paywall to see the offending piece.

Along with the announcements comes updated positioning such as:

• Better SQL than the MapReduce alternatives have.
• Better MapReduce than the SQL alternatives have.
• Easy(ier) way to do complex analytics on multi-structured data. (Aster has embraced that term.)

and of course

• Now also with Teradata’s beautifully engineered hardware and system management software!

As might also be expected, the announcements are accompanied by pictures along the lines of “There are your various data sources; there’s Teradata; there’s Aster; there’s Hadoop; look at all the nice arrows connecting them!”

Teradata Aster further decided it was time for a 5.0 DBMS release. Highlights include:

• Aster’s SQL-MapReduce has more flexible inputs. Specifically, if you view SQL/ MapReduce as steroid-enhanced table functions, those functions can now each have multiple tables as input. Aster is rightly positioning this as the key feature of the Aster 5.0 release.
• Workload management now explicitly manages not only CPU and I/O, but also RAM. That surely makes it safer to use algorithms which aggressively create temporary data structures. And the allocation is dynamic, in that it can be throttled back if workloads require.
• There’s more SQL functionality — I think this is minor, as Aster seems to have had pretty good SQL coverage already.
• Performance has been improved; i.e., Bottleneck Whack-A-Mole has progressed in multiple ways. One improvement Aster thinks is cutting-edge is a hybrid kind of join that tries to be a hash, then reverts to a merge if it has to spill out of memory. (E.g., if the available RAM is throttled back.)

Also, Aster is always expanding its library of prebuilt analytic functions/packages — often in connection with specific customer engagements — and took this opportunity to mention numerous recent or near-future additions to the list.

Part of Aster’s motivation in making multiple input tables available to its parallel analytic functions seems to be to allow the use of intermediate result sets alongside raw data. In some ways, this seems to be an alternative to the MPI-based approach favored by SAS, and highlights limitations of the vanilla MapReduce paradigm. The specific examples given were k-means clustering and — which I’d never heard of before — SAX pattern matching.

For an example of two true data tables being used as inputs, Aster offered a case of advertising attribution, with the data being about impressions and also conversions. Frankly, I suspect a “join them all and let MapReduce sort them out” strategy would also work for that application; if you join on something like Customer_ID, just how big would the result set really be? Even so, we can imagine other cases in which messy boundaries for graphs or time series makes that strategy unappealing, and — you read it here first! — Aster’s target use cases are focused on time series and graphs.

And finally: Whenever I ask the Aster folks “So, how big are Aster databases that are actually in production?”, they try to convince me that this is the wrong thing to ask. But — without actually answering the question — they did say:

• The new Teradata Aster appliance has been tested to a couple hundred terabytes.
• They are very confident about scaling Aster to a few hundred terabytes.
• They don’t have much in the way of proof in the 1 petabyte range.

The basic idea of Teradata Columnar is:

• Each table can be stored in Teradata in row format, column format, or a mix.
• You can do almost anything with a Teradata columnar table that you can do with a row-based one.
• If you choose column storage, you also get some new compression choices.

The “mix” option is like Vertica’s FlexStore, in that different columns (e.g. different components of a street address) can be grouped into a mini-row, even if you otherwise choose to store that table in a columnar way. Teradata does not at this time offer the Greenplum or Aster way of mixing rows and columns, whereby some of the rows in a table can be stored in a column-store way, while other rows are stored in entire-row row-store solidarity

Thus, Teradata Columnar gives you many of the basic I/O and compression benefits of columnar DBMS, along with all the usual Teradata goodness of concurrency, workload management, system management, concurrency, SQL support, and so on. By way of comparison:

• Similar things are true of Greenplum’s offering (except for the parts about concurrency, advanced workload management, and so on).
• Aster doesn’t have columnar compression.
• Oracle has columnar compression but no true columnar storage.*

Also, as I noted above, Teradata mixes rows and columns in a different way than Aster or EMC Greenplum do.

*However, I won’t be surprised if Oracle soon announces true hybrid-columnar as well. I originally heard about Teradata Columnar and Oracle’s efforts to develop true hybrid-columnar storage the same week, 23 months ago.

Going hybrid-columnar is a big deal. Aster Data, for example, told me that a considerable fraction of all its workloads ran faster with columnar than row-based storage.* And it’s of extra importance to a vendor that, like Teradata, needs to play catch-up in the compression derby.

*Anything in which the queries eliminated more than half or so of the columns (60%, if I recall correctly, but it was definitely an approximate figure). That pretty much means any query except full and near-full table scans.

Teradata’s columnar compression story is pretty complicated. To quote from a forthcoming press release:

Teradata automatically chooses from among six types of compression: run length, dictionary, trim, delta on mean, null and UTF8. based on the column demographics.

The trickiest words in that are “automatic” and “dictionary”. Teradata divides column-store data into “column containers” of, say, 8 KB. (Current thinking is 8 KB default, 65 KB maximum, but that could change by the time of product release.) By default, Teradata software decides separately for each column container which compression algorithm(s) to use. It can even change its mind dynamically over time, as the contents of the container change.

What I find weird about Teradata’s columnar dictionary compression is that the dictionary is container-specific. One benefit versus having a more global dictionary is that, since you compress fewer items, compression tokens can each be shorter. (The length of a typical token is a lot like the log of the cardinality of the dictionary.) Another benefit is that smaller dictionaries are faster to search. The obvious offsetting drawback is that a larger and more global dictionary has the potential to compress various items that wind up being left uncompressed in this smaller-scale scheme.

Other notes about Teradata compression include:

• Teradata has for a while had a more manual form of dictionary compression.
• Teradata also has block-level compression.
• You can do block-level compression even on top of the columnar compression described above.
• The Teradata/Rainstor partnership for archiving-level compression that Rainstor made so much fuss about doesn’t seem to actually be happening; Teradata seems content with the other compression choices it offers.

And finally, Teradata 14 extends Teradata Virtual Storage with a feature called Compress on Cold. The idea is that “cold” data can safely get (extra) compression — that block-level stuff — automatically. If the data heats up again (e.g. by becoming relevant for a while to the latest year-over-year comparisons) it can be just as automatically removed from compression. Teradata thinks this is significantly better than the alternative of making manual compression choices based on not-so-granular range partitions.

Unsurprisingly, Teradata lacks some features and benefits found in certain columnar-first analytic DBMS. One biggie is that, absent clever workarounds such as Vertica’s in-memory write-optimized store, columnar DBMS have a single-row-update performance problem, because you are putting the information in many places on disk rather than just one. I generally take it for granted that a columnar-first vendor has such a workaround. Row-based vendors gone columnar, however, are a different story. Teradata et al. are also likely to decompress data and reassemble it into full rows as soon as it hits RAM, which obviates the potential benefit that you have less data per row clogging up cache.* (Edit: As per Todd Walter’s comments below, this is not accurate — and that’s a potentially important feature.)

*Late decompression actually depends on columnar compression, not columnar storage, and hence can also be enjoyed by row-based DBMS such as DB2.

To use Teradata Columnar, you need to be using round-robin data distribution rather than, say, hash. Teradata jargon for this is NoPI, where the “PI” stands for Primary Index.* Drawbacks to that include:

• You don’t get the hash distribution benefit of saving a data redistribution step on joins whose join key happens to be the same as the hash key.
• In Teradata-land, NoPI implies append-only, so you get the garbage collection/compactification that implies.

However, that’s a physical append-only; you can still do logical updates.

*PI is not to be confused with PPI, which stands for Primary Partition Index, and is Teradata’s name for range (or case-statement-based) partitioning. PPI works just fine with Teradata Columnar. As of Teradata 14, you can do PPI up to 62 levels deep.

The Teradata folks also sent along a slide deck laying out parts of the Teradata Columnar story. But it’s not one of the better Teradata decks I’ve ever posted.

Also, Aster no longer focuses much on the general internet industry where it got its earliest sales, its continued success at LinkedIn and a recent win at an (NDA) fairly-big-name internet new account Razorfish notwithstanding. That said, the first target market Aster did share with me was “digital marketing optimization,” which includes “marketing optimization” (duh), search engine optimization (SEO), clickstream analysis, and the like. Also, Aster is going after “data scientists” in general, and that’s a term I’m still seeing used most frequently in the internet area.

I’m seeing ever more granularity as companies break down internet-related market segments. DataStax showed me a chart last week of 15 different market segments it had sold into, and at least 14 were in some way internet-related.

Rather, if Aster is to name three industries in which it has pleasingly strong sales traction, it would say manufacturing (which in Teradata lingo includes resource extraction), financial services (including insurance), and retail. A cynic might note that that breakdown, like many similar ones, adds up to fairly large swaths of the economy and the computer market, but never mind that part. (Other firms might have thrown in telecommunications and health care as well, to get even more coverage.

Two of Aster’s other favorite application areas are social network analysis/influencer identification and — which is analytically very similar — fraud detection/prevention. Taken together, that’s a whole lot of graph analysis. And I note with interest that the influencer identification stuff does NOT seem to be concentrated in telecom, which is the traditional sector one would imagine it being used in; all those call records are a lovely source of graph edges. Rather, the influencers seem to be identified from sources such as social media and credit card data .

Once again, this kind of thing gives me privacy jitters.

The match between Aster’s favorite industries and application areas is pretty much as you might expect — fraud in financial services, influencer analysis in retailing (and probably consumer financial services too), and digital marketing in both. As for manufacturing, the opportunities there seem to be focused on machine-generated data. That would be at least in high-tech manufacturing (I bet especially in flow-oriented stuff such as semiconductor fab) and oil/gas. Smart grid opportunities don’t seem to have arisen yet for Aster the way they have for a couple other vendors.

As for general Aster business trends, I think they’re good, while Aster would perhaps want to portray them as very good. Aster named a couple of impressive joint Teradata/Aster wins under NDA, but only a couple. Ramping up sales headcount is proving challenging, and some sales leadership turnover probably hasn’t helped. I do believe Aster’s spin that this is a matter of somebody being promoted quickly to a bigger job, and am optimistic about the current team — still, such moves tend to have at least short-term cost.

Let’s try eight categories instead. While no categorization is ever perfect, these each have at least some degree of technical homogeneity. Figuring out which types of analytic database you have or need — and in most cases you’ll need several — is a great early step in your analytic technology planning. 

Enterprise data warehouse (Full or partial)

• Kinds of data likely to be included: All, but especially operational
• Likely use styles: All
• Canonical example: Central EDW for a big enterprise
• Stresses: Concurrency, reliability, workload management

The enterprise data warehouse (EDW) ideal says that you copy all your data into one place, and drive all decision-making from there. Full EDWs are pipedreams. Still, a partial EDW makes sense for most large enterprises, and many indeed already have one. The first product lines to consider for classical EDWs are Teradata, DB2, Exadata, and maybe Microsoft SQL Server, especially if you’re going to stress concurrency and/or operational use cases.

Traditional data mart

• Kinds of data likely to be included: All
• Likely use styles: Business intelligence, budgeting/consolidation, investigative
• Examples: Reporting servers, planning/consolidation servers, anything MOLAP, etc.
• Stresses: Performance, concurrency, TCO

Whether or not you have something like an enterprise data warehouse, it’s common to have lighter-weight data marts as well. A traditional data mart might drive reports and dashboards. Or it might be specialized for budgeting, planning, and/or consolidation.  Some investigative analytics may be in the mix as well.

Any DBMS that can support an EDW can also support a data mart, but it may not be the most cost-effective way to do so. Columnar DBMS might have more attractive performance and TCO (Total Cost of Ownership); the same goes for Netezza. Some of them — e.g. Sybase IQ and Vertica — have excellent track records in concurrent usage as well. Ted Codd pushed what amounts to MOLAP (Multidimensional OnLine Analytic Processing) systems for these use cases. But relational DBMS commonly do a better job, which is one reason most major MOLAP products have wound up at RDBMS companies.

Investigative data mart — agile

• Kinds of data likely to be included: All, especially customer-centric
• Likely use styles: Investigative
• Canonical example: A few analysts getting a few TB to examine
• Stresses: Ease of setup/load, ease of admin, price/performance

Besides the traditional data mart, there are at least two other kinds. Both are focused on investigative analytics, but they’re differentiated by database size.

If you have just a few analysts,* looking at no more than a few terabytes of data (perhaps even just some gigabytes) — and if that data is “single-subject” and fairly homogenous — your watchwords should be “cheap”, “easy”, and “fast”. You don’t need to invest in much hardware, in expensive software, in much administrative effort (the analysts can be their own DBAs),  nor should you endure much set-up time. Just grab a product, grab some data, and start running queries (or extracts into the statistical tool of your choice).

*If you have dozens or even hundreds of analysts hitting the same database, you’re probably back to the more concurrency-oriented scenarios outlined above.

Infobright is often cost-effective among columnar analytic DBMS. Other vendors might cut you a price break as well. If you have multiple terabytes of data, don’t rule out Netezza’s lowest-end products (even if they’d really rather sell you something bigger). Or, if you’re in the sub-terabyte range, maybe you can get by with an in-memory BI tool such as QlikView, and not do anything special on the DBMS side at all.

Investigative data mart — big

• Kinds of data likely to be included: All, especially customer-centric, logs, financial trade, scientific
• Likely use styles: Investigative
• Canonical example: Single-subject 20 TB – 20 PB relational database
• Stresses: Performance, scale-out, analytic functionality

But if you’re looking at tens of terabytes of relational data, or even more, you really do have a “big data” problem. Performance and scalability are major challenges, usually best addressed by MPP (Massively Parallel Processing) systems, such as Netezza, Vertica, Aster Data, ParAccel, Teradata, or Greenplum. Performance POCs (Proofs Of Concept) are a big part of the buying process. Vendor price negotiations are crucial too.

Actually, in the low tens of terabytes you might be able to get away with a shared-disk system that has excellent compression — e.g., columnar products like Sybase IQ, Infobright, or SAND, rather than just Vertica and ParAccel.

Assuming you have affordable, scalable query performance, the competitive differentiator can switch to additional analytic functionality. Aster, Netezza, ParAccel, Vertica, and Greenplum either offer full analytic platforms, or seem to be on the path to doing so. Teradata, which now owns Aster Data, offers substantial built-in analytic capability in its traditional products as well, and the same goes for Sybase IQ.

Continued in Part 2, where we cover some of the more difficult use cases.