• Oracle thinks flash memory is the most important hardware technology of the decade, one that could lead to Oracle being “bumped off” if they don’t get it right.
• Juan believes the “bulk” of Oracle’s business will move over to Exadata-like technology over the next 5-10 years. Numbers-wise, this seems to be based more on Exadata being a platform for consolidating an enterprise’s many Oracle databases than it is on Exadata running a few Especially Big Honking Database management tasks.

And by the way, Oracle doesn’t make its storage-tier software available to run on anything than Oracle-designed boxes.  At the moment, that means Exadata Versions 1 and 2. Since Exadata is by far Oracle’s best DBMS offering (at least in theory), that means Oracle’s best database offering only runs on specific Oracle-sold hardware platforms.

*E.g., I was sitting upstairs in my parents’ apartment in Columbus, OH having the call while their doctor, who I’ve never met, was visiting downstairs. He offered to make a special trip back Saturday afternoon because he missed me Wednesday, but he’s notorious for not coming when he says he will. Update: He didn’t come Saturday. On Saturday he said he’d come Sunday. He didn’t do that either.

Other high- and lowlights of our conversation included:

• Flash is the main new hardware element in Exadata Version 2. Otherwise, Exadata 2 is just an annual refresh of Exadata Version 1 to include updated components (Nehalem chips, bigger disk drives, etc.)
• Juan thinks it’s suboptimal to use flash memory through the bottleneck of disk controllers, favoring PCIe cards instead. (I emphatically agree.)
• Juan resolutely ducked questions about actual Exadata production deployment. Literally the only fact he shared in that regard is that there are at least 2 Exadata production systems running that each have 2 or more racks cabled together.
• Juan stressed that Exadata runs apps written over Oracle DBMS unchanged.
• When making mixed-workload claims for Exadata 2, Juan stressed consolidation of multiple databases, some OLTP and some analytic. He didn’t really argue with my skepticism about integrating OLTP and analytics in the same database, with one exception:
• Juan pointed out that in major OLTP apps such as ERP systems, there often is actually more processing going on in reporting and other batch stuff than there is in true OLTP.
• Exadata 2’s flash memory is designed as a disk cache, smarter than LRU (Least Recently Used). The two examples Juan gave of “smarter than LRU” are that backups and table scans don’t flush the cache.
• I forget whether this is new in Exadata 2 (I think it is), but anyhow – Exadata has a “Storage Index” that’s a lot like a Netezza zone map. I.e., for each megabyte or so of data it stores the min and max value of every column; if a query predicate rules out those ranges, that megabyte is never retrieved.
• Oracle has long offered what sounds like flexible workload management capability, and this has now been extended to specifically include I/O resources on the storage tier.
• This isn’t Exadata-specific, but Oracle has built a file system on top of its DBMS, optimized for speed, which helps with, e.g., ELT (Extract/Load/Transform). Evidently, it’s not at all the same thing as Mark Benioff’s 1990s Microsoft-annoying IFS (Internet File System) project, which seems to have morphed into a content management SDK.

Highlights specifically in the area of parallelization included:

• Juan stressed that all databases consolidated onto an Exadata machine are/should be striped across all storage units.
• On the other hand, Juan said that different databases should be confined to specific cores or CPUs on the database tier.
• But on the third hand, Juan also stressed – in what could be called a “private cloud” pitch – that there’s great elasticity as to which databases are matched to which server CPUs.
• Contrary to what I thought he and/or his colleagues told me a year ago, Juan said RAC (Real Application Clusters) is a big part of Oracle’s data warehouse processing.
• However, Juan says that what I regard(ed) as a major objection to Oracle’s database-tier parallelization — the need to manually specify “degrees of parallelism” — has now been obviated by automation. Juan thinks that few data warehouse DBAs will now need to manually tune parallelism, with minor exceptions. One exception he cites is that if a nightly report really is non-urgent, it can just be forced to run on a single core with no chance to grab more resources. (However, Juan thinks manual tuning of parallelism will continue to play a greater role in OLTP.)

OK. That’s all I can get done tonight (see above re: inconvenience of timing). Follow-on subjects I’d like to and indeed plan to post about include:

• What Juan said about hybrid columnar compression
• Oracle’s delightfully non-confidential slide deck, and a few comments about same

• Registration for tomorrow’s webinars
• Replay of the first webinar
• My slides from the first webinar

The main subjects of the webinar will be:

• Some review of material from the first webinar (all three presenters)
• Discussion of how MapReduce can help with three kinds of analytics:
  • Pattern matching (Jonathan will give detail)
  • Number-crunching (I’ll cover that, and it will be short)
  • Graph analytics (I haven’t written the slides yet, but my starting point will be some of the relationship analytics ideas we discussed in August)

Arguably, aspects of data transformation fit into each of those three categories, which may help explain why data transformation has been so prominent among the early applications of MapReduce.

As you can see from Aster’s title for the webinar (which they picked while I was on vacation), at least their portion will be focused on customer analytics, e.g. web analytics.

Aster Data is officially announcing the 4.0 release of nCluster. There are two big pieces to this announcement:

• Aster is offering a slick vision for integrating big-database management and general analytic processing on the same MPP cluster, under the not-so-slick name “Data-Application Server.”
• Aster is also offering a sophisticated vision for workload management.

In addition, Aster has matured nCluster in various ways, for example cleaning up a performance problem with single-row updates.

Highlights of the Aster “Data-Application Server” story include:

• At its core, the Aster “Data-Application Server” is the Aster nCluster MPP analytic DBMS, enhanced with basic application server functionality (I didn’t ask for details of that part), running on the same nCluster worker nodes that answer SQL queries.
• Thus, Aster is eliminating a lot of the data movement that plagues three-tier architectures and other less-integrated approaches.
• The Aster “Data-Application Server” further offers integrated workload management for applications and queries; more on that below.
• The Aster “Data-Application Server” requires applications to be parallelized and invoked via Aster’s SQL/MapReduce.
• As befits a MapReduce-based system, the Aster “Data-Application Server” lets you write your applications in lots of different languages (the usual suspects, and it also does .NET).
• The Aster “Data-Application Server” runs applications in their own process spaces, protecting the DBMS server from crashes and other damaging behavior.
• The Aster “Data-Application Server” allows applications to manage memory themselves, persistently, and not just via relational constructs. Thus, if you want your application to maintain a graph, mini rules engine, and/or finite state machine, you can, without doing SQL contortions.

In a compelling proof point for the Aster Data-Application Server’s slickness, Aster has leapfrogged Teradata and Netezza in the extent to which SAS functionality is integrated into Aster’s DBMS. (Aster and SAS both say that you can do full SAS modeling in parallel on Aster, but even so I wouldn’t be surprised to discover there were some parts of SAS’ system that turned out to be exceptions.) Of course, Aster is hardly the only analytic DBMS vendor to have the idea of explicitly enhancing general analytic processing; that’s why we see lots of MapReduce announcements, and it’s also why Teradata enhanced its UDFs (User-Defined Functions) to have some kind of persistent memory.* But I don’t know of anybody else whose approach is quite so elegant and general at this time.

*Unfortunately, I don’t yet know much about Teradata’s UDF enhancements. I neglected to drill down on Global Persistent Memory when it was mentioned a couple of times at Teradata Partners last week, and Teradata was unable to accommodate my request this week for a rapid follow-up briefing on the subject.

Aster’s approach to workload management is similarly stylish. The idea is:

• Lots of variables are available to be taken into account (e.g., user role, expected query duration, actual duration of a running query, etc.)
• SQL statements can be written against any of these variables.
• The SQL statements serve as rules to set query/task priorities.
• There seem to be a few different ways to measure priority, including explicit allocation of CPU or I/O resources, as well as more conventional “This group of queries gets higher priority than that one” kinds of metrics.
• The whole thing provides integrated workload management for queries, applications, load jobs, data redistribution, and so on.

Right now the interface is – well, you’re manipulating a SQL table. A more conventional workload management GUI is slated for the second quarter of 2010.

Discussing subjects such as mirroring and ILM (Information Lifecycle Management) with Aster can be tricky, as Aster uses the word “partition” in confusing ways. Anyhow, Aster has a few different levels of compression, and the ability to apply different levels of compression to different partitions, to change compression levels via ALTER TABLE, and to alter (presumably increase) compression on the fly when doing online backup. Aster is also part of a growing trend to eschew RAID, instead doing mirroring in its own software. (Other examples of this strategy would be Vertica, Oracle Exadata/ASM, and Teradata Fallback.) Prior to nCluster 4.0, this caused a problem, in that the block sizes for mirroring were so large as to create a lag in transactional updating. But Aster says this problem is now solved, and indeed claims that nCluster 4.0 is superior to most rivals in transactional efficiency.

And finally, while I was talking w/ Aster Data anyway, I checked up on cloud and MapReduce customer penetration. The answers were:

• Aster has two serious production cloud users, both of which have been disclosed for a while, namely:
  • ShareThis, which runs Aster nCluster on Amazon EC2
  • Didit, which runs Aster nCluster on AppNexus
• Outside of those two, Aster sees some cloud use for test, development, prototyping, etc.
• Every single Aster customer uses SQL/MapReduce — i.e., they invoke MapReduce via Aster nCluster SQL queries.
• Some of those customers use MapReduce for ETL, some use it for actual analytics.

]]> http://www.dbms2.com/2009/10/30/aster-data-application-server-ncluster/feed/ 6 http://www.dbms2.com/2009/10/27/teradatas-nebulous-cloud-strategy/ http://www.dbms2.com/2009/10/27/teradatas-nebulous-cloud-strategy/#comments Tue, 27 Oct 2009 19:41:47 +0000 Curt Monash http://www.dbms2.com/?p=1180 As the pun goes, Teradata’s cloud strategy is – well, it’s somewhat nebulous. More precisely, for the foreseeable future, Teradata’s cloud strategy is a collection of rather disjointed parts, including:

• What Teradata calls the Teradata Agile Analytics Cloud, which is a combination of previously existing technology plus one new portlet called the Teradata Elastic Mart(s) Builder. (Teradata’s Elastic Mart(s) Builder Viewpoint portlet is available for download from Teradata’s Developer Exchange.)
• Teradata Data Mover 2.0, coming “Soon”, which will ease copying (ETL without any significant “T”) from one Teradata system to another.
• Teradata Express DBMS crippleware (1 terabyte only, no production use), now available on Amazon EC2 and VMware. (I don’t see where this has much connection to the rest of Teradata’s cloud strategy, except insofar as it serves to fill out a slide.)
• Unannounced (and so far as I can tell largely undesigned) future products.

Teradata openly admits that its direction is heavily influenced by Oliver Ratzesberger at eBay. Like Teradata, Oliver and eBay favor virtual data marts over physical ones. That is, Oliver and eBay believe that the ideal scenario is that every piece of data is only stored once, in an integrated Teradata warehouse. But eBay believes and Teradata increasingly agrees that users need a great deal of control over their use of this data, including the ability to import additional data into private sandboxes, and join it to the warehouse data already there.

The Teradata Elastic Mart(s) Builder Viewpoint portlet automates the inclusion of outside data. If you’re already an authorized Teradata data warehouse user, you can fill in a very short form (three or so fields) and add authorization to import outside data, e.g. from a .CSV file. No fuss, little bother. Trivial as that sounds, when you combine it with Teradata’s pre-existing robust workload management tools, it creates a pretty good virtual data mart story.

Spinning out and maintaining consistency with physical data marts is a different matter. Teradata doesn’t seem too sure it believes in those. And while Teradata is obviously planning to increase its capability in that regard anyway, I didn’t get a lot of detail beyond the reference to Data Mover 2.0.

Related links

• My Greenplum-inspired post on the future of data marts, outlining issues in “private cloud” data warehousing.
• eBay’s “Analytics as a Service” pitch (about 1 ½ years old)
• A post by Teradata’s Dan Graham explaining the Teradata Agile Analytics Cloud and Elastic Mart(s) Builder Viewpoint portlet
• Home page and complete screen shot for the Teradata Elastic Mart(s) Builder Viewpoint portlet

• Teradata is productizing “private cloud,” under names including “Teradata Enterprise Analytics Cloud,” “Teradata Agile Analytics Cloud,” and “Teradata Elastic Mart Builder.” I.e., Teradata hopes to leapfrog Greenplum in its “Enterprise Data Cloud” strategy. This is only fair, in that Greenplum lifted the idea from Teradata and eBay in the first place. It also provides major support for what I think is an extremely sensible trend. Give or take issues of who announces and ships what a couple months before or after a competitor, my early thinking is that the main differences between Greenplum and Teradata in this regard will be:
  • Virtual as opposed to just physical data marts, based on robust workload management software. (Advantage: Teradata)
  • Pricing, deployment options. (Advantage: Greenplum)
  • Features that don’t directly relate to enterprise/private cloud. (Advantage: Either, often Teradata.)
• Teradata is generally strengthening its data movement technology, e.g. for making various appliances work in sync. I’m not too clear yet on the details of that. I think this is what Teradata’s phrase “ecosystem management” refers to.
• Teradata is (pre-)announcing – at least as a statement of direction — an appliance based on solid-state drives (SSDs). I’ve thought for a while that Teradata was a leader in thinking through the issues around solid-state memory in data warehousing, so it makes sense that they’re among the leaders in actually coming to market as well. I plan to say more after meeting with, e.g., Carson Schmidt.
• Teradata has achieved a 300%ish speed-up in geospatial processing. I gather this is largely a byproduct of the parallel analytics work Teradata did around strengthening its SAS integration. However, there don’t seem to be a lot of Teradata geospatial users yet.
• Teradata Express, Teradata’s free Windows-based crippleware, is being ported to Amazon EC2 and VMware as well. Presumably to avoid cannibalizing Teradata product sales, there are quite a few limitations on Teradata Express, including system capacity, database size, and “no production use.”
• Teradata continues to extend its optimizations to handle queries issued by business intelligence tools. Previously, the focus of what Teradata discussed in this regard was query rewrite. But soon automatic recommendation and creation of Aggregate Join Indexes – i.e.., materialized views – will be included as well.

• Some are in heavy production with Hadoop, and closely engaged with Cloudera. Others are active Hadoop users but are very secretive. Yet others signed up for initial Hadoop training last week.
• Some have Hadoop clusters in the thousands of nodes. Many have Hadoop clusters in the 50-100 node range. Others are just prototyping Hadoop use. And one seems to be “OEMing” a small Hadoop cluster in each piece of equipment sold.
• Many export data from Hadoop to a relational DBMS; many others just leave it in HDFS (Hadoop Distributed File System), e.g. with Hive as the query language, or in exactly one case Jaql.
• Some are household names, in web businesses or otherwise. Others seem to be pretty obscure.
• Industries include financial services, telecom (Asia only, and quite new), bioinformatics (and other research), intelligence, and lots of web and/or advertising/media.
• Application areas mentioned — and these overlap in some cases — include:
  • Log and/or clickstream analysis of various kinds
  • Marketing analytics
  • Machine learning and/or sophisticated data mining
  • Image processing
  • Processing of XML messages
  • Web crawling and/or text processing
  • General archiving, including of relational/tabular data, e.g. for compliance

We went over this list so quickly that we didn’t go into much detail on any one user. But one example that stood out was of an ad serving firm that had an “aggregation pipeline” consisting of 70-80 MapReduce jobs.

I also talked yesterday again w/ Omer Trajman of Vertica, who surprised me by indicating a high single-digit number of Vertica’s customers were in production with Hadoop — i.e., over 10% of Vertica’s production customers. (Vertica recently made its 100th sale, and of course not all those buyers are in production yet.) Vertica/Hadoop usage seems to have started in Vertica’s financial services stronghold — specifically in financial trading — with web analytics and the like coming on afterwards. Based on current prototyping efforts, Omer expects bioinformatics to be the third production market for Vertica/Hadoop, with telecommunications coming in fourth.

Unsurprisingly, the general Vertica/Hadoop usage model seems to be:

• Do something to the data in Hadoop
• Dump it into Vertica to be queried

What I did find surprising is that the data often isn’t reduced by this analysis, but rather exploded in size. E.g., a complete store of mortgage trading data might be a few terabytes in size, but Hadoop-based post processing can increase that by 1 or 2 orders of magnitude. (Analogies to the importance and magnitude of “cooked” data in scientific data processing come to mind.)

And finally, I talked to Aster a few days ago about the usage of its nCluster/Hadoop connector. Aster characterized Aster/Hadoop users’ Hadoop usage as being of the batch/ETL variety, which is the classic use case one concedes to Hadoop even if one believes that MapReduce should commonly be done right in the DBMS.

Related link

• An August, 2008 round-up of MapReduce applications.

• A data model based on multidimensional arrays, not sets of tuples
• A storage model based on versions and not update in place
• Built-in support for provenance (lineage), workflows, and uncertainty
• Scalability to 100s of petabytes and 1,000s of nodes with high degrees of tolerance to failures
• Support for “external” data objects so that data sets can be queried and manipulated without ever having to be loaded into the database
• Open source in order to foster a community of contributors and to insure that data is never “locked up” — a critical requirement for scientists

However:

• I think that’s a dream/wish list. A lot of good could be done without meeting each of those six requirements in full.
• I think at least some of the XLDB/SciDB leaders realize this.
• In my opinion, a highly useful subset of the dream/wish list is achievable in the reasonably-intermediate future, in either of two ways:
  • Through a Hadoop-centric open source effort, especially since HadoopDB opens up the possibility of letting DBMS creators offload MPP scaling challenges to somebody else.
  • From commercial MPP software-only (as opposed to appliance) DBMS vendors. I think they can develop the needed technology. I also think it could be in their business interest to make licensing arrangements of the sort that the scientific and research communities would need.
• Talking about “scientific” big data is unhelpfully vague. Let’s just focus on multi-dimensional measurement- or model-centric data, from disciplines such as seismology (under the Earth’s surface), climatology (over the surface), and astronomy (outer space). That would also include disciplines whose three-spatial-dimensions-plus-time data comes from inside a laboratory or other man-made environment, such as high-energy physics, fluid dynamics, and so on.
• One place in all that where there should be a commercial-company market is in oil/gas extraction. And by the way, the energy industry is increasing its uptake of data warehousing technology faster these days than any other sector I can think of, except perhaps for …
• … web companies that do log file analysis. Facebook’s log data has arrays-within-arrays reminiscent of the scientists’. eBay has been a major backer of XLDB/SciDB. It’s far from fully known yet just how much overlap there is between log-file-analyzers’ data management needs and those of big-data scientists. But there clearly are at least some commonalities.
• I don’t get the impression that scientists focused on modeling — e.g. climate-predictors — have been big participants in XLDB. That’s a pity for at least two reasons. First, modeling is at the heart of some of the most important global issues scientists address (e.g., climate change). Second, it might be an area of particularly rich overlap with commercial data management needs.

Now let’s step back and consider approximately what is meant by the requirements listed above.

• The requirements for an array structure are evidently pretty deep. You can glean some of the reasons from the scientific database use cases posted on the SciDB website. In particular:
  • Coordinate data naturally fits into arrays.
  • Coordinate data also naturally fits into geospatial ranges and the like.
  • The “grid” for the array can be imprecise — or calculated via transformation — for a whole lot of different reasons.
  • Different measurements may be available for different points in the array. (I think this may be the essence of the array-valued-arrays requirement.)
• Some reasons scientists want versioning and support for data provenance are pretty obvious — you never want to lose the record of what the instrument readings said, or ever were believed to say. But it goes further. Data is “cooked” — i.e., transformed/reduced — and stored in huge volumes. So you’d like to later on be able to go back to the raw data and re-cook it.
• The workflow requirement seems to stem in many cases from data movement needs, that in turn in some cases stem from political issues. I haven’t yet understood why workflow would actually need to be baked into a scientific DBMS.
• By the time the database management systems we’re talking about could conceivably be ready, the need will be at least in the 10s of petabytes. 100s of petabytes is a reasonable design goal.
• Not that I’ve run any numbers on the matter, but it seems plausible that query fault-tolerance will be needed, at least in some cases.
• In many sciences (astronomy seems to be an exception), the default choice is to keep data in files rather than a DBMS. For example, CERN has a 10 terabyte or so Oracle database holding just the metadata for a vastly larger collection of data files. Even if the pendulum swings toward greater use of DBMS, the ongoing need for external file access is pretty obvious.
• I suspect that the insistence on open source is part legitimate, part knee-jerk excessive.
  • “Free” is the best possible price, of course.
  • Beyond cash cost, scientists want data access to be free of licensing encumbrance. There are two main reasons. First, people might want to manage subsets or copies of data remotely from its central repository, for a variety of reasons. Not all of those reasons are easy to overcome, so any closed-source licensing would have to be very comprehensive (e.g., global or at least continent-wide “site” licensing).
  • Second, they want assurance that data will always be accessible, even if licenses expire. That seems a little overwrought. Yes, moving data from one multi-petabyte repository to another could be a bit slow. But it’s not an eventuality to panic about.
  • As for actual community development — scientists sure have a variety of exotic data management needs. But I’m not sure how much talent or resource there is among scientists to do true DBMS development (as opposed to, say, refining some UDFs). Yes, one XLDB attendee was both an astronomer and a PostgreSQL Major Contributor, but he seemed like an exception. On the other hand, it’s not entirely implausible that, in the right framework, some people with database talent could be recruited to donate some time to the general advancement of science.
• I don’t know much about management of uncertain data, and will duck that subject for now.

Related links

• Martin Kersten’s response
• Jacek Becla’s response
• Scientific data sharing

• Sybase IQ is an analytic DBMS with a columnar/column-store architecture
• Unlike most analytic DBMS, Sybase IQ has a shared-disk architecture.
• The Sybase IQ indexing story is a bit complicated, with a bunch of different index kinds. Most are focused on columns with low cardinality, and it least in some cases are a lot like bitmaps. (Sybase IQ when first introduced was a pure bitmap index product, with a single index type “Fast Project”.) But one index kind, “High Group” — designed for columns with high cardinality – is an exception to most generalities about other Sybase IQ index kinds, and instead is more akin to a b-tree.
• Unlike Vertica, Sybase stores each column of data only once. I don’t see how it would make sense to have multiple indexes on the same column, but I didn’t actually ask whether doing so is possible or common.
• Sybase estimates that Sybase IQ requires ¼ the DBA effort of, say, Oracle. (Frankly, that’s not a particularly good figure.) Obviously, this is just a broad-brush average.
• Sybase recently repurposed an acquired ETL tool to be focused on Sybase IQ. IQ of course also works with various third-party tools, certified or otherwise.
• Sybase’s Power Designer CASE (Computer-Aided Software Engineering)/database design tool works with Sybase IQ.
• Sybase is proud of Sybase IQ’s new in-database analytics capabilities, but I haven’t yet grasped what, if anything, is differentiated about them.
• Sybase has an ILM (Information Lifecycle Management) story built around the point that different columns can be stored on different kinds of media.

Highlights of the Sybase IQ compression story include:

• Sybase IQ applies compression to both columns and pages
• A (the?) major kind of column compression is called “projection” — why? — but boils down to token/dictionary compression. Tokens can be 1, 2, or 3 bytes or length – whichever is the best fit for the column’s cardinality.
• I don’t have details about the other kinds of compression.
• Data is kept compressed in memory “until the latest point possible.”

Highlights of the Sybase IQ update and load story include:

• Sybase claims that only the “High Group” index is costly to update. Specifically, “High Group” costs about as much to update as the database itself. Other indexes are fairly trivial to update. (Upon reflection, I don’t immediately see why that makes sense.)
• There’s pipelining of some sort when a High Group index is updated.
• Sybase claims that bulk loads of Sybase IQ are very fast.
• Loading Sybase IQ doesn’t block queries. Rather, Sybase IQ has some kind of versioning system in which a query just executes against older data.
• Sybase IQ updating is done in parallel. (That would be parallel among servers, of course, since Sybase IQ is shared-disk.)
• Trickle feed loading of Sybase IQ is slow. When you need to do microbatch loading with latency in the 2-15 minute range, Sybase recommends staging via an OLTP DBMS, whether from Sybase or otherwise. Sybase PowerDesigner generates scripts for this, and Sybase Replication Server helps with the execution.

Highlights of the Sybase IQ concurrency, scalability, and workload management story include:

• Sybase points out that, because of Sybase IQ’s shared-disk architecture, queries can execute on a single server in the “grid.” Thus, if you have enough cores, it can be possible to isolate long-running queries from shorter ones.
• Similarly, Sybase notes that you can meet different SLAs by putting different users’ queries on more- or less-crowded Sybase IQ servers.
• Sybase further observes that not having to move data among nodes saves Sybase IQ from a lot of overhead true MPP systems endure.
• Sybase makes the usual claim that, because Sybase IQ is so efficient, queries finish quickly, and hence there’s less stress on concurrency than one might otherwise think.
• I don’t get the sense that Sybase IQ actually boasts a lot of direct workload management features. However, there are such features in Sybase’s flagship ASE product, so hopefully adding something similar to Sybase IQ is a product future.

Related links

• Sybase IQ business notes

• Performance (of course, performance is a point of emphasis for almost any release of any analytic DBMS product), especially but not only in the areas of aggregates, ETL (Extract/Transform/Load), and UDFs.
• UDFs (User Defined Functions), especially but not only in the areas of data mining and geospatial analysis.

To put it even more concisely, the focus of Teradata 13 is on advanced analytic performance, although there of course are some enhancements in simple query performance and in analytic functionality as well.

*Teradata development chief Scott Gnau said a couple of customers have already received Teradata 13, although this was recent enough that presumably nobody has it in production. But let’s not take all that too literally, since — for example — I heard nothing about the length or breadth of the beta cycle.

As just one example, when I asked Scott what was different between Teradata 13 as it is shipping now vs. Teradata 13 as it was foreshadowed back in October, he cited:

• Improved performance
• Additional “content,” including:
  • Faster loading (sounds like an aspect of performance to me)
  • In-database data mining initiatives (these fit in both the “UDF” and “performance” buckets).

But the parts of Teradata 13 that Scott already discussed back in October, 2008 largely boil down to performance and/or UDFs as well.

Scott also foreshadowed an area of emphasis for future Teradata releases — temporal data analysis. Teradata 13 offers a new PERIOD datatype, which Scott thinks is a “sleeper” on its own for the value customers will find in it. And Scott made it clear that Teradata plans much more functionality for temporal data analysis in the future.

As I understand it, PERIOD works like this: Suppose you have a table that maintains, say, address or employment status. When you update it, you naturally create a Start_Date and End_Date for the validity of certain information. Teradata’s PERIOD datatype automagically uses this to maintain a Period where information was true, even when that period is wholly in the past. Thus when you update a row with new information, you wind up with two rows — the newly changed row, and also a second row with the old information and an effectiveness period for same.

Note: I have no further detail about Teradata’s PERIOD datatype at this time. Even what I said includes enough guesswork that there are probably at least small errors in it.

The Teradata 13 UDF, in-database data mining, and SAS integration stories seem to go something like this:

• Teradata offered UDF support in C before Teradata 13. With Teradata 13 it supports Java UDFs as well.
• Any Teradata UDF is automatically parallel, running across all nodes, etc.
• Teradata 13 cleans up a variety of UDF issues, including:
  • Allowing the use of UDFs in certain aggregates that didn’t support them before.
  • Recursion, whatever that means in this context. (Perhaps the prior point is a hint.)
  • Extended memory management/making more memory available to UDFs.
• Teradata’s work to enhance SAS integration has been focused on its general UDF framework. The memory management extensions seem to have particularly important to running SAS. (Note: That link refers to putting SAS on a “single node” in a Teradata grid. Scott gave me the impression that no such thing was possible. So I’m a bit confused. I’m also not sure it matters much.)
• Teradata expects this same general UDF framework to support integration with a variety of analytic technologies. But the only examples actually discussed were SAS and geospatial.
• Actually, we didn’t really discuss geospatial much either, so I’ll just refer you back to my October, 2008 post (already linked above) about Teradata’s geospatial datatype.

Besides UDFs, the other performance focus in Teradata 13 seems to be aggregations and OLAP. One Teradata 13 performance boost lies in aggressive query rewriting. Business intelligence tools, written to support multiple analytic DBMS (including non-current versions), can produce very messy SQL queries. Teradata 13 takes an optimizing compiler mindset to those, and in some cases can get significant speedup as a results. I get the impression there was work on other OLAP and aggregation speed-ups as well.

Also, Teradata 13 added a feature for load performance that Scott cites as being useful in the cases of heavy ETL (actually, it sounded more like ELT — Extract/Load/Transform) and OLAP aggregate-building. Namely, for the first time Teradata lets you turn off hash distribution. Teradata still wants you to hash-distribute whatever you’re going to persist to disk. But if you’re just creating a temporary table that will be dropped as soon as the load process completes, you’re now allowed to skip the hash distribution step. Scott says this can lead to >30% improvements in load performance.

Well, guess what?  In a throwaway line in a comment on Daniel Abadi’s blog, Barry Zane of ParAccel pointed out

we posted a load rate of almost 9TB/hour, which is, of course record breaking on its own

Quite right.

I hope the nonsense stops there, but I’m not optimistic …