Anyone who has met with us knows ParAccel shies away from hype.

But like many things ParAccel says, that is not true.

The latest whoppers came in the form of several customers ParAccel listed on its website who hadn’t actually bought ParAccel’s DBMS, nor even decided to do so. It is fairly common to to claim a customer win, then retract the claim due to lack of permission to disclose. But that’s not what happened in these cases. Based on emails helpfully shared by a ParAccel competitor competing in some of those accounts, it seems clear that ParAccel actually posted fabricated claims of customer wins.

Another thing that was both technically and substantively false was ParAccel’s claim to be CERTIFIED price-performance leader. Obviously, this was meant to give the impression that ParAccel had been “certified” as the leader in price/performance, when the closest thing to that that was remotely true was that ParAccel had a leading position in the category of “price/performance measurements that happen to have a certification process.” At least, that was true for a short time; then ParAccel’s certification was found to have been erroneous, and got revoked, which did not however inspire ParAccel to immediately take the claim off the front page of its website.

ParAccel’s website also reflects a lot of praise from flagship customer LatiNode. What it perhaps understandably neglects to mention is that LatiNode is in a dormant state, placed there by acquirer Elandia due to LatiNode’s criminally corrupt customer acquisition practices.

I also don’t believe ParAccel’s endlessly-repeated claim that is has never lost a benchmark on performance. However, I must in fairness note that while I’ve been given names of customers who are supposed counterexamples to this claim by somebody I trust, I’ve never been able to actually verify those supposed ParAccel losses.

• Like Infobright, Calpont’s InfiniDB is a columnar DBMS consisting of a MySQL front end and a columnar storage engine.
• Community edition InfiniDB runs on a single server.
• One of commercial/enterprise edition InfiniDB’s main claims to fame will be MPP support.
• There’s no announced time frame for commercial edition InfiniDB.
• InfiniDB’s current compression story is dictionary/token only, with decompression occurring before joins are executed. Improvement is a roadmap item.
• Indeed, InfiniDB has many roadmap items, a few of which can be found here. Also, a great overview of InfiniDB’s current state and roadmap can be found in this MySQL Performance Blog thread. (And follow the links there to find performance discussions of other free analytic DBMS.)
• One thing InfiniDB already has that is still a roadmap item for Infobright is the ability to run a query across multiple cores at once.
• One thing free InfiniDB has that Infobright only offers in its Enterprise Edition is ACID-compliant Insert/Update/Delete. (Note: I wish people would stop saying that Infobright Enterprise Edition isn’t ACID-compliant, since that point was cleared up a while ago.)
• InfiniDB has no indexes or materialized views.
• However, InfiniDB’s retrieval is expedited by something called “Extents,” which sounds a lot like Netezza’s zone maps.

Being on vacation, I’ll stop there for now. (If it weren’t for Tropical Storm/ depression Ida, I might not even be posting this much until I get back.)

General SenSage highlights include:

• SenSage used to be known as Addamark.
• SenSage used to characterize itself as being in the Security Information Management (SIM) market.
• Now SenSage characterizes itself (approximately) as selling technology built around a columnar DBMS that happens to be pretty good at log analysis, compliance, and/or archiving.
• More concisely, SenSage says it is in the event data warehouse category. (The same could arguably be said of Splunk.)
• SenSage says it has >400 paying customers, of which ~200 are direct.
• SenSage has >120 employees and, like Splunk, is profitable.
• SenSage has enjoyed >50% annual revenue growth the past four years.
• Some SenSage deals are in the multiple-million dollar range.
• A major SenSage channel partner – dozens of installations — is SAP, which resells SenSage software on HP hardware is a “Compliance Log Warehouse.”
• A hot market for SenSage is CDRs (Call Detail Records).
• SenSage says that, among analytic DBMS vendors, it competes with Oracle, IBM, Teradata, Netezza and, to some extent, Vertica and Greenplum.

Technical SenSage highlights include:

• SenSage’s core technology is an append-only columnar DBMS, with no master node.
• SenSage’s DBMS uses no indexes and requires “no” database administration.
• SenSage’s database is range-partitioned, with the range-partition key always being time.
• SenSage has something it calls SQO (Sparse Query Optimization), which sounds a lot like Netezza zone maps. SQO never yields a false negative on whether data is in a block, never yields a false positive on equality predicates, and only rarely yields a false positive on range predicates.
• SenSage’s database uses large block sizes – typically 250,000 records/block, at 200-250 bytes per record. (That’s in the range of 64 megabytes/block.)
• SenSage says its software can load 10-50,000 records/second/node. If I’m doing the arithmetic correctly, that’s roughly 7-40 gigabytes/node/hour.
• SenSage collects log data into its event data warehouse in what it characterizes as an agentless manner. Even so, it seems that for a majority of kinds of data sources one does have to write custom agents. The two other ways to get data into SenSage – and presumably most of the data volume comes through these – are:
  • File transfer in the usual way
  • syslog
• SenSage says its software can read 100s of data sources, and that this is a huge competitive advantage. I’m not totally sure how that jibes with the prior point.
• SenSage says it gets 5X compression on CDR data, 10-20X on other kinds of logs. That’s not too far off from Vertica’s compression figures.
• SenSage says that it has datatype-aware compression as well as more standard stuff, with VARCHAR compressing particularly well.
• In particular, SenSage uses both dictionary/token and delta compression.
• SenSage’s software is pretty agnostic with respect to storage kind – DAS (Direct Attached Storage), SAN (Storage-Area Network), or content-addressable. In particular, there’s only about a 4% performance hit for using content-addressable storage.
• When using WORM (Write Once Read Many) storage like EMC’s Centera, SenSage leaves record locator information behind on ordinary storage and otherwise queries the WORM storage just like it queries anything else.
• SenSage says it has been using MapReduce since “Day 1”.
• Probably not coincidentally, you can use Perl and other aggregates in SenSage SQL statements.
• Perhaps also not coincidentally, SenSage says it has a number of advanced built-in analytic functions, including some focused on sessionization.

In addition to all that, SenSage offers a built-in event processing engine, consisting of:

• A finite-state machine correlation engine.
• A proprietary event processing language.
• A GUI to “abstract” (i.e., generate?) the event processing language.

The SenSage event processing engine is used to generate alerts. Data that comes into SenSage actually is passed to two places at once, namely to both the event processing engine and the database itself.

*I gather from the company that they forgot to tell Merv pricing was NDA.

** Merv cited a price as “starting” that I believe to be top-of-the-line. No criticism of Merv is implied in that; Kickfire has not been very clear in communicating hard numbers.

All that said, if one takes Kickfire’s marketing statements literally, Kickfire list pricing is around $20-50K per terabyte for a few small, fixed, high-performance configurations. That’s all-in, for plug-and-play appliances. What’s more, that range is based on the actual published user data capacity numbers for various Kickfire models, which I think are low for several reasons:

• Kickfire doesn’t officially admit that its model with 14.4 terabytes of disk can manage more than 6 terabytes of data, even though it clearly can.
• Actually, those 14.4 terabytes of disk can be increased or lowered as you choose.
• The basic compression figures implied in those calculations seem conservative.
• Compression figures are a lot more conservative yet, in that Kickfire assumes you’ll have a lot of actual indexes on your data. I’m not sure that’s necessary for most workloads.

Highlights include:

• Column orientation is a special case of what Greenplum is calling Polymorphic Data Storage.*
• As per product management chief Ben Werther’s blog post, what Greenplum’s polymorphic data storage boils down to is that you can store different tables in different storage paradigms. This is transparent to the SQL or any other API; it’s just a performance choice.
• Indeed, Greenplum lets you store different partitions of the same table in different storage and/or compression schemes. So Greenplum now has a kind of ILM (Information Lifecycle Management) story, although it doesn’t offer the faster vs. cheaper storage media differentiation options of Sybase IQ or Vertica.
• Greenplum now has, depending on how one counts, three or four main types of table:
  • Traditional PostgreSQL, which has been available since Day One
    
  • Row-oriented append-only (compressible and scan-optimized), available since Greenplum 3.2 (July, 2008)
  • Columnar append-only (new in Greenplum 3.3.4, shipping now)
  • External, in which Greenplum treats something external – in a relational DBMS or otherwise – as if it were a Greenplum table
• Greenplum offers multiple versions of LZ (Lempel-Ziv) and gzip compression, any of which you can choose on a table-by-table or partition-by-partition basis.
• Greenplum offers the same compression algorithms for both row-oriented and column-oriented tables.
• Greenplum says that compression is typically at least 50% better (i.e., to 2/3 as much space) in columnar vs. row storage, for the same algorithm.
• Just as it doesn’t offer columnar-specific compression algorithms, Greenplum also doesn’t sport other columnar features Daniel loves, such as in-memory compression or late materialization. (But then, VectorWise doesn’t do in-memory compression either, and Daniel likes VectorWise.)
• All the Greenplum choices I’ve mentioned have to be made manually by DBAs.
• Similarly, I doubt Greenplum can match Vertica’s engineering for getting updates and trickle feeds quickly into a column store – a traditional columnar Achilles heel that Vertica has invested a lot of effort to circumvent.

*The term “polymorphic” is somewhat, shall we say, overloaded these days.

*Guys like Greg Rahn and Kevin Closson are huge assets to Oracle, which is absurdly and self-defeatingly unhelpful through conventional public/analyst relations channels.

Six key tweets by Greg Rahn said:

I think the HCC 10x compression is a slideware (common) number. Personally I’ve seen it in the 12-17x range on customer data…

This was on a dimensional model. Can’t speak to the specific industry. I do believe Oracle is working on getting industry #s.

As far as I know, Exadata HCC uses a superset of compression algorithms that the commonly known column stores use…

…and it doesn’t require the compression type be in the DDL like Vertica or ParAccel. It figures out the best algo to apply.

The compression number I quoted is sizeof(uncompressed)/sizeof(hcc compressed). No indexes were used in this case.

Exadata HCC is applicable for bulk loaded (fact table) data, so a significant portion (size wise) of most DWs.

Summing up, that seems to say:

• Oracle claims 12-17X compression on a kind of data similar to that on which Vertica — which also uses 10X as a single-point overall compression marketing estimate where needed — claims 20X.
• Oracle selects compression algorithms automagically.
• Oracle’s compression doesn’t quite apply to all the data. Actually, this may be more of an issue for the caching benefits of compression than for the I/O or disk storage gains. (If you join a retail transaction fact table to a customer dimension table, and you have a lot of customers, fitting the uncompressed customer table into RAM could be problematic.)

Omer and I happened to have a call scheduled to discuss MapReduce yesterday evening, but wound up using most of the time to talk about Vertica’s compression and physical layout features instead. Highlights included:

• Greg, like many Vertica competitors, was wrong about Vertica requiring manual, low-level DDL (Data Description Language) for — well, for much of anything. Vertica does all that automatically, at least in theory, and suggests that in real life you can indeed often get by without manual intervention.
• Vertica can do trickle feeds into its compressed columnar storage. Greg seemed to suggest Oracle Exadata can not. (However, I won’t be surprised if, when his comments are expanded to more than 140 characters, he winds up saying the opposite. )
• Omer characterized the lowest latency with which you can get data into Vertica and have it be available for query as “seconds”, vs. “minutes” for other columnar vendors.
• Vertica recommends often keeping multiple copies of a column, for high availability and/or performance. This is not directly reflected in compression estimates. In particular, if you’re going to keep redundant copies of data for data-safety reasons anyway, Vertica recommends that you:
  • Run queries against more than one copy of the data, for performance/throughput.
  • Store different copies of the columns in different sort orders — e.g., according to different likely join keys — so that the copies are optimized for performance on different classes of queries.
• Vertica doesn’t have indexes.
• Vertica sorts columns on ingest. This sorting is, of course, commonly based on attributes from columns other than the one being sorted. Even so, Omer maintains that sorting helps compression, because of the correlation between columns. Examples (and I didn’t get these all from him) might include:
  • City/postal code
  • Customer_ID/store location
  • Customer_ID/product_ID
  • Product_ID/price
• Vertica, based on the recent introduction of FlexStore, has an ILM (Information Lifecycle Management) story much like Sybase IQ’s. E.g., you can keep different data ranges for different columns on fast storage, while the rest of the data is relegated to slower/cheaper equipment.

Analyzing Oracle Exadata pricing is always harder than one would first think. But I’ve finally gotten around to doing an Oracle Exadata 2 pricing spreadsheet. The main takeaways are:

• If we believe Oracle’s claims of 10X compression, Exadata 2 costs more per terabyte of user data than Netezza TwinFin — $22-26K/TB vs. TwinFin’s <$20K — but less than the Teradata 2550.
• These figures are highly sensitive to assumptions about Oracle’s hybrid columnar compression.
• Similarly, if Netezza or Teradata were to significantly upgrade their own compression, the price comparison would look quite different.
• Options such as Data Mining or Oracle Spatial add 12% or so each to Exadata’s total system price.

Longer version

When Oracle introduced Exadata last year it was, well, expensive. Exadata 2 has now been announced, and it is significantly cheaper than Exadata 1 per terabyte of user data, based on:

• Similar overall pricing
• Twice the disk capacity
• Better compression

Compression is the big question mark. Row-based DBMS vendors have traditionally been, if anything, conservative in their compression claims, although Netezza recently went with a not-sandbagged 2.25X compression estimate to get below the $20K/terabyte price point. Columnar software vendors have tended to be more aggressive, with figures of 10X or more casually thrown around, or 40-50X for archival storage. But since columnar vendors sell mainly on a software-only basis, those claims haven’t generally shown up in per-terabyte total system cost comparisons, which most commonly focus on data warehouse appliance product lines.*

*Kickfire, the one columnar pure-play appliance vendor, hast to date been quite conservative in its compression marketing claims.

Oracle, however, recently announced a feature called hybrid columnar compression, and is now making compression claims with the usual columnar grandiosity. Oracle’s story is 10X compression, and they’re sticking to it, perhaps because 10 is such a nice round number.* Since Oracle hybrid columnar compression is part of 11g Release 2, and isn’t Exadata-specific, wWe can hope to eventually get a sense from the field of what levels of compression are actually realistic. (Edit: Actually, it seems that hybrid columnar compression only works with Exadata, at least at this time.) But for now, we don’t have much to go on except Oracle’s claims.

*Greg Rahn of Oracle tweeted me yesterday that one customer is getting 12-17X compression on “dimensional model” data. That sounds comparable to Vertica’s claim of 20X on “marketing analytics” and 30X on “consumer data” datasets.

Based on 10X compression (vs. Netezza’s 2.25X and Teradata’s lower figure), Oracle Exadata 2 is somewhat more expensive than Netezza TwinFin, and significantly cheaper than Teradata’s 2550. Specifically, Oracle Exadata 2 comes in around $22K/TB of user data for a full rack, and $26K/TB for a quarter rack, which is the Exadata 2 configuration more comparable to a TwinFin rack in user data capacity. This is if you look at the Exadata hardware version that uses 600 GB SAS drives (vs. 1 TB SAS drives for TwinFin and 300 GB SAS drives for Teradata). With 2 TB SATA drives, at the same system pricing, Exadata prices are 70% lower, getting down to $6K/TB for a full rack. You can see how I got these figures on my Oracle Exadata 2 pricing spreadsheet linked above.

Obviously, price/terabyte is just one metric. Throughput is often even more important, but also is a lot harder to quantify simply. Oracle Exadata 2 offers more raw I/O than Netezza TwinFin. Netezza TwinFin, with its FPGA-based pipelining, probably has more processing oomph than Oracle Exadata. Oracle’s compression could lead to better use of RAM cache. And so it goes.

Meanwhile, two factors that in my opinion don’t matter much to the analysis are:

• The re-usability of Oracle licenses on other hardware. Most of Exadata’s cost is either for hardware, or for server software that’s priced on a per-core basis. Neither of those is going to manage much (or any) more data three years from now than it can today.
• What Oracle claims as pricing metrics. Oracle’s comments on Exadata pricing generally sow confusion, which is why I do my own spreadsheets.

Related links

• The hunt for Oracle Exadata production references (but hopefully some will be revealed at Oracle Open World)
• A favorable rumor about Exadata sales
• Issues in integrating OLTP and data warehousing in a single system
• James Kobielus of Forrester tweets that there are “plenty” of pleased Exadata users
• More on Oracle Exadata hybrid columnar compression

]]> http://www.dbms2.com/2009/10/05/oracle-exadata-2-capacity-pricing/feed/ 13 http://www.dbms2.com/2009/09/21/notes-on-the-oracle-database-11g-release-2-white-paper/ http://www.dbms2.com/2009/09/21/notes-on-the-oracle-database-11g-release-2-white-paper/#comments Mon, 21 Sep 2009 17:12:33 +0000 Curt Monash http://www.dbms2.com/?p=923

The Oracle Database 11g Release 2 white paper I cited a couple of weeks ago has evidently been edited, given that a phrase I quoted last month is no longer to be found. Anyhow, here are some quotes from and comments on what evidently is the latest version.

The In-Memory Database Cache (IMDB Cache) option of Oracle Database 11g Release 2, allows data to be cached and processed in the memory of the applications themselves, off-loading the data processing to middle tier resources. Any network latency between the middle tier and the back-end database is removed from the transaction path, with the result that individual transactions can often be executed up to 10 times faster. This is particularly useful where very high rates of transaction processing is required, such as those found under market trading systems, Telco switching systems, and Real Time manufacturing environments. All data in the middle tier is fully protected through local recovery, and asynchronous posting to the back end Oracle Database. With Oracle Database 11g Release 2, the ability to transparently deploy IMDB Cache with existing Oracle applications becomes much easier – with common data types, SQL and PL/SQL support, and native support for the Oracle Call Interface (OCI).

At a guess, this sounds like it’s based on Oracle’s TimesTen acquisition.

Oracle Database 11g Release 2 adds further optimizations, including capabilities to automatically determine the most optimal degree of parallelization for a query, based on available resources. With this comes automated parallel statement queuing, where the database determines that, based on current resource availability, it is more effective to queue a query for later execution once required resources have freed up.

Sounds like a kind of automatic workload management — i.e., the kind of optimization vendors of mature products get around to putting into their systems. It does not sound like query pipelining, however.

Oracle Database 11g Release 2 will automatically distribute a large compressed table (or a smaller non-compressed table), into the available memory across all the servers in the Grid, and will then localize parallel query processing to the data in memory on the individual nodes. This dramatically improves query performance, and is especially useful where large tables can be entirely compressed into the available memory using compression capabilities.

So Oracle caches compressed data. Not stated is which compression techniques are covered.

Each Exadata Storage Server stores up to 7 Terabyte [sic] of uncompressed user data, and also comes enabled with 384 GB of solid-state Flash cache. This Flash Cache automatically caches active data of the magnetic disks in the Oracle Exadata Storage Server, delivering a 10x performance gain for read and write operations under OLTP applications.

Sounds like the Flash memory is positioned for OLTP use.

In the past, Database Administrators and System Administrators have spent a great deal of time determining to how best place data across these disk arrays, to get maximum performance and availability. The best procedure for data placement is to simply Stripe And Mirror Everything; stripe data blocks equally across all disks in an array, and then mirror the blocks on at least two disks. This approach provides the perfect balance between performance, disk utilization, and ease of use.

This is a big part of what could be called the “Administering Oracle doesn’t suck nearly as badly as it used to” pitch. (Mitchell Kertzman, who was Sybase CEO after the mid-1990s meltdown, told me his motto was “We suck less every day.” But I digress …)

Automatic Storage Management (ASM), a feature of Oracle Database 11g automates the striping and mirroring of database without the need to purchase third party volume management software. As data volumes increase, additional disks can be added, and ASM will automatically restripe and rebalance the data across available disks to ensure optimal performance. Similarly, disks that report errors can be removed from the disk array, and ASM will re-adjust accordingly.

I.e., you can add nodes without taking the system down. That’s becoming a pretty standard feature for serious parallel DBMS.

Oracle Database 11g Release 2 improves ASM in significant areas. New intelligent data placement capabilities store infrequently accessed data on the inner rings of the physical disks, while frequently accessed data is placed on the outer rings, offering better performance optimization.

Also pretty standard.

Oracle has been enhancing partitioning capabilities for over ten years. Oracle Partitioning, an option of Oracle Database 11g Release 2, allows very large tables (and their associated indexes) to be partitioned into smaller, more manageable units, providing a “divide and conquer” approach to very large database management. Partitioning also improves performance, as the optimizer will prune queries to only use the relevant partitions of a table or index in a lookup. Oracle Database 11g Release 2 provides multiple methods for partitioning data, and also allows different levels of partitioning on the same table, so that a single partitioning strategy can be used to improve both performance and manageability.

Even better might be a system that doesn’t lean heavily on complex partitioning to achieve good performance.

Oracle Partitioning can also manage the lifecycle of information. Typically, all databases have active data – the information being processed this month or quarter, and historical data that is primarily read-only. Organizations can take advantage of the inherent lifecycle of data to implement a multi-tiered storage solution and lower their overall storage costs. For example, a large table within an order-entry system could contain all the orders processed in the last 7 years. Oracle Partitioning can be used to set up monthly partitions, with the current last four months of order data partitioned onto a high-end storage array, with all the other partitions placed on a lower-cost storage solution, often 2-3 times less cost than the high end storage environment.

This is becoming a standard feature for any parallel DBMS that can support multiple kinds of storage in one system.

Oracle Database 11g also provides advanced compression techniques to further reduce storage requirements. Using Oracle Advanced Compression, an option to Oracle Database 11g, all data in a table can be compressed using a continuous table compression capability that achieves a 2-4 times compression ratio with little performance impact on OLTP or Data Warehousing workloads. This compression technology replaces duplicate values in a table with a single value, and continuously adapts to data changes over time, so compression ratios are always maintained.

Sounds like dictionary/token compression.

With Oracle Database 11g Release 2, the Exadata Storage Servers in the Sun Oracle Database Machine also enable new hybrid columnar compression technology that provides up to a 10 times compression ratio, with corresponding improvements in query performance. And, for pure historical data, a new archival level of hybrid columnar compression can be used that provides up to 50 times compression ratios.

I thought they said 40X before. But even if my memory isn’t playing tricks regarding that, single-point compression ratio estimates are always very approximate.

Any hardware component in an Oracle Grid can be dynamically added or removed as required. Disks can be added or removed online with ASM, with the data automatically rebalanced across the new disk infrastructure. Additional servers can also be easily added or removed to a Real Application Cluster with users connected to these nodes rebalanced across the infrastructure. This ability to migrate users from one server to another in a RAC cluster also enables rolling patching of the database software. If a patch needs to be applied, then a server can be removed from the cluster, patched, and then put back into the cluster. The same operation can be repeated for the next server in the cluster, and so on.

Nice. And the paper goes on in that vein for quite a while.

Oracle Total Recall, an option to Oracle Database 11g, provides a solution for the retention of historical information. With Oracle Total Recall, all changes made to data are kept to provide a complete change history of information. This means that auditors can not only see who did what when, but they can also see what the actual information was at the time – something that previously has only be [sic] available by building into the application, or by expensive backup retention policies.

Timestamping/time-travel/whatever is increasingly becoming a standard feature as well, especially given the number of PostgreSQL-based DBMS on the market.

New internal control requirements found in regulations can be difficult and expensive to implement in an environment with multiple applications. Oracle Database Vault, an option to Oracle Database 11g, allows access controls to be transparently applied underneath existing applications. Users can be prevented from accessing specific application data, or from accessing the database outside of normal hours; separation-of-duty requirements can be enforced for different Database Administrators without a costly least privilege exercise. And Oracle Advanced Security, an option to Oracle Database 11g, can be used to transparently encrypt data at all levels – data in transit on the network; data at rest on physical storage and in backups. Similarly, the Data Masking pack can be used to obfuscate data as it moves from production to development, reducing the potential violation of privacy regulations or risking sensitive data leaks.

Oracle is the gold standard in database security.

Oracle’s self-management approach takes two tacks. Firstly, wherever possible, repeatable, labor intensive and error prone tasks that can be fully automated in the database have been. For example, Storage Management, Memory Management, Statistics collection, Backup and Recovery, and SQL Tuning have all been automated. Secondly, where operations cannot be fully automated, intelligent advisors are built into the database to mentor Database Administrators on how to get the best out of their systems. Advisors are provided for Configuration Management, Patching, Indexing, Partitioning, Performance Diagnostics, Data Recovery, and, new in Oracle Database 11g Release 2, Compression and Maximum Availability.

And boy are they needed.

Recent studies performed by an independent research company shows that Database Administrators can expect to spend 26% less time managing their 11g environments over their 10g environments, and as much as 50% when compared to older Oracle9i deployments.

50% of way too much is still way too much.

The word hybrid is important.

Rows are still used. They are stored in an object called a Compression Unit. Compression Units can span multiple blocks. Like values are stored in the compression unit with metadata that maps back to the rows.

So, “hybrid” is the word. But, none of that matters as much as the effectiveness. This form of compression is extremely effective.

That sounds a whole lot like PAX. Specifically, in Oracle’s case I would guess “hybrid columnar compression” provides the compression benefits of column stores, but not column stores’ I/O benefits, and also not any kind of in-memory compression.

*Actually, Oracle has indicated to me multiple times that the reason is I won’t let Oracle review what I write before I publish it. My stance is that such “review” is an extremely time-wasting courtesy, in which one spends a lot of time diplomatically explaining to a vendor that, contrary to what it hopes, one really does know the difference between marketing puffery and sober fact.  I rarely do white paper projects any more, notwithstanding that my fee for those now exceeds $2,000/page. I’m not about to go through the “review” hassle for something I write for free, about a vendor who isn’t otherwise a paying client.

• Sybase IQ is an analytic DBMS with a columnar/column-store architecture
• Unlike most analytic DBMS, Sybase IQ has a shared-disk architecture.
• The Sybase IQ indexing story is a bit complicated, with a bunch of different index kinds. Most are focused on columns with low cardinality, and it least in some cases are a lot like bitmaps. (Sybase IQ when first introduced was a pure bitmap index product, with a single index type “Fast Project”.) But one index kind, “High Group” — designed for columns with high cardinality – is an exception to most generalities about other Sybase IQ index kinds, and instead is more akin to a b-tree.
• Unlike Vertica, Sybase stores each column of data only once. I don’t see how it would make sense to have multiple indexes on the same column, but I didn’t actually ask whether doing so is possible or common.
• Sybase estimates that Sybase IQ requires ¼ the DBA effort of, say, Oracle. (Frankly, that’s not a particularly good figure.) Obviously, this is just a broad-brush average.
• Sybase recently repurposed an acquired ETL tool to be focused on Sybase IQ. IQ of course also works with various third-party tools, certified or otherwise.
• Sybase’s Power Designer CASE (Computer-Aided Software Engineering)/database design tool works with Sybase IQ.
• Sybase is proud of Sybase IQ’s new in-database analytics capabilities, but I haven’t yet grasped what, if anything, is differentiated about them.
• Sybase has an ILM (Information Lifecycle Management) story built around the point that different columns can be stored on different kinds of media.

Highlights of the Sybase IQ compression story include:

• Sybase IQ applies compression to both columns and pages
• A (the?) major kind of column compression is called “projection” — why? — but boils down to token/dictionary compression. Tokens can be 1, 2, or 3 bytes or length – whichever is the best fit for the column’s cardinality.
• I don’t have details about the other kinds of compression.
• Data is kept compressed in memory “until the latest point possible.”

Highlights of the Sybase IQ update and load story include:

• Sybase claims that only the “High Group” index is costly to update. Specifically, “High Group” costs about as much to update as the database itself. Other indexes are fairly trivial to update. (Upon reflection, I don’t immediately see why that makes sense.)
• There’s pipelining of some sort when a High Group index is updated.
• Sybase claims that bulk loads of Sybase IQ are very fast.
• Loading Sybase IQ doesn’t block queries. Rather, Sybase IQ has some kind of versioning system in which a query just executes against older data.
• Sybase IQ updating is done in parallel. (That would be parallel among servers, of course, since Sybase IQ is shared-disk.)
• Trickle feed loading of Sybase IQ is slow. When you need to do microbatch loading with latency in the 2-15 minute range, Sybase recommends staging via an OLTP DBMS, whether from Sybase or otherwise. Sybase PowerDesigner generates scripts for this, and Sybase Replication Server helps with the execution.

Highlights of the Sybase IQ concurrency, scalability, and workload management story include:

• Sybase points out that, because of Sybase IQ’s shared-disk architecture, queries can execute on a single server in the “grid.” Thus, if you have enough cores, it can be possible to isolate long-running queries from shorter ones.
• Similarly, Sybase notes that you can meet different SLAs by putting different users’ queries on more- or less-crowded Sybase IQ servers.
• Sybase further observes that not having to move data among nodes saves Sybase IQ from a lot of overhead true MPP systems endure.
• Sybase makes the usual claim that, because Sybase IQ is so efficient, queries finish quickly, and hence there’s less stress on concurrency than one might otherwise think.
• I don’t get the sense that Sybase IQ actually boasts a lot of direct workload management features. However, there are such features in Sybase’s flagship ASE product, so hopefully adding something similar to Sybase IQ is a product future.

Related links

• Sybase IQ business notes