• You really, really want to have multiple data streams coming out of temp space, as close to simultaneously as possible.
• The storage performance characteristics of such a workload are more reminiscent of “random” than “sequential” I/O.

If everybody else is cool with it too, I can live with that.

Meanwhile, I talked again with Tim Vincent of IBM this afternoon. Tim endorsed the temp space/Flash fit, but with a different emphasis, which upon review I find I don’t really understand. The idea is:

• Analytic DBMS processing generally stresses reads over writes.
• Temp space is an exception — read and write use of temp space is pretty balanced. (You spool data out once, you read it back in once, and that’s the end of that; next time it will be overwritten.)

My problem with that is: Flash typically has lower write than read IOPS (I/O per second), so being (relatively) write-intensive would, to a first approximation, seem if anything to disfavor a workload for Flash.

On the plus side, I was reminded of something I should have noted when I wrote about DB2 compression before:

Much like Vertica, DB2 operates on compressed data all the way through, including in temp space.

• Technology it already has released*, but has not published any reference architectures for
• A Barney partnership**

In other words, Vertica has succumbed to the common delusion that it’s a good idea to put out half-baked press releases the week of TDWI conferences. But if we look past that kind of all-too-common nonsense, Vertica is highlighting an interesting technical story, about how the analytic DBMS industry can exploit solid-state memory technology.

*Upgrades to Vertica FlexStore to handle Flash memory, actually released as part of Vertica 4.0

** With Fusion I/O

To set the context, let’s recall a few points I’ve noted in the past:

• Solid-state memory’s price/throughput tradeoffs obviously make it the future of database storage.
• The Flash future is coming soon, in part because Flash’s propensity to wear out is overstated. This is especially true in the case of modern analytic DBMS, which tend to write to blocks all at once, and most particularly the case for append-only systems such as Vertica.
• Being able to intelligently split databases among various cost tiers of storage – e.g. Flash and disk – makes a whole lot of sense.

Taken together, those points tell us:

For optimal price/performance, analytic DBMS should support databases that run part on Flash, part on disk.

While all this is a future for some other analytic DBMS vendors, Vertica is shipping it today.* What’s more, three aspects of Vertica’s architecture make it particularly well-suited for hybrid Flash/disk storage, in each case for a similar reason – you can get most of the performance benefit of all-Flash for a relatively low actual investment in Flash chips: 

• Vertica lets you split tables by column, and Vertica FlexStore is versatile enough to let you put only the most-used columns in Flash. (Vertica offers a figure that 85% of usage calls on only 15% of columns, but I don’t know how rigorously grounded those numbers are.)
• To the extent that Vertica data is more compressed than many of Vertica’s competitors’ (which it probably is, debates over the magnitude of Vertica’s advantage notwithstanding), the total storage-hardware cost of sticking stuff in Flash is less when you use Vertica than with other systems.
• Vertica has relatively less need for temp space than some other systems. (Vertica uses figures of <20% of total storage, vs. 30%+ for some other systems.) If you want to use Flash for temp space, so as to accelerate your toughest queries, that can save you some cash …
• … and by the way, temp space is an especially good use of Flash, because temp space is accessed in a less sequential manner than data storage is.

The least obvious of those points are about temp space; I only understood the particulars when Vertica development chief Shilpa Lawande explained them to me Thursday.

* At least in theory; customer adoption may be a different matter.

But before drilling down on temp space, let me first note that there’s one offsetting factor to all those “We need somewhat less Flash than the other guys” Vertica advantages. Like all serious databases, a Vertica installation keeps two or more copies of all data, to that there’s no storage single point of failure. In a flexible system like Vertica, you can put one copy on Flash and one on disk. But if you do that in Vertica, you forgo fully exploiting one possible benefit of Vertica’s architecture – the ability to store different copies of a column in different orders, which are beneficial for accelerating different groups of queries.*

*More precisely, you don’t get the full benefits of Flash acceleration for every query touching those columns.

OK. Back to temp space. There are four kinds of things you can put in storage if you’re running a database management system:

• The software itself.
• Persistent data. (I.e., tables, if the DBMS you’re running is relational.)
• Metadata, especially the kind that lets you find data – indexes, zone maps, catalogs, etc.
• Temporary data constructs built as part of, say, a sort-merge join. These, by definition, are what populate temp space.

Just to be clear, those constructs are NOT temporary tables of the sort created by, say, Microstrategy; such tables are handled like any other data. Rather, they are ephemeral creations and, so far as I can tell, not tables at all.

Vertica offered two theories as to why its DBMS requires less temp space than competitors do:

• To the extent data is decompressed before being operated on in memory by the DBMS, that decompression would of course also apply to temp space as well. Vertica prides itself on keeping data compressed all the way through, and seems to get away with smaller temp space allocations as a benefit.
• Since Vertica can store columns in expedient sort orders, it does less sorting overall, and sorting is a big use of temp space.

Obviously, no matter which DBMS you use, the amount of temp space you need is surely workload-dependent. Even so, Vertica’s claim to something of an advantage seems legit.

Truth be told, I’m not convinced the savings involved are great enough to matter a whole lot – but it’s a fun subject to think through.

And finally: One of my biggest surprises since starting to look at analytic-DBMS-on-Flash has been the centrality of temp space. Talking to Vertica Thursday, I finally uncovered a key reason why: Temp space tends to be accessed via multiple streams of data at once. I’m still struggling with WHY that is true, with two reasons suggested being:

• Temp space can be accessed by multiple operations at once. (But isn’t that also true of the rest of storage?)
• Merge sorts, a common use of temp space, read multiple streams of data. (Couldn’t you tweak your software to make that not be true?)

But if we grant that temp space naturally is accessed in multiple places at once – well, that’s a lot like random I/O, and if you’re doing a lot of random reads, you’d love to use something other than spinning disk.

]]> http://www.dbms2.com/2010/08/16/vertica-flash-temp-space/feed/ 10 http://www.dbms2.com/2010/06/21/netezza-ibm-db2-compression/ http://www.dbms2.com/2010/06/21/netezza-ibm-db2-compression/#comments Mon, 21 Jun 2010 12:05:47 +0000 Curt Monash http://www.dbms2.com/?p=2320 Thursday, I spent 3 ½ hours talking with 10 of Netezza’s more senior engineers. Friday, I talked for 1 ½ hours with IBM Fellow and DB2 Chief Architect Tim Vincent, and we agreed we needed at least 2 hours more. In both cases, the compression part of the discussion seems like a good candidate to split out into a separate post. So here goes.

When you sell a row-based DBMS, as Netezza and IBM do, there are a couple of approaches you can take to compression. First, you can compress the blocks of rows that your DBMS naturally stores. Second, you can compress the data in a column-aware way. Both Netezza and IBM have chosen completely column-oriented compression, with no block-based techniques entering the picture to my knowledge. But that’s about as far as the similarity between Netezza and IBM compression goes. 

IBM’s basic DB2 compression strategy is remarkably simple. In every table (not column) – or in each range partition in a range-partitioned table — the 4096 most common* values are identified; these are all encoded into 12-bit strings. And that’s that. This has been happening since DB2 9.1, released 4 ½ years ago. DB2’s compression persists through logs, buffer pools (i.e., RAM cache), and so on. In DB2 9.7, the most recent release, IBM extended the use of the compression to a few areas it hadn’t stretched before, such as log-based replication, native XML, or CLOBs (Character Large OBjects) that happen not to be too big.

*Actually, I’d presume it’s not exactly the “most common”; there surely is some minimum length of a value to be encoded, or some bias toward length. Also, the determination of what to encode is probably a little imprecise. E.g., I forgot to ask whether the choice of values ever changes as data got updated.

The sophisticated part of DB2’s simple compression strategy is its breadth of applicability; DB2 compression can apply to:

• Values in columns (numeric, character, whatever)
• Substrings of values in columns
• Groups of columns (e.g., city/state/zip code)

Except for the 4096 values limit, that sounds at least as flexible as the Rainstor/Clearpace compression approach.

Netezza, unlike IBM, takes a grab-bag approach to compression – try out a bunch of techniques, see which work best, and incorporate those in the product. Netezza first introduced compression a couple of years ago, for numeric columns only, especially integer.  Techniques  used in Netezza numeric compression include but are not limited to:

• Delta compression, wherein you store the increment between a value and its predecessor rather than a whole new value.
• Ways of indicating that a value or increment was just the same as in the row before.

This was via something called Compress Engine,* now being renamed to Compress Engine 1. Netezza’s new Compress Engine 2 improves on what Netezza did in Compress Engine 1 for numeric data, most notably by trimming away excess field length. (Netezza says it got 28% better compression on a test data set with almost no character strings, primarily from that enhancement.) Further, Netezza Compress Engine 2 adds new compression techniques, allowing it to handle VARCHAR – i.e. character strings — as well.

*Fortunately, the original name or at least description of “Compiled Tables” is retreating ever more from view.

Netezza’s Compress Engine 2 has two ways to compress character fields/text strings – prefix compression and Huffman coding. By way of contrast, Netezza tested suffix compression and decided it wasn’t beneficial enough to bother messing with.

• The idea behind prefix compression is that if two strings start with the same characters, for the second one you only have to record the part that’s different. Prefix compression has a lot of the same merits as delta compression; like delta compression, it works best on sorted columns. (An example of where prefix compression makes obvious sense is URLs, which tend to all start in similar ways.)
• In Netezza’s version of Huffman coding, the alphabet is encoded symbol-by-symbol, with more common characters getting codes of shorter length. These codes are chosen on a column-by-column basis. (I presume the “/” character gets shorter code in a URL column than it would, for example, in one that stored addresses.)

While I didn’t ask explicitly, it seems pretty obvious that Compress Engine 2’s functionality is a strict superset of Compress Engine 1’s. Netezza is going to run Compress Engines 1 and 2 side by side, but expects pages to move from Compress Engine 1’s purview to Compress Engine 2’s as part of the new “table grooming” process.

Related links

• IBM kindly permitted me to post some of its slides in the area of compression
• Microsoft SQL Server seems to rely on prefix and dictionary compression

• Netezza planned to be “generous” in its original TwinFin FPGA capacity, anticipating software upgrades like the ones it’s introducing now. It is satisfied that this strategy worked. More on this below.
• The same surely applies to CPU.
• What’s more, I get the sense that the CPU turned out in practice to be even more over-provisioned than they anticipated …
• … at least when one just considers Netezza’s base NPS software.
• However, I suspect that if the advanced analytics capability takes off, Netezza will determine that more CPU is always better.
• And by the way, NEC is making versions of Netezza appliances with more advanced chips than Netezza is. So if anybody should really, really need more CPU in their Netezza boxes, there’s a very straightforward way to make that happen. (And if there were nontrivial demand for that, appropriate support plans could surely be structured.)
• Everybody needs to be careful about RAM. Netezza is surely no exception.

The major parts of Netezza’s FPGA software are:

• Compress Engine 2. This is Netezza’s new way of doing compression.
• Compress Engine 1. This is Netezza’s old way of doing compression. It is being kept around so that existing Netezza tables don’t suddenly have to be changed or reloaded.
• Project Engine. Guess what this does.
• Restrict Engine. Ditto.
• Visibility Engine. This enforces ACID and handles row-level security. It is “sort of a corner of” the Restrict Engine (Actually, Netezza seems to waver as to whether to describe “Restrict” and “Visibility” as being two engines or one.)
• Miscellaneous plumbing.

If I understood correctly, each Netezza FPGA has two each of the engines in parallel.

Related link

• An August, 2009 post on what Netezza does in its FPGA

*Or, as it now appears to be called, Qlik Technologies.

Let’s start with some quick reminders:

• QlikTech makes QlikView, a widely popular business intelligence (BI) tool suite.
• QlikView is distinguished by the flexibility of navigation through its user interface.
• To support this flexibility, QlikView preloads all data you might want to query into memory.

Let’s also dispose of one confusion right up front, namely QlikTech’s use of the word associative: 

• Notwithstanding QlikTech’s repeated use of phrases like “QlikView’s unique, patented in-memory associative technology,” there is nothing “associative” about QlikView’s data structures.
• Rather, “associative” is a term that can reasonably be used to describe the functionality of QlikView’s user interface. In particular, QlikView can “associate” over fields that have the same name, in that it makes it easy for users to join across them.

With that out of the way, let’s turn to some highlights of QlikView’s underlying technology.

• For the most part, QlikView’s in-memory data structures are quite simple. In particular:
  • QlikView data is stored in a straightforward tabular format.
  • QlikView data is compressed via what QlikTech calls a “symbol table,” but I generally call “dictionary” or “token” compression.
  • QlikView typically gets at its data via scans. There is very little in the way of precomputed aggregates, indexes, and the like. Of course, if the selection happens to be in line with the order in which the records are sorted, you can get great selectivity in a scan.
  • One advantage of doing token compression is that all the fields in a column wind up being the same length. Thus, QlikView holds its data in nice arrays, so the addresses of individual rows can often be easily calculated.
• To get its UI flexibility, QlikView implicitly assumes a star/snowflake schema. That is, there should be no more and no less than one possible join path between any pair of tables. In some cases, this means one will want to rename fields as part of QlikView load scripts. For example,
  • If two keys are meant to be joined on, you might want to give them the same name.
  • If two columns have the same name and mean different things (e.g., different kinds of dates), you can give them different names.
  • You can mark which columns you do or don’t want to have “qualified” names – i.e., table-specific modifications that force the names to be unique.
• QlikView is designed for gigabytes-scale databases. (More precisely, it’s constrained by how much RAM you can address in a single box, and that’s how the numbers currently work out.) In particular:
  • QlikTech recommends 2-4 gigabytes of compressed data per core. QlikTech says 10X is a good rule of thumb for compression, although it sounded like that’s a little (not a lot) on the high side when compared simply to raw data.
  • QlikTech further recommends RAM amounting to another 10% of data size be set aside for each concurrent user (e.g., for cache). However, Hakan said that’s really too pessimistic, and in most cases 5% would suffice.
  • Bottom line: QlikView “comfortably” handles databases with 10-20 gigabytes of compressed data, at whatever product of record count and record length you like. (E.g., 1 billion relatively narrow records.) That’s on the order of 100 gigabytes of raw data.
  • Indeed, several QlikView customers manage several billion records each.
• The main ingredient of the performance secret sauce in QlikView is that selections are compiled straight into machine code. (QlikTech gave me the impression that this post is the first time that will be publicly revealed.) Notes on that include:
  • In the old days, QlikTech thought compilation gave a 10X performance benefit vs. interpreted code. However, 5X might be a more up-to-date figure.
  • It’s not just code; part of the compilation is to create temporary lookup tables.
  • A single calculation can use multiple cores. QlikTech thinks it’s done a very solid job of engineering efficient multicore parallelism. (Note: So far as I could tell, Hakan was using “calculation” to refer both to queries and, well, calculations.)
  • There’s a good reason QlikView runs only on Intel-compatible processors. A port would be painful.
• In QlikView’s world, one set of users accesses one set of applications against one database on one machine. However, different subsets (or copies of the same subset) of the same underlying database(s) can of course be run on different machines.
• Naturally, QlikView caches results and tries to re-use them. One smart thing about QlikView’s caching algorithm is that it takes into account the cost of generating the calculated results. This has the happy effect that large result sets, which are often the ones most likely to be useful in a subsequent calculation, are the ones most likely to be retained.
• One thing I unfortunately forgot to ask about is loading QlikView data into memory, something that has at times been problematic.

One last thing: QlikTech is going public. That means there is a QlikTech S-1, from which I learned, among other things, that QlikTech now seems to be called Qlik Technologies. Dave Kellogg offers an outstanding overview of the information in QlikTech’s filing(s). The points I’d add to Dave’s are primarily from the QlikTech balance sheet:

• Deferred revenue, which Dave calls out as high in absolute terms, is also growing faster than revenue (or any major component of revenue).
• Accounts receivable are also growing faster than revenue or any major component thereof.
• One possible explanation is weirdness with international distributors, which is at least potentially consistent with what QlikTech says is a shift in geographical mix.
• Another explanation is increasing deal size/complexity, something that is anyway common among enterprise software companies gaining market share, and that is also consistent with what QlikTech says is a growing fraction of revenue coming from existing customers.

]]> http://www.dbms2.com/2010/06/12/the-underlying-technology-of-qlikview/feed/ 20 http://www.dbms2.com/2010/06/11/ingres-vectorwise-technical-highlights/ http://www.dbms2.com/2010/06/11/ingres-vectorwise-technical-highlights/#comments Fri, 11 Jun 2010 11:28:18 +0000 Curt Monash http://www.dbms2.com/?p=2261 After working through problems w/ travel, cell phones, and so on, Peter Boncz of VectorWise finally caught up with me for a regrettably brief call. Peter gave me the strong impression that what I’d written in the past about VectorWise had been and remained accurate, so I focused on filling in the gaps. Highlights included: 

• VectorWise is indeed a shared-everything analytic DBMS.
• The VectorWise front-end is Ingres. Ingres VectorWise supports almost all SQL that Ingres does (there are a few edge-case exceptions).
• Conversely, Ingres VectorWise doesn’t support any SQL Ingres doesn’t, most notably SQL-99 Analytics. Naturally, SQL-99 Analytics is a roadmap item for Ingres/VectorWise.
• Ingres VectorWise 1.0 is pretty purely columnar. There’s a bit of PAX, but it’s mainly automagic/under the covers. The one user-controlled exception I understood was that one can ensure that composite keys are stored together.
• The main Ingres VectorWise performance secret sauce ingredients we touched on were:
  • Vectorization of operations (hence VectorWise’s name).
  • Compression that is tuned for speed rather than to minimize storage utilization.
• We unfortunately didn’t have time to revisit the other big part of the Ingres VectorWise performance story, namely clever design for modern microprocessor architectures. High-level generalities about that do pervade the Ingres VectorWise press release, but – well, they’re very high level.
• Unlike Vertica but like most other columnar DBMS vendors, Ingres VectorWise wants you to store your data once. You can index-organize the data. You can also organize multiple tables in the same order, to make joins among them fast.
• Support for actual join indexes is an Ingres VectorWise roadmap item.
• As do ever more analytic DBMS, Ingres VectorWise has something akin to Netezza zone maps.
• When I asked Peter what had changed most from the initial VectorWise development plan, other than the above, he basically said that their performance priorities had shifted a bit. Specifically, he said.
  • They had originally been “blinded” (his word) by the TPC-H benchmark, but figured out that they were overly focused on it. (Well, duh.)
  • They learned about the importance of other things such as data loading speeds.

]]> http://www.dbms2.com/2010/06/11/ingres-vectorwise-technical-highlights/feed/ 1 http://www.dbms2.com/2010/06/05/algebraix/ http://www.dbms2.com/2010/06/05/algebraix/#comments Sat, 05 Jun 2010 08:54:59 +0000 Curt Monash http://www.dbms2.com/?p=2227 I talked Friday with Chris Piedemonte and Gary Sherman, respectively the Cofounder/CTO and Chief Mathematician of Algebraix, who hooked up together for this project back in 2003 or 2004. (Algebraix is the company formerly known as XSPRADA.) Algebraix makes an analytic DBMS, somewhat based on the ideas of extended set theory, that runs on SMP (Symmetric MultiProcessing) boxes. Like all analytic DBMS vendors, Algebraix has on some occasions run some queries orders of magnitude faster than they ran on the systems users were looking to replace.

Algebraix’s secret sauce is that the DBMS keeps reorganizing and recopying the data on disk, to optimize performance in response to expected query patterns (automatically inferred from queries it’s seen so far). This sounds a lot like the Infobright story, with some of the more obvious differences being: 

• Infobright has fixed data structures, with what serve for indexes and precomputed aggregates added on the fly. Algebraix apparently reorganizes everything, including data partitions. (I also presume that Algebraix’s indexes and aggregates look rather different than Infobright’s data packs.)
• Infobright compresses data aggressively. Algebraix doesn’t yet compress.
• Infobright’s product is presumably much more mature.

So far as I can tell, that’s about it. Experience teaches that when a small company claims that some big mathematical breakthrough lets it have huge product superiority in DBMS or analytic tools, the claim doesn’t pan out. Maybe the company has good technology somewhat inspired by the mathematics, but the more breathless claims are always overwrought. Examples include Infobright (rough set theory), CrossZ/QueryObject (fractals), and KXEN (support vector machines). Algebraix (extended set theory) shows every sign of being another such case. An extended discussion about whether or not join operations are commutative – without benefit of any examples of their supposed non-commutativity – did little to convince me otherwise.

Finally, as per email from diligent PR guy David King,

• Algebraix has 22 employees in Austin and at its HQ in San Diego.
• Algebraix’s biggest customer is the Department of Defense (through BAE Systems).
• Algebraix’s product just became commercially available on March 1.
• Algebraix has completed one round of funding for $11 million.
• Algebraix is 100% angel-backed.
• Algebraix’s target market is “any mid-large sized organization that needs rapid access to analytics on large volumes of data.”

• Slide 2 has some market success figures and so on. (>3100 copies at >1800 users, >200 sales last year)
• Slides 6-11 give more detail on Sybase’s indexing and data access methods than I put into my recent technical basics of Sybase IQ post.
• Slide 16 reminds us that in-database data mining is quite competitive with what SAS has actually delivered with its DBMS partners, even if it doesn’t have the nice architectural approach of Aster or Netezza. (I.e., Sybase IQ’s more-than-SQL advanced analytics story relies on C++ UDFs  — User Defined Functions — running in-process with the DBMS.) In particular, there’s a data mining/predictive analytics library — modeling and scoring both — licensed from a small third party.
• A number of the other later slides also have quite a bit of technical crunch. (More on some of those points below too.)

Sybase IQ may have a bit of a funky architecture (e.g., no MPP), but the age of the product and the substantial revenue it generates have allowed Sybase to put in a bunch of product features that newer vendors haven’t gotten around to yet.

More recently, Sybase volunteered permission for me to preannounce Sybase IQ Version 15.2 by a few days (it’s scheduled to come out this week). Sybase IQ seems to be focused on large part on the government/intelligent market, with three major features being:

• A kind of data federation, querying external databases, that makes sense mainly in the context of rigorous security rules. (I find that confusing, since Sybase IQ’s indexes tend to hold all the information in the database, but I didn’t push the point.)
• An upgrade to Sybase IQ’s built-in text indexing. I doubt anybody would confuse this with best-of-breed text search, but evidently that intelligence community is satisfied with less. But even before 15.2, Sybase IQ could do both LIKE and WHERE CONTAINS searching.
• Improved LOB (Large OBject) management.

One part of my Sybase IQ conversations I haven’t blogged yet in much details is scale-out, concurrency, and “multiplexing.”

• Sybase feels that Sybase IQ’s competitive sweet spot, especially in terms of performance, is reached when there are 20 or more concurrent queries.
• In general, Sybase asserts that a shared-everything architecture is great for concurrency — just run different queries on different boxes, all against the same data.
• The ability to use a bunch of boxes run Sybase IQ is called “multiplexing.”  This is a chargeable option, without which one is limited to a single SMP box.
• Just under 20% of the top 250 Sybase IQ customers have multi-node scale-out configuration (vs. single-node SMP scale-up). And around 8% have it overall.
• Sybase IQ nodes can be heterogeneous (e.g., in compute power).
• Sybase IQ nodes can be dedicated to be read-only, or can be read-write. Indeed, Sybase IQ nodes can change roles dynamically, for example becoming write-only during nightly batch load. (I didn’t clarify whether all this applies just to nodes-as-boxes, or if some parts apply to specific processors or cores within the same box.)
• Sybase noted that data mart outsourcers can offer differentiated SLAs (Service Level Agreements) depending upon which nodes they give which customers access to.
• Most Sybase IQ installations start at 8 cores or more. The Sybase IQ Small Business Edition, limited to 4 cores, is not a big seller.
• Sybase IQ has a straightforward round-robin load-balancing story via third-party technology.

Finally, along the way in the discussions I picked up various tidbits about the Sybase IQ user base. Unfortunately, Sybase is pretty vague in discussing database sizes — are they user data? Are they compressed? What do the numbers mean? With that huge caveat:

• By some metric or other, a couple of classified customers are approaching petabyte scale.
• The largest commercial Sybase IQ customer — a credit card company — has a couple hundred terabytes or so.
• The largest financial services Sybase IQ databases are 50-70 terabytes. This sounds low, frankly, so maybe those are compressed figures, with user data being 200+ terabytes. But I’m just speculating there.
• Sybase IQ has a little less than 100 customers in the “data aggregator” market, which is a lot like what I call “data mart outsourcer.”
• Sybase IQ’s ILM technology is a chargeable option, with Sybase being “cautious” about sales. Compliance is a big market driver for it.
• Sybase IQ’s #1 vertical market is financial services. Other biggies are government, telecom, marketing services, and to some extent retail.
• As of February, there were 40-45 production users of Sybase IQ 15.0 and 15.1.

For starters, Sybase IQ is not just a bitmapped system, but it’s also not all that closely akin to C-Store or Vertica. In particular,

• Sybase IQ stores data in columns – like, for example, Vertica.
• Sybase IQ relies on indexes to retrieve data – unlike, for example, Vertica, in which the column pretty much is the index.
• However, columns themselves can be used as indexes in the usual Vertica-like way.
• Most of Sybase IQ’s indexes are bitmaps, or a lot like bitmaps, ala’ the original IQ product.
• Some of Sybase IQ’s indexes are not at all like bitmaps, but more like B-trees.
• In general, Sybase recommends that you put multiple indexes on each column because — what the heck – each one of them is pretty small. (In particular, the bitmap-like indexes are highly compressible.) Together, indexes tend to take up <10% of Sybase IQ storage space.

Sybase IQ is not immune to Sybase’s confusing choices in version numbering. Thus:

• Sybase IQ Version 15.2 will be announced and released soon.
• Sybase IQ Version 15.1 was a set of “binary replacements” rather than an “upgrade release” for Sybase IQ Version 15.0.
• Sybase IQ Version 15.0 was launched in February, 2009 and released for general availability some time thereafter.
• The prior version of Sybase IQ was 12.7.
• GA isn’t always GA, and some language localizations and so on weren’t ready for a while. Consequently, a lot of Sybase IQ sales continued to be of Version 12.7 even in the second half of 2009.

Now let’s get down to some technical particulars.

Sybase IQ columns are always stored in RowID order. However, RowIDs are logical and not physical, and hence take up little disk space. A small amount of per-page metadata lets you find the specific cell you want. (Cells are commonly fixed-width, in which case finding the cell of choice is a simple calculation.) So RowIDs are not much of an I/O overhead issue, although I’m not sure at what point they get unpacked and start needing to be carried around as the data travels through silicon.

Sybase IQ has 9 or so kinds of indexes. The choice of index has a lot to do with cardinality. In the extreme low-cardinality case, a simple bitmap might do. With intermediate cardinality, you might go to a modified kind of bitmap – e.g., if there there are 2^16 possible values, you can represent a value in 16 bits, and bitmap operations are approximately 16 times as costly as if the number of possible values were only 2^1. For very high cardinality, there’s a B-tree-like index called “High Group”.

Note: Surely every Sybase index name, at some time, made sense to at least one engineer.

Sybase IQ’s execution engine does seem to rely quite a bit on bitmaps. E.g., intermediate query results are stored as bitmaps, which helps them play nicely with each other and with many of the indexes. Sybase claims that Sybase IQ’s bitmap orientation often makes WHERE clauses execute very quickly. Sybase IQ reoptimizes queries after WHERE clauses are evaluated. Complex expressions are, when possible, evaluated once per unique value, not once per row.

Speaking of unique values – Sybase IQ’s compression story doesn’t currently match that of some other columnar products, but it seems to stack up pretty well against row-based systems. In particular:

• Sybase says IQ compression is most commonly 50-70%.
• Sybase further says that, in most cases, compression falls into the range 40-85%.
• Page-level LZ compression is decompressed upon read (duh).
• Dictionary/token compression may be decompressed later. For example, GROUP BYs are commonly done on tokens, and JOINs sometimes are.

Sybase IQ boasts pipelining, in two senses. First, IQ tries to read pages for multiple queries at the same time. Second, Sybase IQ tries to prefetch pages into cache before they’re needed. Sybase points out that these prefetched pages have the WHERE clauses already executed, and that no extra baggage is being dragged into cache that doesn’t need to be there.

Highlights of Sybase IQ’s update and load story include:

• Sybase IQ is optimized for large bulk loads. No surprise there.
• Sybase IQ has several options for microbatching and/or trickle feeds.
  • The coolest is Sybase RAP.
  • More generally, microbatching is based on Change Data Capture. Sybase has various ETL/replication technologies, creating a confusing array of options in that regard.
  • Sybase says that one customer is microbatching 1000s of rows with 1 minute latency.
• There’s something about snapshotting and hence loads not interfering with queries. I’m not clear on the details.
• Assuming you have enough parallelism, you can dedicate some nodes to queries while others are dedicated to load. (Recall that Sybase IQ is shared-disk.)

I’ve lost track a little bit as to which “advanced analytics” functionality is in Sybase IQ 15.1, which will be in 15.2, and what’s a future beyond that, which is a great excuse for me to leave it out of what has already become a rather long post. But anyhow, except perhaps for the future stuff and/or some time series functionality, none of it seems terribly advanced. Sybase IQ does have two stored procedure languages, namely the ones for Sybase ASE (T-SQL) and for Sybase Anywhere or Adaptive Server Anywhere or whatever it’s called this week (Watcom SQL, which Sybase asserts is similar to the ANSI SQL stored procedure language).

Similarly, I’ll leave a lot of other stuff out as well, and for now stop here.

Related links

• I haven’t repeated every detail here from my August, 2009 technical post about Sybase IQ
• And here’s more about Sybase IQ, including some Sybase IQ 15.2 features, some market penetration info, and a slide deck

On the whole, the releases look pretty lame. Highlights seem to include:

• Maybe a claim of enhanced data compression.
• Otherwise, no obvious new technology except product packaging and bundling.
• Aggressive plans to throw capital at the Sun channel to convert it to selling IBM gear. (A figure of $1/2 billion is mentioned, for financing.

Disappointingly, IBM shows a lot of confusion between:

• Text data
• Machine-generated data such as that from sensors

While both highly important, those are very different things. IBM has not in the past shown much impressive technology in either of those two areas, and based on these releases, I presume that trend is continuing.

Edits:

I see from press coverage that at least one new IBM model has some Fusion I/O solid-state memory boards in it. Makes sense.

A Twitter hashtag has a number of observations from the event. Not much substance I could detect except various kind of Oracle bashing.