• If your goal is to keep a certain class of queries from taking too many resources, Tim thinks a great way of doing that is to control how many of them are allowed to run concurrently.
• By way of contrast, Tim is cautious about the common approach of just lowering a query’s priority. His concern is that a long-running query could linger even longer, creating a long-lasting bottleneck in, for example, temp space.
• When running over (I believe) Linux and AIX, DB2 workload management is integrated with operating system workload management. I.e., the same “service class” or “workload class” (at a guess, the former is the official term and the latter is the term that makes sense) of queries and associated processes gets the same treatment in both DB2 and the OS.
• DB2’s workload management extends to buffer pools, to inhibit low-priority queries from evicting a higher-priority query’s data from cache.
• Sometimes, workload management doesn’t throttle a query, but just decides to collect stats for future analysis. (This is on the eminently reasonably theory that the best stats to collect are the ones that are live when performance problems are actually occurring.)

Finally, Tim spoke of what I regard as the weirdest workload management requirement, one I also heard about from Netezza (but didn’t explicitly mention) in June. Sometimes, it seems, you simply don’t want queries to finish too fast. Why? Because if you give great performance when the machine is lightly loaded, then business users might expect that performance too when the machine is heavily loaded and you can’t deliver it. Apparently, in some environments it’s better to never deliver great query performance than it is to do so only inconsistently.

• You really, really want to have multiple data streams coming out of temp space, as close to simultaneously as possible.
• The storage performance characteristics of such a workload are more reminiscent of “random” than “sequential” I/O.

If everybody else is cool with it too, I can live with that.

Meanwhile, I talked again with Tim Vincent of IBM this afternoon. Tim endorsed the temp space/Flash fit, but with a different emphasis, which upon review I find I don’t really understand. The idea is:

• Analytic DBMS processing generally stresses reads over writes.
• Temp space is an exception — read and write use of temp space is pretty balanced. (You spool data out once, you read it back in once, and that’s the end of that; next time it will be overwritten.)

My problem with that is: Flash typically has lower write than read IOPS (I/O per second), so being (relatively) write-intensive would, to a first approximation, seem if anything to disfavor a workload for Flash.

On the plus side, I was reminded of something I should have noted when I wrote about DB2 compression before:

Much like Vertica, DB2 operates on compressed data all the way through, including in temp space.

This piqued my interest, so — recalling that ANTs is a public company — I decided to take a look at just how successful their software products business is. Well, for the quarter ended March 31, 2010, ANTs’ 10-Q filing says (emphasis mine): 

The Company’s revenues for the three months ended March 31, 2010 and 2009 include service revenues representing managed and professional service fees for database and network maintenance and support services. Revenues for the three months ended March 31, 2010 were $1.5 million, an increase of $0.1 million compared to $1.4 million for the three months ended March 31, 2009.  For the three months ended March 31, 2010, two customers accounted for 96% of the Company’s gross revenues (Company A, 72% and Company B, 24%) compared to three customers that accounted for 97% of the Company’s gross revenues for the three months ended March 31, 2009 (Company A, 57%, Company B, 29% and Company C, 10%). The increase in revenues for the three months ended March 31, 2010 over the comparable period in 2009 is primarily attributable to professional service projects for Company A that were initiated during or subsequent to the three months ended March 31, 2009, partially offset by professional service projects for Company B and Company C that were completed subsequent to March 31, 2009.

Conditional on the Company’s technology developments being successful, the presence of customer demand and the Company having a competitive advantage, future revenues may include sales and licenses of its ANTs Compatibility Server (“ACS”) product and managed services revenue related to existing and new contracts and professional services revenue from pre- and post-sales consulting related to ACS and other database consolidation technologies. Sales of the Company’s first ACS product, which translates from Sybase to Oracle, have been limited due to the structure of the sales arrangement and go-to-market strategy. As such, the Company has structured the go-to-market strategy for the second ACS product differently via the use of an Original Equipment Manufacturer (“OEM”) agreement. Pursuant to the OEM agreement, ANTs is responsible for technology development specifically tailored to the OEM’s needs. The OEM will assume responsibility for marketing, sales and support of the technology on a worldwide basis, while ANTs will be the preferred service provider for migration projects. The Company is currently in the process of developing the second ACS product for a planned announcement and release in mid-2010. The Company intends to develop additional ACS products based on market demand and the availability of resources for development.

In other words, as of four months ago ANTs had had $0 in business in what it says is its main product area, which is pretty much the range the company has been in throughout its complicated history.  Kozak’s post did link to a claim that IBM has experienced over 300 migrations to DB2. However, that figure includes Oracle-to-DB2 migrations that having nothing to do with ANTs. And by the way, IBM’s migration strategy is focused largely on ISVs, so the whole Sybase-ANTs dust-up may be about a type of business (direct capture by DB2 of Sybase ASE enterprise customers) nobody’s sales force is seriously pursuing.

True, the Sybase-to-DB2 emulation technology hadn’t been released as of then. Even so, I think it’s a wee bit early for ANTs to be acting as if there’s been any proof it ever has had or will have any significant market success.

• Netezza now intends to come out with a Flash-based appliance earlier than it originally expected.
• Indeed, Netezza has suspended — by which I mean “scrapped” — prior plans for a RAM-heavy disk-based appliance. It will use a RAM/Flash combo instead.*
• Tim Vincent of IBM told me that customers seem ready to adopt solid-state memory. One interesting comment he made is that Flash isn’t really all that much more expensive than high-end storage area networks.

Uptake of solid-state memory (i.e. Flash) for analytic database processing will probably stay pretty low in 2010, but in 2011 it should be a notable (b)leading-edge technology, and it should get mainstreamed pretty quickly after that. 

*So far as I can tell, that’s one of the two significant roadmap changes between the 2009 and 2010 editions of Enzee Universe. The other one is that the robust form of appliance-to-appliance replication technology is coming out later than Netezza had originally planned and hoped.

There also is increasing reason to think that the issues with Flash memory wearing out are overwrought.  And by the way, the entire history of enterprise solid-state memory use is basically shorter than the time in which these products supposedly will wear out, so it’s not as if there have been a lot of real-life failures out there.)

• First, clever things are being done in the area of error correction codes, although for the most part I defer that part of the discussion to Petascan’s Camuel Gilyadov. E.g., this seems to be the idea behind Anobit.
• Second, analytic DBMS are pretty much an ideal use case for Flash reliability. Suppose, as is the case for many products and implementations, you only write things in big blocks. Then you are, ipso facto, resetting the Flash bits only in big blocks. Thus, at least in theory, you automatically have pretty perfect wear leveling.

• Subsecond load latency is substantially impossible. Doing that amounts to OLTP.
• 5 seconds or so is doable with aggressive investment and tuning.
• Several minute load latency is pretty easy.
• 10-15 minute latency or longer is now very routine.

There’s generally a throughput/latency tradeoff, so if you want very low latency with good throughput, you may have to throw a lot of hardware at the problem.

I’d expect to hear similar things from any other vendor with reasonably mature analytic DBMS technology. Low-latency load is a problem for columnar systems, but both Vertica and ParAccel designed in workarounds from the getgo. Aster Data probably didn’t meet these criteria until Version 4.0, its old “frontline” positioning notwithstanding, but I think it does now.

Related link

• Just what is your need for speed anyway?

When you sell a row-based DBMS, as Netezza and IBM do, there are a couple of approaches you can take to compression. First, you can compress the blocks of rows that your DBMS naturally stores. Second, you can compress the data in a column-aware way. Both Netezza and IBM have chosen completely column-oriented compression, with no block-based techniques entering the picture to my knowledge. But that’s about as far as the similarity between Netezza and IBM compression goes. 

IBM’s basic DB2 compression strategy is remarkably simple. In every table (not column) – or in each range partition in a range-partitioned table — the 4096 most common* values are identified; these are all encoded into 12-bit strings. And that’s that. This has been happening since DB2 9.1, released 4 ½ years ago. DB2’s compression persists through logs, buffer pools (i.e., RAM cache), and so on. In DB2 9.7, the most recent release, IBM extended the use of the compression to a few areas it hadn’t stretched before, such as log-based replication, native XML, or CLOBs (Character Large OBjects) that happen not to be too big.

*Actually, I’d presume it’s not exactly the “most common”; there surely is some minimum length of a value to be encoded, or some bias toward length. Also, the determination of what to encode is probably a little imprecise. E.g., I forgot to ask whether the choice of values ever changes as data got updated.

The sophisticated part of DB2’s simple compression strategy is its breadth of applicability; DB2 compression can apply to:

• Values in columns (numeric, character, whatever)
• Substrings of values in columns
• Groups of columns (e.g., city/state/zip code)

Except for the 4096 values limit, that sounds at least as flexible as the Rainstor/Clearpace compression approach.

Netezza, unlike IBM, takes a grab-bag approach to compression – try out a bunch of techniques, see which work best, and incorporate those in the product. Netezza first introduced compression a couple of years ago, for numeric columns only, especially integer.  Techniques  used in Netezza numeric compression include but are not limited to:

• Delta compression, wherein you store the increment between a value and its predecessor rather than a whole new value.
• Ways of indicating that a value or increment was just the same as in the row before.

This was via something called Compress Engine,* now being renamed to Compress Engine 1. Netezza’s new Compress Engine 2 improves on what Netezza did in Compress Engine 1 for numeric data, most notably by trimming away excess field length. (Netezza says it got 28% better compression on a test data set with almost no character strings, primarily from that enhancement.) Further, Netezza Compress Engine 2 adds new compression techniques, allowing it to handle VARCHAR – i.e. character strings — as well.

*Fortunately, the original name or at least description of “Compiled Tables” is retreating ever more from view.

Netezza’s Compress Engine 2 has two ways to compress character fields/text strings – prefix compression and Huffman coding. By way of contrast, Netezza tested suffix compression and decided it wasn’t beneficial enough to bother messing with.

• The idea behind prefix compression is that if two strings start with the same characters, for the second one you only have to record the part that’s different. Prefix compression has a lot of the same merits as delta compression; like delta compression, it works best on sorted columns. (An example of where prefix compression makes obvious sense is URLs, which tend to all start in similar ways.)
• In Netezza’s version of Huffman coding, the alphabet is encoded symbol-by-symbol, with more common characters getting codes of shorter length. These codes are chosen on a column-by-column basis. (I presume the “/” character gets shorter code in a URL column than it would, for example, in one that stored addresses.)

While I didn’t ask explicitly, it seems pretty obvious that Compress Engine 2’s functionality is a strict superset of Compress Engine 1’s. Netezza is going to run Compress Engines 1 and 2 side by side, but expects pages to move from Compress Engine 1’s purview to Compress Engine 2’s as part of the new “table grooming” process.

Related links

• IBM kindly permitted me to post some of its slides in the area of compression
• Microsoft SQL Server seems to rely on prefix and dictionary compression

And it’s not going to get any better. Next week = the oft-postponed elder care trip. Then I’m back for a short week. Then I’m off on my quarterly visit to the SF area. Soon thereafter I’ve have a lot to do in connection with Enzee Universe. And at that point another month will have gone by.

Anyhow:

• Back in January, Oracle finally briefed me on Exadata 2. I also requested and got permission to post what I regarded as pretty interesting slides, then never got around to doing so. Well, here they are. (Pay no attention to the word “Confidential”.)
• Two people I have a lot of respect for, Cindi Howson and Doug Henschen, seem bullish on SAP’s in-memory NewDB efforts. But for a variety of execution reasons, I’m skeptical that this will matter for anything except SAP’s analytics suite. I.e., I don’t think anybody much except SAP will write OLTP apps to it, and I don’t think that without OLTP apps being written to it it’s much more than Business Objects’ answer to QlikView.
• I just learned that Netezza’s previous geospatial technology didn’t get ported to TwinFin. However, Netezza obviously found a geospatial alternative.

I ‘m beginning to make a habit of asking vendors for a postable version of their slide decks. Sybase IQ is another example.

• Google is doing something called BigQuery that is “SQL-like” for big data analytics. I don’t know anything about it.
• I also don’t know anything about IBM BigSheets yet. It sounds something like Datameer, but that could be way off the mark.

Cast Iron sold an integration appliance, most focused on integrations that involved SaaS applications such as Salesforce, with an option for doing all this purely in the cloud. IBM is accordingly spinning Cast Iron as a major cloud player, which is something of an exaggeration.

IBM will surely get value from whatever specific connectors Cast Iron does a better job at than IBM’s current offerings do. What I’m more curious about is whether Cast Iron’s core technology will survive in a form that continues it’s core message of “simplicity, simplicity, simplicity.”

On the whole, the releases look pretty lame. Highlights seem to include:

• Maybe a claim of enhanced data compression.
• Otherwise, no obvious new technology except product packaging and bundling.
• Aggressive plans to throw capital at the Sun channel to convert it to selling IBM gear. (A figure of $1/2 billion is mentioned, for financing.

Disappointingly, IBM shows a lot of confusion between:

• Text data
• Machine-generated data such as that from sensors

While both highly important, those are very different things. IBM has not in the past shown much impressive technology in either of those two areas, and based on these releases, I presume that trend is continuing.

Edits:

I see from press coverage that at least one new IBM model has some Fusion I/O solid-state memory boards in it. Makes sense.

A Twitter hashtag has a number of observations from the event. Not much substance I could detect except various kind of Oracle bashing.

• Tom Foremski outlined the dire (at least in theory) privacy risks of geolocation services, going into a lot more detail on that point than I ever have. However, he topped that off with the odd claim that people pay toll (rather than using an electronic service) to cross the Bay Bridge because they fear being tracked, rather than for reasons of time or money.
• Oracle had an earnings conference call. Larry Dignan did a good job of covering the highlights; the gory details are on the Seeking Alpha transcript, especially pp. 3-5.  Oracle now claims to be getting lots of multi-system deals for Exadata. (But I still haven’t seen much in the way of production customers named.) ULAs, which I presume are Unlimited License Agreements, are important on the software side. Besides picking on IBM and SAP, Oracle even touted a competitive win vs. EMC, which not coincidentally seems to be working on partnering with almost every Oracle competitor it can find.
• Brian Prentice of Gartner basically accused open source of being Dotcom 2.0, in terms of dubious business models and the hype associated with same. I agree with many of his particulars, and indeed often steer vendor clients away from open source strategies. For marketing purposes, I do feel that sometimes free can be a real cool price; but open source is not the only way to be free.
• Akiban, which I wrote about a couple of days ago, seems to be building out its website. As of this writing the website is still pretty raw, with bewildering messaging, carelessly repeated paragraphs, and a notable lack of clues as to who’s in company leadership. Even so — unless I missed some of the current stuff before, the site has come a long way in a few days, so maybe there’s hope.
• Groovy Corporation, which introduced the Groovy SQL Switch just last summer, seems to be doing something different now. It’s merged into a company called uCirrus (where the u is really a mu), but uCirrus doesn’t have a meaningful website yet, whereas Groovy does. There’s stuff there about a “push data cloud,” stressing the importance of not being a DBMS, under the name Cortex, whatever that all means. Groovy seems to have an online gaming deal for Cortex with MySpace, or maybe Cortex is just the name of a specific Groovy/MySpace project.
• Mike Mooney offered a long rant on the problems with database (design) version control. He did concede that the most recent Microsoft Visual Studio might help, for those who are bought into (and can afford) the Microsoft stack. Frankly, I think that’s what views are for, updatable or otherwise. In many cases, they’ll let you build what you need, quickly and without breaking anything, and you can leave it to the DBAs to sort out database performance later.
• I just discovered Chad Stewart’s programming blog. While he’s evidently a game programmer, a lot of his comments have broader applicability.
• Chip Hazard offered a VC’s perspectives on the difficulties facing enterprise IT startups. (Hat tip to Miriam Tuerk for turning me on to him.) Although he didn’t phrase it this way, his bottom line (at least the part I agree with) is that the startup’s products have to be amazingly superior to the alternatives (big vendors or in-house).