The immediate hook here is that I had a short conversation with Scott Gnau of Teradata yesterday, triggered by Teradata’s acquisition of Kickfire’s assets. Takeaways from that part included:

• The acquisition is all about Kickfire’s data pipelining technology.
• Scott (in my opinion rightly) thinks that isn’t particularly tied to Kickfire’s choice of particular DBMS architecture (fairly vanilla columnar).
• No decision has been made about whether the right vehicle for this technology is an FPGA (Field Programmable Gate Array), conventional Intel CPU, RAM, etc.

If you want to handicap Teradata’s future data pipelining strategy, you might note that:

• Kickfire’s own choice – and hence its existing implementation – is an FPGA.
• VectorWise’s approach to pipelining is Intel-based, apparently at the cost of being closely tied to specific generations of Intel CPUs.
• XtremeData’s approach to pipelining is FPGA-based.
• Teradata has a lot more development resources than any of those other companies, as well as important existing products, and hence has both means and motive to shoehorn new technology into older system designs.

While I had Scott on the phone, I brought up a few other subjects too. Highlights included:

• Teradata’s Flash-based appliance is doing just fine in beta test and customer POCs (Proofs of Concept).
• Other kinds of Teradata appliance are not inconceivable.
• Scott thinks Michael McIntire’s condemnation of Active-Active architectures is overstated. That said,
  • Scott does acknowledge a need for greater Active-Active scalability, and suggests that the reason Xkoto’s current products are being discontinued is their lack of scaling.
  • Scott seems quietly confident the scaling will get done.
• Scott is emphatic that Teradata is not going to go to a two-tier architecture. In particular, the point of splitting storage/lightweight database processing and heavyweight database processing on separate tiers is generally to save bandwidth, and Teradata’s BYNET is typically less than 10% loaded.
• Scott didn’t dispute my claim that this all suggests Teradata Virtual Storage is the future, at the expense of a rigid delineation among specific use-case-focused product lines.
• Unlike Netezza or Aster, Teradata doesn’t seem to plan analytic capability that works outside the UDF (User Defined Function) framework. However, Scott noted that Teradata has long had the capability that Aster and Netezza now also have of letting you run analytic code either in “protected mode” (if the process fails the whole database doesn’t crash) or in the database kernel (best performance, if you’re sufficiently confident in the code’s stability to take the risk). Scott also spoke of the release later this quarter of Teradata FastPath, which will offer yet better performance (however, there’s a gotcha to Teradata FastPath that’s still NDA).

Putting all that together with the rest of what we know about Teradata, I’m going to call out three pillars of Teradata’s long-term product strategy:

• Same fundamentals as always. Teradata’s core product strategy is:
  • Single DBMS, capable of meeting all analytic needs while running in a single instance, usually running on …
  • … proprietary hardware …
  • … built from conservatively-chosen parts.
• Selective vertical application stack. No matter how horizontally-oriented they are, many companies that have been in the analytic technology business for a while wind up with some vertical applications. It sort of just happens. Teradata is no exception. Teradata also likes to sell services to its product customers, and some of those are quite vertical-aware.
• Mutable, modular platform. This is what I highlighted above. Note that it’s philosophically attuned with the one-system-does-everything approach Teradata prefers. More subtly, please also note that it goes well with customer-by-customer price customization, which is almost a must for Teradata given the Innovator’s Dilemma kind of pricing box it finds itself in.

So far, that’s not too exciting, except in the details of how Teradata’s engineers make that all work. But there’s a fourth pillar to Teradata’s technical strategy as well, and it’s a wild card: tight partnerships. Every time I talk with Teradata hardware chief Carson Schmidt, he seems excited about some particular version of a part or other – sometimes from a reasonably established vendor (once it was LSI Logic), sometimes from a tiny one (notably the “stealth” start-up on which Teradata bet its first solid-state product.) In the future, I expect tight business intelligence partnerships as well. Cognos BI will be increasingly integrated with IBM’s DBMS and hardware; Business Objects’ BI will increasingly be integrated with SAP’s applications; and Oracle’s BI will eventually be integrated with everything. How do you compete with that if you‘re Microstrategy? Well, you try to have superior product, of course – but you also partner as closely with DBMS vendors as you can, an approach Microstrategy has already started. Predictive analytics stalwart SAS, of course, is on a partnership binge as well.

Teradata has a larger installed base than almost all its competitors, and enjoys richer third-party software and service support as a result. But I suspect that going forward, for Teradata to remain a leading competitor at price points it is willing to accept, Teradata’s “ecosystem” advantages will need to ratchet up one or several notches.

• Aster Data showed me a lot of customer names and deal sizes, across a bunch of industries (mainly enterprise rather than web). Yes, Aster’s market success is for real. (But almost all those details are NDA.)
• Sybase’s product plans for IQ are pretty impressive. (But the most interesting parts are, you guessed it, NDA.)
• I’ve kissed and made up* with ParAccel, now that they’ve replaced their CEO, replaced their marketing chief, and stopped the worst of the marketing nonsense I used to complain about. ParAccel has some interesting plans for ParAccel 3.0 which are, naturally, NDA.
• The Peoplesoft guys are doing it over again at Workday. Only this time, their platform isn’t a relational DBMS. Rather, it’s an in-memory, completely object-oriented data model, with disk used only on a “Just in case the power ever goes out” basis. (Thankfully, nothing at all about our conversation was NDA.)
• I’m finally feeling good about Northscale’s memcached-compatible persistent store Membase. The main reason is that they showed me a near-term path to interfaces that are richer than key-value. Also, Todd Hoff reassured me that even pure persistent memcached has a place.
• Rumor says that even the one app for which Facebook was using Cassandra — in-box search — has been decommissioned. On the other hand, numerous other scale-0ut DBMS (SQL or otherwise) seem to have Facebook footholds. But details are — all together now! — NDA.

*If you know ParAccel’s new marketing chief Michael Weir, you surely guessed I mean that only in a figurative sense.

From elsewhere:

• Daniel Abadi offered his analysis of Kickfire’s demise. In general I agree, but Daniel neglected to mention one hugely important factor — the chicken-egg negative effect of Kickfire’s lack of market or marketing traction. Customers were extremely reluctant to buy from Kickfire because they perceived, correctly, that Kickfire’s survivability was far from assured.
• While the InfiniDB forums suggest that there are at least a couple of production users of Calpont’s free InfiniDB, Calpont seemingly has a long way to go to be even as successful as Kickfire. But Calpont does have a bit of money to spend on lead generation; maybe some day they’ll even have actual customers.
• In a response to a question I messaged over, XtremeData tells me they have actual customers now. Press releases to follow.
• The admiration for the job Mark Hurd did at HP is in my opinion overstated. Sure, the financial/operational management appeared to work, but HP did little on Hurd’s watch to strengthen its reputation or customers’ loyalty. In particular:
  • HP’s analytics efforts have accomplished little.
  • HP’s data warehouse appliance efforts have failed pathetically.
  • From what I hear, HP’s execution in its Exadata partnership was not good.
  • HP’s server business in general is distinguished mainly by HP being a big company.
  • HP’s EDS acquisition has been rocky, not that EDS was sailing so smoothly on its own beforehand.
  • HP’s success in PCs amounts to “arguably, HP sucks a little less than the other guys”.
  • HP’s elite reputation is long gone (admittedly, for the most part that predates Hurd).
• Doug Henschen evidently favors really strong intellectual property protection for software, even forbidding plug-compatible reverse engineering. I agree with Doug up to the point that it should be forbidden to copy proprietary software, but I don’t see why he (or a court) would view such behavior as copying.

Most of what I get paid for is in some form or other consulting. (The same would be true for many other analysts.) And so I can be a bit stingy with my advice toward non-clients. But my non-clients are a distinguished and powerful group, including in their number Oracle, IBM, Microsoft, and most of the BI vendors. So here’s a bit of advice for them too.

Oracle. On the plus side, you guys have been making progress against your reputation for untruthfulness. Oh, I’ve dinged you for some past slip-ups, but on the whole they’ve been no worse than other vendors.’ But recently you pulled a doozy. The analyst reports section of your website fails to distinguish between unsponsored and sponsored work.* That is a horrible ethical stumble. Fix it fast. Then put processes in place to ensure nothing that dishonest happens again for a good long time.

*Merv Adrian’s “report” listed high on that page is actually a sponsored white paper. That Merv himself screwed up by not labeling it clearly as such in no way exonerates Oracle. Besides, I’m sure Merv won’t soon repeat the error — but for Oracle, this represents a whole pattern of behavior.

Oracle. And while I’m at it, outright dishonesty isn’t your only unnecessary credibility problem. You’re also playing too many games in analyst relations.

HP. Neoview will never succeed. Admit it to yourselves. Go buy something that can. 

Smaller BI vendors. Analytic DBMS evaluations commonly include BI strategy and tool selection as well. If an analytic DBMS expert tells you he needs to learn more about your product line, don’t blow him off. In fact, you should be particularly embracing anybody who’s shown a fondness for small DBMS vendors; maybe he or his clients will like small BI vendors as well. That means (among others) you, Jaspersoft, Endeca, and Tableau.

Information Builders. Is there anything about your BI products that is in any way technologically differentiated? If so, you might want to mention some examples to somebody some time.

Kalido. I’ve said this to you before, but it bears repeating — your positioning translates to “I-CASE for analytics,” and that’s not a good thing. If your product is not as cumbersome and entrapping as that sounds, you need to do a much better job of explaining why not.

SenSage. You are what you are. Sell out while the selling is good. You don’t have the corporate personality to make it into the analytic DBMS mainstream on your own.

Ingres. You need to be more engaged with analysts than you are. Ingres navel-gazed too much 25 years ago, and evidently you haven’t outgrown it yet.

TIBCO. You probably have a lot of cool analytic technology, but I don’t know of an influencer who has much relationship with or trust in you. Rethink how you’re approaching influencer relations top to bottom.

Tableau. You had a lot of mindshare, but it’s fading. Do something.

MarkLogic, graph DBMS vendors, etc. You’re clinging too hard to the NoSQL label. Nobody is out there deciding among Cassandra, neo4j, and MarkLogic. They might be deciding between MongoDB and MarkLogic, I guess, but if you admit to yourself that’s all it is you’ll probably change your messaging somewhat.

Objectivity. Get real about marketing. Infinite Graph is a cool opportunity. But I didn’t even ping you for a meeting when I’m in your area next week, because I wouldn’t have known who to reach out to.

Everybody (especially Objectivity). “First X deployed in the cloud” is almost surely an inaccurate claim. Don’t make it. And by the way, even if it were true, it probably wouldn’t be interesting.

At this time, I have no idea who the lucky buyer is.

Edit: We now know it’s Teradata.

• Ben says Greenplum wasn’t being shopped, by which he means Greenplum was out raising more capital and the fund-raising was going well.  Note: Half or so of Greenplum’s deals were subscription-priced, so it had weaker cash flow than it would have if it were doing equally well selling perpetual licenses.
• However, joint engineering was also going well with, e.g., Greenplum CTO Luke Lonergan spending time at EMC facilities in Cork, Ireland. And one thing led to another …
• Greenplum has ~ 140 customers, vs. ~65 five quarters ago, 100+ at year-end, and an acquisition rate of 12-15/quarter last fall.
• A typical “small” paying customer for Greenplum starts with 10-20 TB of data.
• Greenplum Chorus isn’t generally available yet, with rollout energy being focused on Greenplum 4.0. Note: As important as it is for overall industry direction, Greenplum Chorus is a product which won’t be a terribly big deal in Release 1 anyway.

Highlights looking forward include: 

• When I challenged him, Ben sounded quite optimistic that Pat Gelsinger will immunize Greenplum against and generally counteract some of EMC’s traditionally stifling bureaucracy. (My words, of course, not his.)
• The initial Greenplum/EMC product vision appears truly centered around “private cloud,” specifically including Greenplum, VMware, and EMC storage arrays.
• Some other areas of potential Greenplum/EMC technical synergy I think are cool obviously haven’t been seriously addressed yet.
• Based on what I heard from Ben about the aura around the deal and also on what I know of the individual executives at Greenplum, I think each of them is a good bet to stick around EMC for a while. (That’s on average. Of course, it would be surprising if 100% of them stayed around very long.) Basically, there’s at least a chance EMC/Greenplum will do some pretty cool stuff, and most of the guys will probably stick around to see if that actually starts to happen.*

*Also, when they do eventually leave, they’ll surely say things to the effect “The cool stuff is well underway; my work here is done.” That party line is almost guaranteed, no matter how things unfold in reality.

I think that even software-only analytic DBMS vendors should design their systems in an increasingly storage-aware manner

promising to explain what I meant later on. So here goes. 

There always have been good technical reasons to tailor hardware to analytic database software. Data moves through disk controller, network, RAM, CPU and more, each with its own data rate. Getting different kinds of parts into the right balance doesn’t completely eliminate bottlenecks – the Wonderful One-Hoss Shay is poetic fiction – but it certainly can help. As a result, every analytic DBMS vendor of any size offers at least one of:

• A Type 0 appliance
• A Type 1 appliance
• A “recommended hardware configuration”

And beyond performance, appliances and pre-specified hardware configurations offer at least the possibility of easing installation, administration, and support.

There also are marketing reasons to offer an appliance or something appliance-like.

• To various extents, Oracle, Teradata, Microsoft, IBM, Netezza, and EMC are all telling the world that your hardware should be optimized for your analytic DBMS.
• Smaller vendors such as Vertica and Aster Data also tend to cobble together some sort of appliance, in part so they don’t have to say they disagree.
• Thus, a “We don’t see any point in special hardware assembly at all” story would leave an analytic DBMS vendor pretty far out on a limb.

Finally, there are three overlapping technical trends that increase the need for storage-awareness in analytic DBMS. First and foremost is the rise of solid-state memory. For starters, I believe:

• Flash will be important for analytic DBMS soon.
• There are good technical reasons for this.
• Oracle’s marketing will make a big deal out of the Flash aspects of Exadata, so other analytic DBMS vendors will need a response. And of course, if Netezza or Teradata preemptively make a big deal of their Flash-based offerings, that just adds to the pressure for Flash adoption on everybody else.
• But it’s not just Flash – Flash, other solid-state memory, and disk will be combined in various ways.

But this move to Flash will require analytic DBMS vendors to be increasingly storage-aware for at least three reasons:

• It just adds another level of complexity to their hardware-balancing challenges.
• Flash overturns some of the fundamental assumptions of modern analytic DBMS, in particular:
  • Sequential reads are hugely better than random
  • The worst bottleneck is at the point where data comes out of storage.
• The Flash technology stack is still immature, and you have to pick your poison in how to deal with it. Vendors are making very different choices in this regard – and they do have to choose.

Another trend that could naturally lead analytic DBMS vendors to be more storage-aware is their incorporation of what could be viewed as hierarchical storage/ILM technologies. Different data is stored in different ways and/or on different kinds of storage hardware. (Vendors pursuing – you guessed it – different approaches to this include Teradata, Greenplum, Vertica, and Sybase.) The more automatic that process is, the more storage-aware the DBMS will need to be.

Finally, there are reasons to think that DBMS should be split between conventional servers and smart storage. This is, of course, the Exadata strategy. Netezza’s two-processor approach, while rather different, also somewhat validates the idea.

The acquisition of Greenplum will be an all-cash transaction and is expected to be completed in the third quarter of 2010, subject to customary closing conditions and regulatory approvals. The acquisition is not expected to have a material impact to EMC GAAP and non-GAAP EPS for the full 2010 fiscal year. Upon close, Bill Cook will lead the new data computing product division and report to Pat Gelsinger. EMC will continue to offer Greenplum’s full product portfolio to customers and plans to deliver new EMC Proven reference architectures as well as an integrated hardware and software offering designed to improve performance and drive down implementation costs.

Greenplum is one of my biggest vendor clients, and EMC is just becoming one, but of course neither side gave me a heads-up before the deal happened, nor have I yet been briefed subsequently. With those disclaimers out of the way, some of my early thoughts include:

• I wish my clients would never buy each other, but it’s inevitable.
• I don’t think anybody evaluating Greenplum should be much influenced by this deal one way or the other. (Whether they will be is of course a different matter.)
  • EMC tends to run its bigger software acquisitions in a fairly hands-off manner. There’s no particular FUD (Fear/Uncertainty/Doubt) reason why this deal should stop anybody from buying Greenplum software.
  • I also don’t think adding a rich parent adds much of a reason to buy from Greenplum. But if you’re the type who’s nervous about smaller vendors — well, Greenplum now isn’t so small.
• Greenplum Chorus could, in principle, work with non-Greenplum DBMS. That possibility suddenly looks a lot more realistic.
• The list of analytic DBMS vendors with an appliance orientation is pretty impressive, including:
  • Oracle, with Exadata
  • Microsoft, partially
  • Teradata
  • Netezza
  • Now EMC/Greenplum, at least partially
  • Weaker players such as:
    • The ailing Kickfire, which a client (not Kickfire itself) tells me is being shopped around
    • The reeling HP Neoview
    • XtremeData, but I’m still waiting to hear of XtremeData’s first real sale
• Greenplum is something of a specialist in large databases. EMC has to love that.
• Greenplum’s weakness is concurrency.
• Greenplum’s “polymorphic storage” is a good fit for a storage vendor with appliance-y ideas.
• And finally — I think that even software-only analytic DBMS vendors should design their systems in an increasingly storage-aware manner, and have been advising my vendor clients of same. I’ll blog that line of reasoning separately when I get a chance, and edit in a link here after I do.

Related links (edit)

• Here’s the promised post as to why analytic DBMS need to be ever more storage-aware.
• Dave Kellogg crunched the EMC/Greenplum numbers, coming up with an estimated valuation range of $3-400 million, the high end of which is rumored to be correct.
• Merv Adrian suggests the big EMC/Greenplum loser is ParAccel, a viewpoint which presumably presupposes that the EMC/ParAccel partnership was significant in the first place.
• I talked with Ben Werther and posted more about Greenplum and EMC.

• Netezza planned to be “generous” in its original TwinFin FPGA capacity, anticipating software upgrades like the ones it’s introducing now. It is satisfied that this strategy worked. More on this below.
• The same surely applies to CPU.
• What’s more, I get the sense that the CPU turned out in practice to be even more over-provisioned than they anticipated …
• … at least when one just considers Netezza’s base NPS software.
• However, I suspect that if the advanced analytics capability takes off, Netezza will determine that more CPU is always better.
• And by the way, NEC is making versions of Netezza appliances with more advanced chips than Netezza is. So if anybody should really, really need more CPU in their Netezza boxes, there’s a very straightforward way to make that happen. (And if there were nontrivial demand for that, appropriate support plans could surely be structured.)
• Everybody needs to be careful about RAM. Netezza is surely no exception.

The major parts of Netezza’s FPGA software are:

• Compress Engine 2. This is Netezza’s new way of doing compression.
• Compress Engine 1. This is Netezza’s old way of doing compression. It is being kept around so that existing Netezza tables don’t suddenly have to be changed or reloaded.
• Project Engine. Guess what this does.
• Restrict Engine. Ditto.
• Visibility Engine. This enforces ACID and handles row-level security. It is “sort of a corner of” the Restrict Engine (Actually, Netezza seems to waver as to whether to describe “Restrict” and “Visibility” as being two engines or one.)
• Miscellaneous plumbing.

If I understood correctly, each Netezza FPGA has two each of the engines in parallel.

Related link

• An August, 2009 post on what Netezza does in its FPGA

It might be best to start with some Netezza product introduction and naming housekeeping:

• Netezza isn’t changing the hardware on any of its existing systems at this time. Rather, Netezza’s product upgrades are contained in a software-only release …
• … except that it isn’t a “release,” but rather a “wave.” There are three points to that terminological distinction:
  • The advanced analytics part doesn’t depend on the new database platform software.
  • Individual functions in the advanced analytics part don’t necessarily depend on advances in the analytics platform.
  • It plays on the surfboard-centric naming of Netezza’s appliances.
• Netezza has wisely scrapped its prior plan to make its advanced-analytics capabilities be a chargeable add-on to it core appliance products. Rather, Netezza is going to offer advanced analytics as part of its core product. Part of the reason is that the interest in these capabilities is broader than Netezza first anticipated. The name for this is is something like i-Class advanced analytics capabilities.
• There is a “release” in all this too, namely NPS 6.0 (Netezza Performance Software). That’s the core DBMS technology.
• It’s all to be shipped in Q3.

Highlights of our NPS 6.0 conversation include:

• As promised, Netezza has improved its compression significantly. Because this was anticipated, this upgrade was planned for in the design of the systems Netezza started introducing last summer. Consequently, the reduction in I/O produced by compression translates almost directly into better performance – the silicon is now more fully loaded than it was before, but few if any actual silicon bottlenecks have been introduced by the I/O improvement.
• Netezza’s other big performance enhancement is the introduction of clustered base tables, which it says can reduce I/O by an order of magnitude or better.
• Netezza says that there are individual queries in which the enhancements take query performance up 30-40X. (Presumably, those would be ones for which clustered base tables are a big win.)
• More interestingly, Netezza says that overall performance is improved by >2X. That’s queries, load, backup, and everything else all blended together.
• Underpinning all this, Netezza went from 125 MHz to a blend of 125 and 250 MHz in its FPGA clock speeds. Also, the width of the FPGA onboard data path went from 16 to 32 bits. Netezza suggests that the naive calculation which says this could increase FPGA throughput 4X isn’t entirely misleading.
• Netezza is pretty content with its workload management capabilities for queries, but nonetheless keeps adding features. Workload management has not yet been extended to cover all the non-query parts of the analytic functionality.
• Netezza continues to enhance its cost-based optimizer and query planner.
• Netezza has long used an internal networking approach that’s rather different from TCP/IP. Netezza views TCP/IP’s strength as recovering gracefully if there’s congestion. However, Netezza would rather do whatever it takes to preclude congestion in the first place, except perhaps in rare edge cases. I’m not aware of what enhancements, if any, have been made to Netezza’s internal networking specifically in NPS 6.0.

The basic idea of clustered base tables (“base tables” are ones that are not, for example, materialized views) is to range partition in multiple dimensions at once. Then you rule out (as in don’t retrieve) all those blocks that fail a match in any one of the cluster dimensions. Netezza says its customers were doing a lot of work to simulate this benefit by multiple sorts; Netezza’s implementation will now handle that much more automatically. Netezza says that talking to customers revealed that 4-5 cluster dimensions was almost always the most somebody would need; they will ship support for 4. That makes sense. In most cases, you’d want to cluster on the answers to “W” questions – Who, What, Where, When (but probably not Why), in one dimension each. However, Netezza does call out as an ideal use case geospatial, precisely because 2 (or more rarely 3) dimensions each have “equal weight.”

I don’t know how other vendors implement clustered base tables, but in Netezza’s case it’s via a space-filling curve. (Actually, they called it a “Hilbert space-filling curve,” but I oppose that phrasing, as it’s apt to lead to extremely incorrect use of the term “Hilbert space.”) I.e., data is mapped to 4-tuples (say) in line with the dimensions, which are then sorted in a linear order in line with a space-filling curve. Happily, Netezza hasn’t experienced problems clustering columns that have particularly challenging cardinality or skew.

If I understood correctly, you can only zone map (and presumably cluster) on integers and dates right now, but that will change soon. (Edit: In blog comments and email, Tim Greenwood of Netezza explained to me that the NPS 6.0 workarounds to that were much more robust than I realized.)

Netezza put a lot of work for NPS 6 into something it calls “table grooming,” which amounts to recopying tables in more beneficial form. Uses for table grooming – which is a manually initiated process – include but probably aren’t limited to:

• Clustering tables and, as needed, reclustering them.
• Getting rid of data that was deleted. (Netezza has Postgres-style multiversion concurrency control – MVCC – but no time-travel, so keeping around deleted data is a waste of space.)
• Recompressing data from Compress Engine 1 to Compress Engine 2.
• Alter Table

The core ideas of table grooming include:

• The Netezza NPS software copies rows from one place to another.
• Netezza NPS then updates the appropriate metadata.
• Metadata updates are transactional, even though the actual data movement is not.

This can be done part of a table at a time. Reads and loads are unaffected by the process, or at least not blocked. Delete commits are indeed blocked during a reorg, but Netezza guesses that the blocks hold for a few minutes during the grooming of a clustered base table, 10-15 seconds if space is being reclaimed, and something similar for an Alter Table.

And finally, here are some notes on Netezza’s query optimization and planning.

• Netezza has a traditional cost-based optimizer, in which all operations have estimated costs, measured in microseconds, irrespective of which parts of the system (CPU, I/O, network, whatever) they most stress. (I have trouble imagining how a cost-based optimizer could work differently from that without incurring huge computational costs.)
• Netezza’s bottleneck is almost always disk I/O.
• Netezza’s optimizer is not/no longer based on the PostgreSQL optimizer.
• Netezza does a lot of query transformation. Key points include:
  • Netezza joins are usually very cheap.
  • Filtered scans are cheap too.
  • More expensive in Netezza are data redistribution (duh), sorts, and unfiltered scans.
  • Most expensive of all are intermediate result sets that don’t fit into memory.
• Specific examples of Netezza query transformation include:
  • Pushing predicates out to nodes.’
  • Flattening query trees and eliminating subqueries.
  • Rewriting windowed aggregates to be joins + grouped aggregates.
  • (New in 6.0) Transforming outer joins into other kinds.
• Netezza does real-time sampling to help with query planning. (But this is only worth doing for queries that are estimated to be expensive.) Zone maps (and clustering too?) are invoked as part of deciding where to sample. Sampling was for scans only prior to NPS 6.0, and will now be done for joins as well.

Related links

• Notes on this week’s spate of Netezza-related blog posts
• How Netezza (and IBM) do database compression
• Netezza’s silicon balance

After the first one, Netezza Marketing VP Tim Young took me aside for two pieces of constructive criticism. The surprising one* was that he felt I had been INSUFFICIENTLY critical of Netezza. Since then, every other conversation we’ve had about content creation has also featured ringing reassurances that Tim truly wants independent, non-pandering work.

*The unsurprising one was that I’d rushed. Well, duh. After months of telling me I had a 1 hour slot, Netezza cut me to ½ hour a few days beforehand. And my talk had been designed to be high-speed even in the longer time slot …

As a result, I accepted a subsequent gig from Netezza that I would barely consider from most other vendors. Namely, for this year’s Enzee Universe – June 21-23, aka Monday-Wednesday of this week, at the Westin Waterfront Hotel in Boston – I would do some contemporaneous blogging. The parameters we agreed on included: 

• I would just blog here on DBMS2, with Netezza allowed to reuse posts in their entirety on its site(s).
• I also would give a talk on the conference’s last day.
• I wouldn’t say much about conference sessions, because:
  • I’m not a session-attending kind of guy. (I wasn’t particularly good at sitting still in class in 8^th grade. I haven’t gotten much better since. And I have a huge aversion to other people’s uninterruptible PowerPoints.)
  • I think Netezza’s sessions are just as hype-filled as anybody else’s. (Much as I enjoyed traveling around the world with Netezza last year, it was painful hearing Jim Baum claim in city after city that Netezza boasts a 50X performance advantage vs. the competition.)
• Rather, I’d base things much more on individual conversations and meetings.
• Because I didn’t see how turnaround time could work otherwise, we’d have some of those meetings beforehand, and others early in the conference.

That last bit didn’t exactly wholly work out; for the second consecutive year Netezza pulled a surprise schedule switch a few days beforehand. But:

• I did have extensive, fascinating meetings at Netezza’s offices on Thursday, which were the fodder for multiple posts going up today.
• I have a nice meeting schedule set up for Tuesday.
• There should be plenty of opportunity for hallway and exhibit-floor conversation as the conference progresses.
• I even have my own private conference room, with a lovely name (the “Paine Room”).

So far as I know, the rest of the plan is still operative.

Posts already written as I draft this one include:

• A long discussion of Netezza’s technology, focusing on the database parts
• A discussion of Netezza’s and IBM’s compression strategies
• Notes on how Netezza balances its silicon and uses its FPGAs
• A quickie on data warehouse loading latency

I still need to write one focusing on Netezza’s advanced analytics strategy, and plan to edit in a link to it when it’s up.