For starters, there’s a lot of new analytic functionality. This isn’t Aster/Netezza-style ambitious. Rather, there’s a lot more SQL-99 functionality, plus some time series extensions of the sort that financial services firms – an important market for Vertica – need and love. Vertica did suggest a couple of these time series extensions are innovative, but I haven’t yet gotten detail about those.

Perhaps even more important, Vertica is cleaning up a lot of its previous SQL optimization and execution weirdnesses. In no particular order, I was told:

• Vertica’s delete performance is up “literally” 30-100X, at least in the case of “large” deletes. Performance for “large” updates has been enhanced as well.
• Vertica has finally cleaned up all vestiges of its prior bias to star schemas. For example, Vertica concedes that its product previously would sometimes force a star execution plan that wasn’t really appropriate.
• It is no longer the case that you need to define projections before you load a table into Vertica. This is now fully automatic.
• Vertica 4.0 automatically redesigns the database when new nodes are added to the system.
• When a database designer does hand-tune projections – and there’s no shame in this still being a possibility in Vertica 4.0 – that hand-tuning is now pulled back into the automatic generation/recommendation/whatever wizards for further projections. I.e., there’s a kind of DBA round-trip engineering going on.
• Vertica used to require that tables being joined be identically “segmented” (I think this means distributed across joins). That is no longer the case in 4.0.
• In connection with this new-found flexibility, Vertica now supports full outer joins directly, rather than requiring the left outer join/right outer join/UNION kluge.
• The Vertica 4.0 optimizer is smarter than its predecessor about things like predicate pushdown into subqueries, or exploiting commonality between predicates and partition keys.
• There’s a fundamental change that I don’t understand very well in the Vertica execution engine basic unit of work. It sounds as if in the past all the disk-based data containers the query needed got opened at once and read into memory, whether or not there was enough RAM and CPU cores to handle them, and this problem has now been fixed.
• Vertica always seemed to say that you could query immediately on new data, because even if it hadn’t hit disk yet – the ROS (Read-Optimized Store) – it was available in memory – the WOS (Write-Optimized Store). And queries were in essence federated between the ROS and WOS. But apparently it’s a new feature in Vertica 4.0 that you can read totally fresh data without locking. I confess to not understanding this very well either. (It has something to do with what  Vertica calls “Epochs”.)
• Temporary tables can now be created in Vertica on a local/session basis without any DDL. Make temporary tables easier and more performant is important for a variety of reasons:
  • Microstrategy, Company V* et al. use lots of temp tables. E.g,, Company V on Vertica has 3000 permanent tables and 5-7000 temporary ones.
  • Vertica rightly points out that temporary tables are also important for ELT (Extract/Load/Transform).
  • Vertica further says that single-node OEMs such as security appliance vendors use lots of temp tables.

*Company V = one of the more prominent vertical-market application providers.

In other Vertica highlights:

• It sounds as if 4.0 is the first Vertica release with what I would regard as serious workload management.
• While Vertica has stored and retrieved Unicode since Vertica 3.5 or so, 4.0 will be the first Vertica release in which Unicode is sorted and collated properly.
• Stored-procedure-like functionality is still a future for Vertica.

If a recent Vertica press release is to be believed, Sybase got clobbered. The meat is:

…  Sybase has admitted that under the claim construction order issued by the Court on November 9, 2009, “Vertica does not infringe Claims 1-15 of U.S. Patent No. 5,794,229.” Sybase further acknowledged that because the Court ruled that all the remaining claims in the patent (claims 16-24) were invalid, “Sybase cannot prevail on those claims.”

For those counting along at home — the patent only has 24 claims in total.

I have no idea whether Sybase can still cobble together grounds for appeal, or claims under some other patent. But for now, this sounds like a total victory for Vertica.

Edit: I’ve now seen a PDF of a filing suggesting the grounds under which Sybase will appeal. Basically, it alleges that the judge erred in defining a “page” of data too narrowly. Note that if Sybase prevails on appeal on that point, Vertica has a bunch of other defenses that haven’t been litigated yet. It further seems that Sybase may have recently filed another patent case against Vertica, in a different venue, based on a different patent.

One annoying blog troll excepted, is anybody surprised at this outcome?

Anyone who has met with us knows ParAccel shies away from hype.

But like many things ParAccel says, that is not true.

The latest whoppers came in the form of several customers ParAccel listed on its website who hadn’t actually bought ParAccel’s DBMS, nor even decided to do so. It is fairly common to to claim a customer win, then retract the claim due to lack of permission to disclose. But that’s not what happened in these cases. Based on emails helpfully shared by a ParAccel competitor competing in some of those accounts, it seems clear that ParAccel actually posted fabricated claims of customer wins.

Another thing that was both technically and substantively false was ParAccel’s claim to be CERTIFIED price-performance leader. Obviously, this was meant to give the impression that ParAccel had been “certified” as the leader in price/performance, when the closest thing to that that was remotely true was that ParAccel had a leading position in the category of “price/performance measurements that happen to have a certification process.” At least, that was true for a short time; then ParAccel’s certification was found to have been erroneous, and got revoked, which did not however inspire ParAccel to immediately take the claim off the front page of its website.

ParAccel’s website also reflects a lot of praise from flagship customer LatiNode. What it perhaps understandably neglects to mention is that LatiNode is in a dormant state, placed there by acquirer Elandia due to LatiNode’s criminally corrupt customer acquisition practices.

I also don’t believe ParAccel’s endlessly-repeated claim that is has never lost a benchmark on performance. However, I must in fairness note that while I’ve been given names of customers who are supposed counterexamples to this claim by somebody I trust, I’ve never been able to actually verify those supposed ParAccel losses.

Vertica offers a post on its 3.5 release, with a riff on the popular theme “We’ve fixed some weaknesses in our prior versions that we didn’t previously say we had.” More important, Vertica is pretty clear on the virtues of its hybrid columnar architecture.

Speaking of which — Oracle is going true hybrid columnar as well. I don’t have details or timing, however.

Dave Kellogg of Mark Logic wrote in to amusedly point out  Oracle’s anti-MarkLogic collateral. The very first charge Oracle levies is that MarkLogic goes beyond the emerging XQuery standard to add additional functionality. Considering Oracle’s approach to SQL standards, I tend to share Dave’s amusement.

Bill Conniff of Xponent LLC wrote in to tell of a vastly cheaper and less functional approach to XML management, apparently geared to looking at very large XML files one at a time.

Cayuga is a Cornell research project in complex event processing (CEP). There’s a Cayuga academic home page, a Sourceforge page for some open source Cayuga CEP code, and so on. Minsheng Hong, writing from a Vertica email address, tipped me off some months ago. The basic idea seems to be to do lots of queries very quickly, rather than a smaller number of queries over and over again. Whether this is an advance in anything but open-sourceness over Apama or Aleri I couldn’t say, but I do think it’s a different focus than that of StreamBase or pre-Aleri Coral8.

And finally, editor Doug Henschen listed his 15 favorite Intelligent Enterprise blog posts of 2009 — four each by Seth Grimes and Doug himself, three by Cindi Howson, two by me,* and one each by Mark Smith and Neil Raden.

*Doug selects up to three posts a month from here to republish.

The top two points from Q&A probably were:

• Big Data and the cloud actually have relatively little to do with each other, a few exceptions notwithstanding, especially if the data is in a shared-nothing DBMS (as opposed to, say, a MapReduce-oriented file cluster). Two principal reasons are:
  • Redistributing data from node to node is a little slow, undermining some of the elasticity benefits of the cloud.
  • Getting data into the cloud in the first place is a lot slow.
• The NoSQL movement is a lot like the Ron Paul campaign — it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don’t have a whole lot in common with each other.

Anyhow, here are my notes for the talk, edited in just a couple of places for readability or linkage.

Quick introduction

• Big Data vs. cloud
• How big is Big Data?
• At the low end of that range, there’s little you can’t do with conventional technology if you have:
  • An unlimited budget for hardware
  • An unlimited budget for software
  • An unlimited budget for people, especially Oracle DBAs

Big Data in OLTP

• Hard-core OLTP
  • Focus of DBMS technology for a long-time
  • Big budgets because each transaction has significant value
  • Tough to get users to change technologies
• Lighter-weight OLTP
  • Classic example = web companies
    • Big ones — retail-oriented ones (eBay, Amazon) partially excepted — rolled their own technology stacks
    • Reluctant to give money to anybody
      • Open source, etc.
  • Difficulty finding market
    • Product vs. feature
      • Clustering/HA/DR/whatever
      • Ditto cloud enablement
    • True products haven’t found much traction yet

Analytic Big Data use cases

• Kinds of data for analytics
  • More of same != big
  • More detail and/or new kinds
    • Complete data sets
    • Transactions
    • Call details
    • Tick/trade history
    • Web clickstreams
    • Network event logs
    • Other machine-generated data
    • CAM bottom line
      • Anything human-generated should and will be retained in its entirety
      • Quantities of machine-generated data retained should and will grow roughly in line w/ computing cost reductions (Moore’s Law, etc.)
• Analytic uses of Big Data
  • Analytics is mainly about three things
    • Problem detection
    • Customer relationship improvement
      • (Those overlap when the customer relationship is bad)
    • Financial statements on steroids
  • Main kinds of analytics
    • What BI vendors traditionally sell
      • General reporting and dashboards
      • Ad-hoc query (now driven from those reports and dashboards)
      • Planning (allegedly integrated with BI)
    • Research
      • Ad hoc relational query (worth mentioning twice because it drives so much of the market)
      • Data mining
      • Most web search and web mining
    • Operational/near-real-time
    • Archiving/compliance
  • What gets Big?
    • Mainly research and archiving
    • But when reporting or operational get Big, you have really interesting computing problems

Technology issues and trends

• Moore’s Law
  • CPUs — All about cores, hence parallelism is key
  • RAM
  • SSDs – hence replace disks
  • Sensors – hence generate lots more data
• Kryder’s Law
  • But rotational speeds up only 12.5X since Eisenhower Administration
  • Hence solid-state memory (or RAM) will soon take over
• In the mean time, I/O bottlenecks have had to be beaten
  • Hence sequential scans
  • Hence index-light architectures
  • Hence columnar
• DBMS “overhead”
  • Raw license and maintenance fees – software increasing fraction of total
  • OLTP vestiges – locking and all that
  • DBAs
    • People costs = huge fraction of total
    • Index-lightness addresses
    • So does appliance
  • Many people don’t really know how to write SQL
• Configuration
  • Appliance/tightly-balanced
    • Netezza
    • Teradata earlier
    • Greenplum/Sun
    • Oracle
    • IBM
    • Microsoft/Madison
  • Commodity/do what you want
    • Vertica
    • Greenplum now
    • Infobright, Aster and others
    • MapReduce-oriented file systems
  • Extreme rigidity is silly
    • Teradata, Oracle have both signaled moving to more modularity
    • Big driver of that = heterogeneous storage
      • Cheap disk
      • Expensive disk
      • Solid-state
      • RAM
    • CPU/storage ratio is even more of a driver

Theoretically defensible ways to segment the market

• Latency requirements
  • High availability and low latency go together
• Query types
  • Simultaneous users for same
• Database size
• Budget

Actual segments right now

• Utter ADW/EDW
• Data mart
  • Size
  • Naturally columnar vs. naturally row-based
• Operational/frontline
• Less dramatic/smaller EDW

• Like Infobright, Calpont’s InfiniDB is a columnar DBMS consisting of a MySQL front end and a columnar storage engine.
• Community edition InfiniDB runs on a single server.
• One of commercial/enterprise edition InfiniDB’s main claims to fame will be MPP support.
• There’s no announced time frame for commercial edition InfiniDB.
• InfiniDB’s current compression story is dictionary/token only, with decompression occurring before joins are executed. Improvement is a roadmap item.
• Indeed, InfiniDB has many roadmap items, a few of which can be found here. Also, a great overview of InfiniDB’s current state and roadmap can be found in this MySQL Performance Blog thread. (And follow the links there to find performance discussions of other free analytic DBMS.)
• One thing InfiniDB already has that is still a roadmap item for Infobright is the ability to run a query across multiple cores at once.
• One thing free InfiniDB has that Infobright only offers in its Enterprise Edition is ACID-compliant Insert/Update/Delete. (Note: I wish people would stop saying that Infobright Enterprise Edition isn’t ACID-compliant, since that point was cleared up a while ago.)
• InfiniDB has no indexes or materialized views.
• However, InfiniDB’s retrieval is expedited by something called “Extents,” which sounds a lot like Netezza’s zone maps.

Being on vacation, I’ll stop there for now. (If it weren’t for Tropical Storm/ depression Ida, I might not even be posting this much until I get back.)

General SenSage highlights include:

• SenSage used to be known as Addamark.
• SenSage used to characterize itself as being in the Security Information Management (SIM) market.
• Now SenSage characterizes itself (approximately) as selling technology built around a columnar DBMS that happens to be pretty good at log analysis, compliance, and/or archiving.
• More concisely, SenSage says it is in the event data warehouse category. (The same could arguably be said of Splunk.)
• SenSage says it has >400 paying customers, of which ~200 are direct.
• SenSage has >120 employees and, like Splunk, is profitable.
• SenSage has enjoyed >50% annual revenue growth the past four years.
• Some SenSage deals are in the multiple-million dollar range.
• A major SenSage channel partner – dozens of installations — is SAP, which resells SenSage software on HP hardware is a “Compliance Log Warehouse.”
• A hot market for SenSage is CDRs (Call Detail Records).
• SenSage says that, among analytic DBMS vendors, it competes with Oracle, IBM, Teradata, Netezza and, to some extent, Vertica and Greenplum.

Technical SenSage highlights include:

• SenSage’s core technology is an append-only columnar DBMS, with no master node.
• SenSage’s DBMS uses no indexes and requires “no” database administration.
• SenSage’s database is range-partitioned, with the range-partition key always being time.
• SenSage has something it calls SQO (Sparse Query Optimization), which sounds a lot like Netezza zone maps. SQO never yields a false negative on whether data is in a block, never yields a false positive on equality predicates, and only rarely yields a false positive on range predicates.
• SenSage’s database uses large block sizes – typically 250,000 records/block, at 200-250 bytes per record. (That’s in the range of 64 megabytes/block.)
• SenSage says its software can load 10-50,000 records/second/node. If I’m doing the arithmetic correctly, that’s roughly 7-40 gigabytes/node/hour.
• SenSage collects log data into its event data warehouse in what it characterizes as an agentless manner. Even so, it seems that for a majority of kinds of data sources one does have to write custom agents. The two other ways to get data into SenSage – and presumably most of the data volume comes through these – are:
  • File transfer in the usual way
  • syslog
• SenSage says its software can read 100s of data sources, and that this is a huge competitive advantage. I’m not totally sure how that jibes with the prior point.
• SenSage says it gets 5X compression on CDR data, 10-20X on other kinds of logs. That’s not too far off from Vertica’s compression figures.
• SenSage says that it has datatype-aware compression as well as more standard stuff, with VARCHAR compressing particularly well.
• In particular, SenSage uses both dictionary/token and delta compression.
• SenSage’s software is pretty agnostic with respect to storage kind – DAS (Direct Attached Storage), SAN (Storage-Area Network), or content-addressable. In particular, there’s only about a 4% performance hit for using content-addressable storage.
• When using WORM (Write Once Read Many) storage like EMC’s Centera, SenSage leaves record locator information behind on ordinary storage and otherwise queries the WORM storage just like it queries anything else.
• SenSage says it has been using MapReduce since “Day 1”.
• Probably not coincidentally, you can use Perl and other aggregates in SenSage SQL statements.
• Perhaps also not coincidentally, SenSage says it has a number of advanced built-in analytic functions, including some focused on sessionization.

In addition to all that, SenSage offers a built-in event processing engine, consisting of:

• A finite-state machine correlation engine.
• A proprietary event processing language.
• A GUI to “abstract” (i.e., generate?) the event processing language.

The SenSage event processing engine is used to generate alerts. Data that comes into SenSage actually is passed to two places at once, namely to both the event processing engine and the database itself.

*I gather from the company that they forgot to tell Merv pricing was NDA.

** Merv cited a price as “starting” that I believe to be top-of-the-line. No criticism of Merv is implied in that; Kickfire has not been very clear in communicating hard numbers.

All that said, if one takes Kickfire’s marketing statements literally, Kickfire list pricing is around $20-50K per terabyte for a few small, fixed, high-performance configurations. That’s all-in, for plug-and-play appliances. What’s more, that range is based on the actual published user data capacity numbers for various Kickfire models, which I think are low for several reasons:

• Kickfire doesn’t officially admit that its model with 14.4 terabytes of disk can manage more than 6 terabytes of data, even though it clearly can.
• Actually, those 14.4 terabytes of disk can be increased or lowered as you choose.
• The basic compression figures implied in those calculations seem conservative.
• Compression figures are a lot more conservative yet, in that Kickfire assumes you’ll have a lot of actual indexes on your data. I’m not sure that’s necessary for most workloads.

Highlights include:

• Column orientation is a special case of what Greenplum is calling Polymorphic Data Storage.*
• As per product management chief Ben Werther’s blog post, what Greenplum’s polymorphic data storage boils down to is that you can store different tables in different storage paradigms. This is transparent to the SQL or any other API; it’s just a performance choice.
• Indeed, Greenplum lets you store different partitions of the same table in different storage and/or compression schemes. So Greenplum now has a kind of ILM (Information Lifecycle Management) story, although it doesn’t offer the faster vs. cheaper storage media differentiation options of Sybase IQ or Vertica.
• Greenplum now has, depending on how one counts, three or four main types of table:
  • Traditional PostgreSQL, which has been available since Day One
    
  • Row-oriented append-only (compressible and scan-optimized), available since Greenplum 3.2 (July, 2008)
  • Columnar append-only (new in Greenplum 3.3.4, shipping now)
  • External, in which Greenplum treats something external – in a relational DBMS or otherwise – as if it were a Greenplum table
• Greenplum offers multiple versions of LZ (Lempel-Ziv) and gzip compression, any of which you can choose on a table-by-table or partition-by-partition basis.
• Greenplum offers the same compression algorithms for both row-oriented and column-oriented tables.
• Greenplum says that compression is typically at least 50% better (i.e., to 2/3 as much space) in columnar vs. row storage, for the same algorithm.
• Just as it doesn’t offer columnar-specific compression algorithms, Greenplum also doesn’t sport other columnar features Daniel loves, such as in-memory compression or late materialization. (But then, VectorWise doesn’t do in-memory compression either, and Daniel likes VectorWise.)
• All the Greenplum choices I’ve mentioned have to be made manually by DBAs.
• Similarly, I doubt Greenplum can match Vertica’s engineering for getting updates and trickle feeds quickly into a column store – a traditional columnar Achilles heel that Vertica has invested a lot of effort to circumvent.

*The term “polymorphic” is somewhat, shall we say, overloaded these days.

*Guys like Greg Rahn and Kevin Closson are huge assets to Oracle, which is absurdly and self-defeatingly unhelpful through conventional public/analyst relations channels.

Six key tweets by Greg Rahn said:

I think the HCC 10x compression is a slideware (common) number. Personally I’ve seen it in the 12-17x range on customer data…

This was on a dimensional model. Can’t speak to the specific industry. I do believe Oracle is working on getting industry #s.

As far as I know, Exadata HCC uses a superset of compression algorithms that the commonly known column stores use…

…and it doesn’t require the compression type be in the DDL like Vertica or ParAccel. It figures out the best algo to apply.

The compression number I quoted is sizeof(uncompressed)/sizeof(hcc compressed). No indexes were used in this case.

Exadata HCC is applicable for bulk loaded (fact table) data, so a significant portion (size wise) of most DWs.

Summing up, that seems to say:

• Oracle claims 12-17X compression on a kind of data similar to that on which Vertica — which also uses 10X as a single-point overall compression marketing estimate where needed — claims 20X.
• Oracle selects compression algorithms automagically.
• Oracle’s compression doesn’t quite apply to all the data. Actually, this may be more of an issue for the caching benefits of compression than for the I/O or disk storage gains. (If you join a retail transaction fact table to a customer dimension table, and you have a lot of customers, fitting the uncompressed customer table into RAM could be problematic.)

Omer and I happened to have a call scheduled to discuss MapReduce yesterday evening, but wound up using most of the time to talk about Vertica’s compression and physical layout features instead. Highlights included:

• Greg, like many Vertica competitors, was wrong about Vertica requiring manual, low-level DDL (Data Description Language) for — well, for much of anything. Vertica does all that automatically, at least in theory, and suggests that in real life you can indeed often get by without manual intervention.
• Vertica can do trickle feeds into its compressed columnar storage. Greg seemed to suggest Oracle Exadata can not. (However, I won’t be surprised if, when his comments are expanded to more than 140 characters, he winds up saying the opposite. )
• Omer characterized the lowest latency with which you can get data into Vertica and have it be available for query as “seconds”, vs. “minutes” for other columnar vendors.
• Vertica recommends often keeping multiple copies of a column, for high availability and/or performance. This is not directly reflected in compression estimates. In particular, if you’re going to keep redundant copies of data for data-safety reasons anyway, Vertica recommends that you:
  • Run queries against more than one copy of the data, for performance/throughput.
  • Store different copies of the columns in different sort orders — e.g., according to different likely join keys — so that the copies are optimized for performance on different classes of queries.
• Vertica doesn’t have indexes.
• Vertica sorts columns on ingest. This sorting is, of course, commonly based on attributes from columns other than the one being sorted. Even so, Omer maintains that sorting helps compression, because of the correlation between columns. Examples (and I didn’t get these all from him) might include:
  • City/postal code
  • Customer_ID/store location
  • Customer_ID/product_ID
  • Product_ID/price
• Vertica, based on the recent introduction of FlexStore, has an ILM (Information Lifecycle Management) story much like Sybase IQ’s. E.g., you can keep different data ranges for different columns on fast storage, while the rest of the data is relegated to slower/cheaper equipment.