• Technology it already has released*, but has not published any reference architectures for
• A Barney partnership**

In other words, Vertica has succumbed to the common delusion that it’s a good idea to put out half-baked press releases the week of TDWI conferences. But if we look past that kind of all-too-common nonsense, Vertica is highlighting an interesting technical story, about how the analytic DBMS industry can exploit solid-state memory technology.

*Upgrades to Vertica FlexStore to handle Flash memory, actually released as part of Vertica 4.0

** With Fusion I/O

To set the context, let’s recall a few points I’ve noted in the past:

• Solid-state memory’s price/throughput tradeoffs obviously make it the future of database storage.
• The Flash future is coming soon, in part because Flash’s propensity to wear out is overstated. This is especially true in the case of modern analytic DBMS, which tend to write to blocks all at once, and most particularly the case for append-only systems such as Vertica.
• Being able to intelligently split databases among various cost tiers of storage – e.g. Flash and disk – makes a whole lot of sense.

Taken together, those points tell us:

For optimal price/performance, analytic DBMS should support databases that run part on Flash, part on disk.

While all this is a future for some other analytic DBMS vendors, Vertica is shipping it today.* What’s more, three aspects of Vertica’s architecture make it particularly well-suited for hybrid Flash/disk storage, in each case for a similar reason – you can get most of the performance benefit of all-Flash for a relatively low actual investment in Flash chips: 

• Vertica lets you split tables by column, and Vertica FlexStore is versatile enough to let you put only the most-used columns in Flash. (Vertica offers a figure that 85% of usage calls on only 15% of columns, but I don’t know how rigorously grounded those numbers are.)
• To the extent that Vertica data is more compressed than many of Vertica’s competitors’ (which it probably is, debates over the magnitude of Vertica’s advantage notwithstanding), the total storage-hardware cost of sticking stuff in Flash is less when you use Vertica than with other systems.
• Vertica has relatively less need for temp space than some other systems. (Vertica uses figures of <20% of total storage, vs. 30%+ for some other systems.) If you want to use Flash for temp space, so as to accelerate your toughest queries, that can save you some cash …
• … and by the way, temp space is an especially good use of Flash, because temp space is accessed in a less sequential manner than data storage is.

The least obvious of those points are about temp space; I only understood the particulars when Vertica development chief Shilpa Lawande explained them to me Thursday.

* At least in theory; customer adoption may be a different matter.

But before drilling down on temp space, let me first note that there’s one offsetting factor to all those “We need somewhat less Flash than the other guys” Vertica advantages. Like all serious databases, a Vertica installation keeps two or more copies of all data, to that there’s no storage single point of failure. In a flexible system like Vertica, you can put one copy on Flash and one on disk. But if you do that in Vertica, you forgo fully exploiting one possible benefit of Vertica’s architecture – the ability to store different copies of a column in different orders, which are beneficial for accelerating different groups of queries.*

*More precisely, you don’t get the full benefits of Flash acceleration for every query touching those columns.

OK. Back to temp space. There are four kinds of things you can put in storage if you’re running a database management system:

• The software itself.
• Persistent data. (I.e., tables, if the DBMS you’re running is relational.)
• Metadata, especially the kind that lets you find data – indexes, zone maps, catalogs, etc.
• Temporary data constructs built as part of, say, a sort-merge join. These, by definition, are what populate temp space.

Just to be clear, those constructs are NOT temporary tables of the sort created by, say, Microstrategy; such tables are handled like any other data. Rather, they are ephemeral creations and, so far as I can tell, not tables at all.

Vertica offered two theories as to why its DBMS requires less temp space than competitors do:

• To the extent data is decompressed before being operated on in memory by the DBMS, that decompression would of course also apply to temp space as well. Vertica prides itself on keeping data compressed all the way through, and seems to get away with smaller temp space allocations as a benefit.
• Since Vertica can store columns in expedient sort orders, it does less sorting overall, and sorting is a big use of temp space.

Obviously, no matter which DBMS you use, the amount of temp space you need is surely workload-dependent. Even so, Vertica’s claim to something of an advantage seems legit.

Truth be told, I’m not convinced the savings involved are great enough to matter a whole lot – but it’s a fun subject to think through.

And finally: One of my biggest surprises since starting to look at analytic-DBMS-on-Flash has been the centrality of temp space. Talking to Vertica Thursday, I finally uncovered a key reason why: Temp space tends to be accessed via multiple streams of data at once. I’m still struggling with WHY that is true, with two reasons suggested being:

• Temp space can be accessed by multiple operations at once. (But isn’t that also true of the rest of storage?)
• Merge sorts, a common use of temp space, read multiple streams of data. (Couldn’t you tweak your software to make that not be true?)

But if we grant that temp space naturally is accessed in multiple places at once – well, that’s a lot like random I/O, and if you’re doing a lot of random reads, you’d love to use something other than spinning disk.

]]> http://www.dbms2.com/2010/08/16/vertica-flash-temp-space/feed/ 10 http://www.dbms2.com/2010/06/21/data-warehouse-load-latency/ http://www.dbms2.com/2010/06/21/data-warehouse-load-latency/#comments Mon, 21 Jun 2010 12:15:17 +0000 Curt Monash http://www.dbms2.com/?p=2319 I took advantage of my recent conversations with Netezza and IBM to discuss what kinds of data warehouse load latency were practical. In both cases I got the impression:

• Subsecond load latency is substantially impossible. Doing that amounts to OLTP.
• 5 seconds or so is doable with aggressive investment and tuning.
• Several minute load latency is pretty easy.
• 10-15 minute latency or longer is now very routine.

There’s generally a throughput/latency tradeoff, so if you want very low latency with good throughput, you may have to throw a lot of hardware at the problem.

I’d expect to hear similar things from any other vendor with reasonably mature analytic DBMS technology. Low-latency load is a problem for columnar systems, but both Vertica and ParAccel designed in workarounds from the getgo. Aster Data probably didn’t meet these criteria until Version 4.0, its old “frontline” positioning notwithstanding, but I think it does now.

Related link

• Just what is your need for speed anyway?

• VectorWise is indeed a shared-everything analytic DBMS.
• The VectorWise front-end is Ingres. Ingres VectorWise supports almost all SQL that Ingres does (there are a few edge-case exceptions).
• Conversely, Ingres VectorWise doesn’t support any SQL Ingres doesn’t, most notably SQL-99 Analytics. Naturally, SQL-99 Analytics is a roadmap item for Ingres/VectorWise.
• Ingres VectorWise 1.0 is pretty purely columnar. There’s a bit of PAX, but it’s mainly automagic/under the covers. The one user-controlled exception I understood was that one can ensure that composite keys are stored together.
• The main Ingres VectorWise performance secret sauce ingredients we touched on were:
  • Vectorization of operations (hence VectorWise’s name).
  • Compression that is tuned for speed rather than to minimize storage utilization.
• We unfortunately didn’t have time to revisit the other big part of the Ingres VectorWise performance story, namely clever design for modern microprocessor architectures. High-level generalities about that do pervade the Ingres VectorWise press release, but – well, they’re very high level.
• Unlike Vertica but like most other columnar DBMS vendors, Ingres VectorWise wants you to store your data once. You can index-organize the data. You can also organize multiple tables in the same order, to make joins among them fast.
• Support for actual join indexes is an Ingres VectorWise roadmap item.
• As do ever more analytic DBMS, Ingres VectorWise has something akin to Netezza zone maps.
• When I asked Peter what had changed most from the initial VectorWise development plan, other than the above, he basically said that their performance priorities had shifted a bit. Specifically, he said.
  • They had originally been “blinded” (his word) by the TPC-H benchmark, but figured out that they were overly focused on it. (Well, duh.)
  • They learned about the importance of other things such as data loading speeds.

]]> http://www.dbms2.com/2010/06/11/ingres-vectorwise-technical-highlights/feed/ 1 http://www.dbms2.com/2010/05/23/sybase-iq-15/ http://www.dbms2.com/2010/05/23/sybase-iq-15/#comments Sun, 23 May 2010 08:34:28 +0000 Curt Monash http://www.dbms2.com/?p=2186 Back in March, Sybase was kind enough to give me permission to post a slide deck about Sybase IQ. Well, I’m finally getting around to doing so. Highlights include but are not limited to:

• Slide 2 has some market success figures and so on. (>3100 copies at >1800 users, >200 sales last year)
• Slides 6-11 give more detail on Sybase’s indexing and data access methods than I put into my recent technical basics of Sybase IQ post.
• Slide 16 reminds us that in-database data mining is quite competitive with what SAS has actually delivered with its DBMS partners, even if it doesn’t have the nice architectural approach of Aster or Netezza. (I.e., Sybase IQ’s more-than-SQL advanced analytics story relies on C++ UDFs  — User Defined Functions — running in-process with the DBMS.) In particular, there’s a data mining/predictive analytics library — modeling and scoring both — licensed from a small third party.
• A number of the other later slides also have quite a bit of technical crunch. (More on some of those points below too.)

Sybase IQ may have a bit of a funky architecture (e.g., no MPP), but the age of the product and the substantial revenue it generates have allowed Sybase to put in a bunch of product features that newer vendors haven’t gotten around to yet.

More recently, Sybase volunteered permission for me to preannounce Sybase IQ Version 15.2 by a few days (it’s scheduled to come out this week). Sybase IQ seems to be focused on large part on the government/intelligent market, with three major features being:

• A kind of data federation, querying external databases, that makes sense mainly in the context of rigorous security rules. (I find that confusing, since Sybase IQ’s indexes tend to hold all the information in the database, but I didn’t push the point.)
• An upgrade to Sybase IQ’s built-in text indexing. I doubt anybody would confuse this with best-of-breed text search, but evidently that intelligence community is satisfied with less. But even before 15.2, Sybase IQ could do both LIKE and WHERE CONTAINS searching.
• Improved LOB (Large OBject) management.

One part of my Sybase IQ conversations I haven’t blogged yet in much details is scale-out, concurrency, and “multiplexing.”

• Sybase feels that Sybase IQ’s competitive sweet spot, especially in terms of performance, is reached when there are 20 or more concurrent queries.
• In general, Sybase asserts that a shared-everything architecture is great for concurrency — just run different queries on different boxes, all against the same data.
• The ability to use a bunch of boxes run Sybase IQ is called “multiplexing.”  This is a chargeable option, without which one is limited to a single SMP box.
• Just under 20% of the top 250 Sybase IQ customers have multi-node scale-out configuration (vs. single-node SMP scale-up). And around 8% have it overall.
• Sybase IQ nodes can be heterogeneous (e.g., in compute power).
• Sybase IQ nodes can be dedicated to be read-only, or can be read-write. Indeed, Sybase IQ nodes can change roles dynamically, for example becoming write-only during nightly batch load. (I didn’t clarify whether all this applies just to nodes-as-boxes, or if some parts apply to specific processors or cores within the same box.)
• Sybase noted that data mart outsourcers can offer differentiated SLAs (Service Level Agreements) depending upon which nodes they give which customers access to.
• Most Sybase IQ installations start at 8 cores or more. The Sybase IQ Small Business Edition, limited to 4 cores, is not a big seller.
• Sybase IQ has a straightforward round-robin load-balancing story via third-party technology.

Finally, along the way in the discussions I picked up various tidbits about the Sybase IQ user base. Unfortunately, Sybase is pretty vague in discussing database sizes — are they user data? Are they compressed? What do the numbers mean? With that huge caveat:

• By some metric or other, a couple of classified customers are approaching petabyte scale.
• The largest commercial Sybase IQ customer — a credit card company — has a couple hundred terabytes or so.
• The largest financial services Sybase IQ databases are 50-70 terabytes. This sounds low, frankly, so maybe those are compressed figures, with user data being 200+ terabytes. But I’m just speculating there.
• Sybase IQ has a little less than 100 customers in the “data aggregator” market, which is a lot like what I call “data mart outsourcer.”
• Sybase IQ’s ILM technology is a chargeable option, with Sybase being “cautious” about sales. Compliance is a big market driver for it.
• Sybase IQ’s #1 vertical market is financial services. Other biggies are government, telecom, marketing services, and to some extent retail.
• As of February, there were 40-45 production users of Sybase IQ 15.0 and 15.1.

For starters, Sybase IQ is not just a bitmapped system, but it’s also not all that closely akin to C-Store or Vertica. In particular,

• Sybase IQ stores data in columns – like, for example, Vertica.
• Sybase IQ relies on indexes to retrieve data – unlike, for example, Vertica, in which the column pretty much is the index.
• However, columns themselves can be used as indexes in the usual Vertica-like way.
• Most of Sybase IQ’s indexes are bitmaps, or a lot like bitmaps, ala’ the original IQ product.
• Some of Sybase IQ’s indexes are not at all like bitmaps, but more like B-trees.
• In general, Sybase recommends that you put multiple indexes on each column because — what the heck – each one of them is pretty small. (In particular, the bitmap-like indexes are highly compressible.) Together, indexes tend to take up <10% of Sybase IQ storage space.

Sybase IQ is not immune to Sybase’s confusing choices in version numbering. Thus:

• Sybase IQ Version 15.2 will be announced and released soon.
• Sybase IQ Version 15.1 was a set of “binary replacements” rather than an “upgrade release” for Sybase IQ Version 15.0.
• Sybase IQ Version 15.0 was launched in February, 2009 and released for general availability some time thereafter.
• The prior version of Sybase IQ was 12.7.
• GA isn’t always GA, and some language localizations and so on weren’t ready for a while. Consequently, a lot of Sybase IQ sales continued to be of Version 12.7 even in the second half of 2009.

Now let’s get down to some technical particulars.

Sybase IQ columns are always stored in RowID order. However, RowIDs are logical and not physical, and hence take up little disk space. A small amount of per-page metadata lets you find the specific cell you want. (Cells are commonly fixed-width, in which case finding the cell of choice is a simple calculation.) So RowIDs are not much of an I/O overhead issue, although I’m not sure at what point they get unpacked and start needing to be carried around as the data travels through silicon.

Sybase IQ has 9 or so kinds of indexes. The choice of index has a lot to do with cardinality. In the extreme low-cardinality case, a simple bitmap might do. With intermediate cardinality, you might go to a modified kind of bitmap – e.g., if there there are 2^16 possible values, you can represent a value in 16 bits, and bitmap operations are approximately 16 times as costly as if the number of possible values were only 2^1. For very high cardinality, there’s a B-tree-like index called “High Group”.

Note: Surely every Sybase index name, at some time, made sense to at least one engineer.

Sybase IQ’s execution engine does seem to rely quite a bit on bitmaps. E.g., intermediate query results are stored as bitmaps, which helps them play nicely with each other and with many of the indexes. Sybase claims that Sybase IQ’s bitmap orientation often makes WHERE clauses execute very quickly. Sybase IQ reoptimizes queries after WHERE clauses are evaluated. Complex expressions are, when possible, evaluated once per unique value, not once per row.

Speaking of unique values – Sybase IQ’s compression story doesn’t currently match that of some other columnar products, but it seems to stack up pretty well against row-based systems. In particular:

• Sybase says IQ compression is most commonly 50-70%.
• Sybase further says that, in most cases, compression falls into the range 40-85%.
• Page-level LZ compression is decompressed upon read (duh).
• Dictionary/token compression may be decompressed later. For example, GROUP BYs are commonly done on tokens, and JOINs sometimes are.

Sybase IQ boasts pipelining, in two senses. First, IQ tries to read pages for multiple queries at the same time. Second, Sybase IQ tries to prefetch pages into cache before they’re needed. Sybase points out that these prefetched pages have the WHERE clauses already executed, and that no extra baggage is being dragged into cache that doesn’t need to be there.

Highlights of Sybase IQ’s update and load story include:

• Sybase IQ is optimized for large bulk loads. No surprise there.
• Sybase IQ has several options for microbatching and/or trickle feeds.
  • The coolest is Sybase RAP.
  • More generally, microbatching is based on Change Data Capture. Sybase has various ETL/replication technologies, creating a confusing array of options in that regard.
  • Sybase says that one customer is microbatching 1000s of rows with 1 minute latency.
• There’s something about snapshotting and hence loads not interfering with queries. I’m not clear on the details.
• Assuming you have enough parallelism, you can dedicate some nodes to queries while others are dedicated to load. (Recall that Sybase IQ is shared-disk.)

I’ve lost track a little bit as to which “advanced analytics” functionality is in Sybase IQ 15.1, which will be in 15.2, and what’s a future beyond that, which is a great excuse for me to leave it out of what has already become a rather long post. But anyhow, except perhaps for the future stuff and/or some time series functionality, none of it seems terribly advanced. Sybase IQ does have two stored procedure languages, namely the ones for Sybase ASE (T-SQL) and for Sybase Anywhere or Adaptive Server Anywhere or whatever it’s called this week (Watcom SQL, which Sybase asserts is similar to the ANSI SQL stored procedure language).

Similarly, I’ll leave a lot of other stuff out as well, and for now stop here.

Related links

• I haven’t repeated every detail here from my August, 2009 technical post about Sybase IQ
• And here’s more about Sybase IQ, including some Sybase IQ 15.2 features, some market penetration info, and a slide deck

That said,  

• Raj said that Sybase’s Aleri acquisition was, if anything, tracking ahead of expectations.
• Raj didn’t seem the slightest bit focused on the Coral8/Aleri CEP-based BI strategy that John Morell had long championed.
• Raj reminded me that Sybase SQL Anywhere has numerous OEMs, not just on the true desktop/laptop or smaller, but also in a return to its server/workgroup roots. Sybase SQL Anywhere even added geospatial indexing recently.

Raj also spoke glowingly of SAP’s in-memory database technology and the potential for Sybase of same — until I asked a follow-up question. At that point, he confessed that he didn’t really know much about about SAP’s in-memory database technology yet. As I said before, I believe SAP is fairly sincere about its belief that its in-memory database technology will conquer the world — but this is a naive and poorly-founded opinion even so.

One tidbit I did get is that SAP’s in-memory database technology is not just son-of-T-REX. A Korean (Raj thinks) company SAP had acquired is also in the mix. Raj also had the impression SAP’s in-memory technology can do rows, columns, or hybrid structures. On the one hand, that makes sense. On the other, it’s not a perfect fit with what Hasso Plattner said last year.

• Sybase’s main businesses are:
  • Classic OLTP DBMS (Sybase ASE, for Adapative Server Enterprise, unless I’ve missed yet another name change).
  • Analytic technology — mainly Sybase IQ, but more generally Sybase RAP.
  • Mobile technology. (The frequently renamed small DBMS SQL Anywhere was the foundational product of and still is included in the mobile division.)
• SAP’s thoughts on in-memory database management are interesting. However, I think SAP’s oft-repeated claim that it has a lot of important in-memory database technology to bring to Sybase (or for that matter SAP customers) is mainly smoke and mirrors. Cool data access methods, good niche database products, and broadly applicable multi-domain DBMS innovations are three different things. Granting that SAP probably has the first and thinks it has the second is not the same as giving it much credence for having the third.
• SAP claims that, 15 years after its refusal to support Sybase turned Sybase into a DBMS also-ran, it by now is “relatively simple” to port SAP’s apps to Sybase ASE, and that they will make that happen. I actually believe that SAP’s apps will soon run on Sybase ASE, where by “soon” I mean “in a couple of years for no-apologies general availability.” (Certifying a DBMS for SAP is a long process.) The main missing features — e.g., row-level locking — were already put into Sybase back in the last millenium. Nor could there be fundamental architectural problems that keep SAP from supporting Sybase ASE, or else SAP couldn’t have supported Microsoft SQL Server (which, long ago, was a Sybase fork).
• I don’t see any market or competitive dynamics that would lead the SAP acquisition to hurt Sybase’s ASE or mobile businesses. General merger management mishegas is, of course, always a possibility.
• SAP Business Objects partners with Sybase IQ’s competitors. That could be a problem. However, coopetition is pretty strong in the business intelligence market. I don’t think any of SAP Business Objects, IBM Cognos, or Oracle Business Intelligence are much held back from partnering by competitive dislike of their parent companies.
• The rest of SAP might be able to drum up some extra business for Sybase IQ.
• It would be natural for IBM/Cognos to now buy a columnar DBMS of its own. Vertica is an obvious first choice. ParAccel would surely come much cheaper. Since ParAccel has little chance of surviving as an independent company — too immature and too little differentiation to overcome that — I’d expect ParAccel’s board to jump at the chance to sell out.
• It would be interesting if SAP Business Objects would revive the CEP-based BI idea.
• I gather Sybase’s AnswersAnywhere concept network/object model-based natural language/speech recognition technology never went anywhere. Unsurprising (it seemed like it needed too much hand-building to scale semantically), but regrettable even so.
• I don’t see anything in this acquisition that would revive PowerBuilder (Sybase’s Visual Basic competitor), Sybase’s CASE (Computer-Aided Software Engineering) tools, and so on.
• And on the personal side — I’ll probably lose Sybase as a customer due to this merger, but it could have been worse. A lot of vendors smaller than Sybase are bigger customers for Monash Research.

Edit: Right after I posted this, I saw email from Sybase clarifying that Sybase’s in-memory technology, while slightly influenced by some ANTs IP Sybase bought non-exclusive rights to, is essentially home-grown. That’s what I thought, but the call sounded like it was saying something different.

Further coverage of SAP/Sybase:

• SAP believes in database proliferation
• More quick reactions to SAP/Sybase
• Technical basics of Sybase IQ
  

• Vertica plans to build a massive, world-conquering sales force.
• If Vertica dips back into negative cash flow to do that and has to raise more venture capital, so be it.
• “Triple-digit” revenue growth is expected for this year.

*I’ve since heard more both from Ralph and his former colleagues, and I’m comfortable taking the move more or less at face value — for some reasons he doesn’t want to spell out, Ralph really wanted to move back home to South Africa.

While they were at it, Vertica also put out a press release reporting very good success in the social gaming market. The biggest and best known of the bunch is Zynga. Three months ago, Wayne Eckerson had figures of 3 TB/day added to the database, 200 nodes, and >40 million users. Now Zynga is using a figure of >65 million daily users and 230 nodes. More precisely, at Zynga:

• There are two Vertica databases with identical data.
• Each Zynga Vertica database runs on 115 nodes.
• Zynga’s two Vertica database clusters are used for different applications.
• It’s undisclosed exactly what Zynga runs on what Vertica cluster. But best practice would be to put mission-critical, fast-response stuff on one cluster, and use the other for longer-running or less-critical queries — plus have it be available as hot standby — given that I don’t see much reason to put data geographically close to users around the world for reasons of latency or whatever.
• An undisclosed amount of data, amounting to all of what Wayne earlier estimated at 3 TB, is added to each of Zynga’s Vertica databases daily.

In other news, Vertica now states its customer count as being >130.

Highlights on the market share/sector side include:

• Infobright’s customer base grew 500% over the past year, to 120 paying customers.
• This included end users (60%), as well as ISVs and SaaS providers (40%) who embed Infobright’s DBMS in their application.
• During the same period, Infobright’s open source software was downloaded 35,000 times.
• The end user applications were heavily clustered around web and online analytics tracking, with a focus on understanding customer behavior on the web.
• Infobright also continues to see the growth of application-specific data marts.
• There is also continued interest and growth in using Infobright technology to analyze IT logs and telecom CDR (Call Detail Record) data, to identify fraud or security issues, to understand and improve network performance, and other purposes.

Product highlights include:

• Infobright be much more transparent in 2010 about its plans.
• Infobright will start posting and commenting on future releases and themes in March of this year. (However, they haven’t run much of that by me yet, and we’re past the middle of March.)
• Infobright expects to drop 3-4 interim releases for every major release, with at least two major releases in 2010.
• Some of Infobright’s major improvements this year will be:
  • Continued SMP performance improvements “without the need for complex hardware configurations or administrative effort”.
  • Extending the “hit rate” of the Knowledge Grid, which is central to Infobright’s performance story.
  • Better international support with UTF-8 extensions.

For starters, there’s a lot of new analytic functionality. This isn’t Aster/Netezza-style ambitious. Rather, there’s a lot more SQL-99 functionality, plus some time series extensions of the sort that financial services firms – an important market for Vertica – need and love. Vertica did suggest a couple of these time series extensions are innovative, but I haven’t yet gotten detail about those.

Perhaps even more important, Vertica is cleaning up a lot of its previous SQL optimization and execution weirdnesses. In no particular order, I was told:

• Vertica’s delete performance is up “literally” 30-100X, at least in the case of “large” deletes. Performance for “large” updates has been enhanced as well.
• Vertica has finally cleaned up all vestiges of its prior bias to star schemas. For example, Vertica concedes that its product previously would sometimes force a star execution plan that wasn’t really appropriate.
• It is no longer the case that you need to define projections before you load a table into Vertica. This is now fully automatic.
• Vertica 4.0 automatically redesigns the database when new nodes are added to the system.
• When a database designer does hand-tune projections – and there’s no shame in this still being a possibility in Vertica 4.0 – that hand-tuning is now pulled back into the automatic generation/recommendation/whatever wizards for further projections. I.e., there’s a kind of DBA round-trip engineering going on.
• Vertica used to require that tables being joined be identically “segmented” (I think this means distributed across joins). That is no longer the case in 4.0.
• In connection with this new-found flexibility, Vertica now supports full outer joins directly, rather than requiring the left outer join/right outer join/UNION kluge.
• The Vertica 4.0 optimizer is smarter than its predecessor about things like predicate pushdown into subqueries, or exploiting commonality between predicates and partition keys.
• There’s a fundamental change that I don’t understand very well in the Vertica execution engine basic unit of work. It sounds as if in the past all the disk-based data containers the query needed got opened at once and read into memory, whether or not there was enough RAM and CPU cores to handle them, and this problem has now been fixed.
• Vertica always seemed to say that you could query immediately on new data, because even if it hadn’t hit disk yet – the ROS (Read-Optimized Store) – it was available in memory – the WOS (Write-Optimized Store). And queries were in essence federated between the ROS and WOS. But apparently it’s a new feature in Vertica 4.0 that you can read totally fresh data without locking. I confess to not understanding this very well either. (It has something to do with what  Vertica calls “Epochs”.)
• Temporary tables can now be created in Vertica on a local/session basis without any DDL. Make temporary tables easier and more performant is important for a variety of reasons:
  • Microstrategy, Company V* et al. use lots of temp tables. E.g,, Company V on Vertica has 3000 permanent tables and 5-7000 temporary ones.
  • Vertica rightly points out that temporary tables are also important for ELT (Extract/Load/Transform).
  • Vertica further says that single-node OEMs such as security appliance vendors use lots of temp tables.

*Company V = one of the more prominent vertical-market application providers.

In other Vertica highlights:

• It sounds as if 4.0 is the first Vertica release with what I would regard as serious workload management.
• While Vertica has stored and retrieved Unicode since Vertica 3.5 or so, 4.0 will be the first Vertica release in which Unicode is sorted and collated properly.
• Stored-procedure-like functionality is still a future for Vertica.