• My most important post of the week was a general guide to IT vendor strategy. That one has already spawned discussion at many companies, from the tiny to the multi-billion-dollar.
• The best comment thread of the week was probably on my post about scale-out relational OLTP choices, in which people discussed the merits of various particular alternatives.
• I recommended that people strongly consider attending XLDB 5 in Menlo Park on October 18-19.

Most of the posts, however, were reactions to news events. In particular:

• Teradata announced that Teradata 14 will be hybrid-columnar, more in Vertica’s way than in Greenplum’s or Aster Data’s. (Pay no attention to the Wall Street Journal’s apparent belief that no other analytic DBMS is hybrid-columnar at all.)
• Aster announced the unsurprising news that there will be a Teradata Aster appliance. Also, Aster talked about greater analytic flexibility in the forthcoming Aster 5.0.
• With Oracle OpenWorld coming up, Oracle decided to get some of its announcing out of the way early. In particular, it announced the Oracle Database Appliance, which is small-business-friendly hardware for running the Oracle DBMS. However, the Oracle Database Appliance doesn’t seem to do much about the complexity of running the Oracle DBMS software.
• In a catch-all Hadoop post, I noted that:
  • Oracle has now clearly said it has a Hadoop appliance coming, no doubt next week at OpenWorld.
  • I still can’t see why Hadoop appliances would succeed, but a lot of smart folks seem to disagree with me.
  • Greenplum announced what looks like a nice but unimportant little product upgrade.
  • It’s a really good thing that previously reported plans to revamp Hadoop are underway.
• DataStax announced that it really is a Cassandra company after all. Pay no attention to previous marketing that seemed to put DataStax in the same Hadoop-alternative category as, say, MapR.
• Ingres has changed its name to Actian. The announcement seems like a confession that Ingres and VectorWise are going nowhere.

• Changing its name to Actian.
• Deemphasizing Ingres, the product.
• Emphasizing a set of products that don’t exist yet (or at least aren’t shipping), namely lightweight mobile apps that are business-intelligence-plus-an-action, and technology for building them. These are called “Action Apps”, and are discussed on the Actian company blog.
• Positioning all this as something to do with “big data” (what a shock).

It turns out that Actian was the name of an ancient athletic competition commemorating Augustus’ defeat of Anthony at Actium, a battle that was more recently memorialized in the movie Cleopatra. Frankly, I think Cleopatra Software might have been a more interesting company name, although that could mean execs would have to arrive at sales calls rolled up in a carpet.

One article said:

Greg Wood, chief financial officer for Actian, told V3 that while the firm would continue to develop and maintain the Ingres database platform, its would be placing the spotlight on its Cloud Action Platform and its line of Action Apps.

“The Ingres database is well-recognised and we will continue to support it, but at the same time that brand was more associated with an older-generation technology,” Wood said.

“We think Actian better reflects where we are going as a company, particularly the application strategy.”

Wood explained that the platform would look to expand on the emerging field of big data applications by adding functionality for end users. The small, specialised applications would link up with data analytics tools, providing alerts and actions when various conditions are spotted within a database.

So what about VectorWise? Notwithstanding Actian’s stated focus on “big data”, I think VectorWise’s chances for market success are slim.* Reasons include:

• The market for shared-disk columnar analytic DBMS is crowded (Sybase IQ, Infobright, SAND). Those vendors also have to compete with MPP columnar analytic DBMS offerings from Vertica and ParAccel.
• I’ve never heard anything to make me believe VectorWise is getting significant market traction.
• Indeed, Daniel Abadi’s well-known flirtation with the idea of using VectorWise in HadoopDB/Hadapt excepted, I don’t recall any marketplace mention of VectorWise at all.

*The possibility of some kind of Action App synergy leads me to elevate them to “slim” from “none”.

The Action App idea actually sounds cool, but it’s quite a change from Ingres’ previous positioning and technology, and I have no basis for judging it as likely to succeed. On the other hand, companies have occasionally made successful transitions into business intelligence from relatively unrelated businesses before, most notably Cognos in the mid-1990s.

• Hadapt is additional software on a cluster that also runs fully functional Hadoop/HDFS. (Cloudera Hadoop more than straight-from-Apache Hadoop to date, but that’s not a requirement.)
• The cluster also runs a DBMS on every node, such as PostgreSQL or one of Infobright/Vectorwise.
• Hadapt’s software manages parallel SQL queries by distributing them to the DBMS living on each node. Hadapt says that the resulting query performance far outshines Hive’s.
• Hadapt further says that, by exploiting the partner DBMS, its SQL functionality outpaces Hive’s as well.
• Target Hadapt use cases are centered around keeping machine-generated or other poly-structured data in Hadoop, and extracting, enhancing, or otherwise deriving some of it to live in the relational store.
• In particular, Hadapt seems like an interesting choice when you want to use that relational data as you work on other data that’s still in HDFS, or if you want to keep using the relational data in other kinds of MapReduce jobs.
• That all fits well with my thoughts about the importance of derived data.

Other evolution from what I wrote about Hadapt a few months ago includes:

• Hadapt is in beta now.
• Hadapt has added adult supervision in the form of Philip Wickline, late of Endeca.

In other news, Hadapt is our newest client.

*At least if the underlying DBMS is a fast one. Hadapt likes VectorWise for that purpose, and is showing performance comparisons that assume VectorWise is underneath.

**It seems that Hadapt in the future is assured of having more SQL coverage than Hive does today.

It’s still early days for the Hadapt company. Funding is on the angel level. There seem to be six employees — Yale professor Daniel Abadi, CEO Justin Borgman, Chief Scientist Kamil Bajda-Pawlikowski,* and three other coders. The Hadapt product will go into beta at an unspecified future time; there currently are a couple of alpha users/design partners. The Hadapt company, a Yale spin-off, obviously needs to move from Connecticut soon. I wasn’t able to detect any particular outside experience in the form of directors or advisors. And Hadapt’s marketing efforts are still somewhat ragged. So basically, the reasons for believing in Hadapt pretty much boil down to:

• Daniel Abadi is a star.**
• Hadapt’s own tests show that Hadapt is a whole lot faster than Hive.

*Bajda-Pawlikowski is one of the two Abadi students who did the HadoopDB work. It turns out he had numerous years of coding experience before entering graduate school. (The other student, Azza Abouzeid, is pursuing an academic career.)

**Vertica was built around Daniel’s C-Store Ph.D. thesis. He was involved in H-Store as well. He has a really good blog. He’s a really nice guy. Etc.

As you might have guessed from the name, the Hadapt guys are proud that their technology is “adaptive,” which communicates their fond belief that Hadapt’s query optimization and planning are more modern and cool than other folks’ query planning and optimization. In particular, Daniel suggested that Hadapt is more thoughtful than most DBMS are about looking at the size of intermediate result sets and  then replanning queries accordingly.

However, the really cool adaptivity point is that Hadapt watches the performance of individual nodes, and takes that into account in query replanning. Daniel asserts, credibly, that this is a Really Good Feature to have in cloud and/or virtualized environments, where Hadapt might not have full control and use of its nodes. I’d add that it could also give Hadapt a lot of flexibility to be run on clusters of non-identical machines.

On the negative side, Hadapt will not at first have any awareness of how its underlying DBMS are optimized; it will plan for VectorWise the same way it does for PostgreSQL. In that regard, this is a DATAllegro 1.0 story. If I understood correctly, Hadapt has specific connectors for a couple of DBMS (probably exactly those two), and can also talk JDBC to anything. PostgreSQL was apparently 5X faster than MySQL when tested (with either ISAM or InnoDB); Daniel snorted about, for example, MySQL’s apparent fondness for nested-loop joins over hybrid hash. On the other hand, he was more circumspect about his reasons for favoring VectorWise over, to name another open source columnar DBMS, Infobright.

And finally, a couple of other points:

• Hadapt will be closed source, although it will of course rely on large amounts of other people’s open source software. Pay no attention to the importance Daniel previously ascribed to HadoopDB’s open source nature.
• Hadapt decompresses data before moving it from node to node, and also before doing non-SQL MapReduce operations on it. Pay no attention to the years Daniel spent insisting columnar DBMS absolutely must operate on data in compressed form.

• VectorWise is indeed a shared-everything analytic DBMS.
• The VectorWise front-end is Ingres. Ingres VectorWise supports almost all SQL that Ingres does (there are a few edge-case exceptions).
• Conversely, Ingres VectorWise doesn’t support any SQL Ingres doesn’t, most notably SQL-99 Analytics. Naturally, SQL-99 Analytics is a roadmap item for Ingres/VectorWise.
• Ingres VectorWise 1.0 is pretty purely columnar. There’s a bit of PAX, but it’s mainly automagic/under the covers. The one user-controlled exception I understood was that one can ensure that composite keys are stored together.
• The main Ingres VectorWise performance secret sauce ingredients we touched on were:
  • Vectorization of operations (hence VectorWise’s name).
  • Compression that is tuned for speed rather than to minimize storage utilization.
• We unfortunately didn’t have time to revisit the other big part of the Ingres VectorWise performance story, namely clever design for modern microprocessor architectures. High-level generalities about that do pervade the Ingres VectorWise press release, but – well, they’re very high level.
• Unlike Vertica but like most other columnar DBMS vendors, Ingres VectorWise wants you to store your data once. You can index-organize the data. You can also organize multiple tables in the same order, to make joins among them fast.
• Support for actual join indexes is an Ingres VectorWise roadmap item.
• As do ever more analytic DBMS, Ingres VectorWise has something akin to Netezza zone maps.
• When I asked Peter what had changed most from the initial VectorWise development plan, other than the above, he basically said that their performance priorities had shifted a bit. Specifically, he said.
  • They had originally been “blinded” (his word) by the TPC-H benchmark, but figured out that they were overly focused on it. (Well, duh.)
  • They learned about the importance of other things such as data loading speeds.

]]> http://www.dbms2.com/2010/06/11/ingres-vectorwise-technical-highlights/feed/ 1 http://www.dbms2.com/2010/06/10/vectorwise-press-release/ http://www.dbms2.com/2010/06/10/vectorwise-press-release/#comments Thu, 10 Jun 2010 11:38:39 +0000 Curt Monash http://www.dbms2.com/?p=2242 Ingres forgot to prebrief me on the VectorWise announcement, and despite valiant efforts hasn’t succeeded in connecting with me since they realized the lapse. Meanwhile, I took a look at the VectorWise press release, and found the quotes to be somewhat amusing.

Daniel Abadi was quoted as saying:

“VectorWise … is now proven to be a terrific fit as database technology underpinning large-scale HadoopDB applications.”

notwithstanding that HadoopDB has never been used for a production application, large-scale or otherwise. Unsurprisingly, Daniel is quite emphatic in noting he didn’t actually say that.*

*Daniel actually said “has proven,” which to him means “[not] officially proven, but proven in a more colloquial sense.”

Meanwhile, a quote from the Rohatyn Group starts:

“For the past 20 years, we’ve been searching for the killer database …”

notwithstanding that the Rohatyn Group hasn’t been around for even 10 years, let alone 20. (I infer this from the LinkedIn profile of a “founder” who’s been working there only since 2002.) However, Peter Boncz assures me via e-mail that this is a genuine quote even so, and speculates quite reasonably that this database quest started at some other entity.

Otherwise, the quotes are the usual stuff, with highlights including:

• At least some queries run much faster on VectorWise than they do on a non-analytic DBMS.
• A couple of system integration partners are enthused about how happy their VectorWise customers will be if and when they ever have any.
• Ingres and VectorWise executives think their product is really cool.

Related link

• A press release quote that REALLY went awry

• Analytic queries commonly return only a fraction of all possible columns.
• Only returning the columns needed
  • Saves I/O
  • Saves cache space
  • Reduces processing
  • Facilitates compression
• Presumably all those row-based MPP vendors just went row-based because they had a fine row-based DBMS (usually but not always PostgreSQL) to build on.

Pushbacks to this argument from row-based vendors include:

• Yes, but it’s harder to update a column store
• Yes, but there are more steps to retrieving a bunch of columns than there are to retrieving the same information from row stores

plus generous dollops of:

• We’re doing just fine, thank you
• We’re not seeing column stores much in the marketplace
• Don’t believe all that academic hype
• Column stores reek of elderberries, and are powered by hamster wheels

(OK, I made that last one up, but I do hear the other claims frequently.)

However, there are at least two ways in which row- and column-stores are beginning to come together. First, there are lots of rumors about row-store vendors bringing out column-store options, even beyond the recent Ingres/VectorWise announcement. (But anything I may know about same beyond noticing the rumors fly by is surely under NDA.) Second, column-store vendors Vertica and VectorWise are bringing out a kind of row/column hybrid storage option.

Vertica 3.5 introduces what Vertica calls “FlexStore.” A key part of FlexStore is the ability to store data not just in pure columnar format, but also to group columns together in what amounts to sub-rows. This is advantageous when data is retrieved together and, I presume, when it is updated. There’s a tradeoff in giving up column stores’ compression advantages, however, and use of this feature is not recommended for columns that are frequently retrieved independently. Vertica also notes that since it typically uses 1 megabyte block sizes, any table smaller than that shouldn’t be broken into columns at all.

VectorWise, of course, doesn’t have a product right now, but has gotten a bunch of recent publicity around the column-store product it plans to ship via its partner Ingres in 2010. When I asked Peter Boncz about row/column hybridization inside VectorWise (not federating between Ingres and VectorWise, but rather truly within VectorWise), he said one of the storage options was PAX, and pointed me at a 2001 paper by a group of academics that includes the ubiquitous Dave Dewitt. PAX turns out to stand, in creative spelling, for Partition Attributes Across.

The PAX idea is to store as many rows of data as can fit into a block, but within the block store them in columns. This preserves some of the compression and cache-efficiency benefits of column stores, while also bringing back whole rows in a single step. (I think Vertica’s FlexStore does something similar to this, but I’m not sure.)

Further confusing things, Peter Boncz of VectorWise told me VectorWise can support “any hybrid” of columnar storage and PAX.

Bottom line: The distinction between row- and column-stores isn’t going to go away any time soon, but it is at least beginning to blur a bit.

• By “integrating Vertica and MapReduce,” Vertica means “integrating Vertica and Hadoop.”
• Vertica’s Hadoop integration is based on Cloudera’s DBInputFormat.
• Omer called out for me several features of Vertica’s Hadoop integration that didn’t just come from Cloudera, namely:
  • Cloudera’s DBInputFormat assumes the database runs on a single computer, or a single head node of an MPP system. Vertica’s technology, however, runs on peer parallel nodes with no head, and so Vertica adapted the DBInputFormat technology accordingly.
  • Vertica lets you push down Map functions to the database. Omer reports a roughly even division among users and prospects between those who want to do this and ones who don’t.
  • Vertica lets you do Reduce functions (or Map functions, if you don’t push them down to the database) on a separate cluster than you run the database software. Vertica asserts that its customers and prospects all want to do this. Right here is the big difference between Vertica’s MapReduce integration and Aster’s or Greenplum’s. (Aster would also say that Vertica’s weaker MapReduce/SQL programming integration is a big difference as well.)
  • Indeed, Vertica lets you Reduce into a different DBMS than Vertica, if you choose.
  • Vertica gives you flexibility on the size of the Map and Reduce clusters. Omer agreed with me when I said there were some limits on how fast one can add or subtract nodes in a Vertica grid, because there’s data redistribution involved. But one can add/change/delete Hadoop clusters extremely quickly.

Apparently, the use cases for Vertica/Hadoop integration to date lie in algorithmic trading and two kinds of web analytics. Specifically:

• One or more Vertica customers are using MapReduce in production to do relatively simple transforms of web log data
• Vertica customers are experimenting with — but have not yet put into production — more sophisticated pattern analysis of web log data.
• Financial services customers are using MapReduce for a lot of experimentation in discovering new algorithms. The idea is that DBMS/MapReduce integration offers rapid prototyping of algorithmic ideas. Those that pan out are then reimplemented for production, presumably in some kind of CEP (Complex Event Processing) system. These users seem to be ones that are pushing down a lot of Map functions to the Vertica DBMS.

By the way, Vertica is based on C-Store, the Ph.D. thesis project of Daniel Abadi, who recently wrote:

To me, it is far more efficient from a performance and a “green” perspective to push the computation to the data. Hence, I am not a fan of decoupling the compute grid and the data grid.

Not coincidentally, Daniel also recently wrote that

If the VectorWise/Ingres solution does get released open source, I believe they will be an excellent column-store storage engine for HadoopDB. I have already requested an academic preview edition of their software to play with.

The VectorWise guys also told me they are looking forward to seeing how the two projects work together.

• VectorWise, the product, will be an open-source columnar analytic DBMS. (But that’s not quite true. Pending productization, it’s more accurate to call the VectorWise technology a row/column hybrid.)
• VectorWise is due to be introduced in 2010. (Peter Boncz said that to me more clearly than I’ve seen in other coverage.)
• VectorWise and Ingres have a deal in which Ingres will at least be the exclusive seller of the VectorWise technology, and hopefully will buy the whole company.
• Notwithstanding that it was once named something like “MonetDB,” VectorWise actually is not the same thing as MonetDB, another open source columnar analytic DBMS from the same research group.
• The MonetDB and VectorWise research groups consist in large part of academics in Holland, specifically at CWI (Centrum voor Wiskunde en Informatica). But Ingres has a research group working on the project too. (Right now there are about seven “highly experienced” people each on the VectorWise and Ingres sides, although at least the VectorWise folks aren’t all full-time. More are being added.)
• Ingres and VectorWise haven’t agreed exactly how VectorWise and Ingres Classic will play together in the Ingres product line. (All of the obvious possibilities are still on the table.)
• VectorWise is shared-everything, just as Ingres is. But plans — still tentative — are afoot to integrate VectorWise with MapReduce in Daniel Abadi’s HadoopDB project.

The MonetDB project is led by Martin Kersten, with whom I chatted at SIGMOD in June (standing up and not taking notes, so I may have some details wrong). I get the impression, based on that conversation, my VectorWise call, and other data:

• Martin has been researching analytic DBMS (mainly but not only relational) since the late 1970s, and has been based at CWI since 1985.
• Peter Boncz has been either second in command of that crew or close to it.
• Martin Kersten, Peter Boncz, and the CWI/MonetDB team in general have gotten all sorts of computer science glory for their work.
• Martin has enjoyed generously stable government research funding for his group, but has found commercialization of the technology more difficult than he might at, stay, Stanford. The figure of 15 MonetDB researchers comes to mind, although I see from Martin’s bio that he oversees a team of ~55 in total.
• One early attempt at commercializing MonetDB turned into a company called Data Distilleries that was sold to SPSS. Peter Boncz was chief architect of Data Distilleries.
• Besides VectorWise, there are at least two other recent spin-off companies from the MonetDB project. One is a zero-headcount shell, set up to facilitate MonetDB project members (and others) consulting to users of the open source MonetDB technology. The other is in stealth mode, focusing on some vertical market.

I further get the impression that VectorWise was actually Marcin Zukowksi’s Master’s Ph.D project, with Peter Boncz being his advisor. VectorWise also boasts another Peter Boncz student, who wrote about updating column stores.

As one might expect from the name, VectorWise does vector processing. I.e., the hard part of Marcin’s work was developing vectorized algorithms for one SQL operation after another. Vectorization, pipelining, and FPGAs might all seem to go together — XtremeData certainly seems to think so — but the VectorWise folks preferred to develop for Intel CPUs anyway, for pretty much the usual reasons. Another major theme is trying to get the right things into CPU cache, because in their opinion RAM cache is just sooooo painfully slow.

Our discussion of VectorWise’s compression was interesting. Highlights included:

• The design requirement is that decompression work at a rate of 3 gigabytes/second or so. That way the system is faster overall than if it operated at 1 gigabyte/second on uncompressed data, which I gather is the alternative.
• VectorWise takes 4-5 steps CPU cycles to decompress a tuple.
• VectorWise says it sacrificed compression ratio to achieve speed. That said, VectorWise claims 3-4X compression on TPC-H data, which is no worse than what ParAccel reported, and enjoys higher compression rates on other kinds of data.
• VectorWise decompresses data before manipulating it, and claims that the advantages of operating on compressed data are only significant if — like Vertica but apparently unlike VectorWise — the database stores columns in multiple sort orders each.
• VectorWise’s compression is mainly on numerical and numerical-like (e.g. date) datatypes. An exception is that VectorWise uses dictionary compression on string data when it makes sense to do so.

Other notes include:

• VectorWise has technology akin to Microsoft SQL Server’s Shared Scans, in which multiple queries that require similar table scans don’t have to repeat all the redundant scanning work. I need to get better at figuring out which other analytic DBMS do similar things.
• While VectorWise hasn’t yet been open-sourced, its code is in the hands of some other academic institutions, used mainly for computer science research (as opposed to, say, as a data store for some kind of scientific experiment).
• VectorWise’s scalability has only been tested up to eight cores.