• Aster Data showed me a lot of customer names and deal sizes, across a bunch of industries (mainly enterprise rather than web). Yes, Aster’s market success is for real. (But almost all those details are NDA.)
• Sybase’s product plans for IQ are pretty impressive. (But the most interesting parts are, you guessed it, NDA.)
• I’ve kissed and made up* with ParAccel, now that they’ve replaced their CEO, replaced their marketing chief, and stopped the worst of the marketing nonsense I used to complain about. ParAccel has some interesting plans for ParAccel 3.0 which are, naturally, NDA.
• The Peoplesoft guys are doing it over again at Workday. Only this time, their platform isn’t a relational DBMS. Rather, it’s an in-memory, completely object-oriented data model, with disk used only on a “Just in case the power ever goes out” basis. (Thankfully, nothing at all about our conversation was NDA.)
• I’m finally feeling good about Northscale’s memcached-compatible persistent store Membase. The main reason is that they showed me a near-term path to interfaces that are richer than key-value. Also, Todd Hoff reassured me that even pure persistent memcached has a place.
• Rumor says that even the one app for which Facebook was using Cassandra — in-box search — has been decommissioned. On the other hand, numerous other scale-0ut DBMS (SQL or otherwise) seem to have Facebook footholds. But details are — all together now! — NDA.

*If you know ParAccel’s new marketing chief Michael Weir, you surely guessed I mean that only in a figurative sense.

From elsewhere:

• Daniel Abadi offered his analysis of Kickfire’s demise. In general I agree, but Daniel neglected to mention one hugely important factor — the chicken-egg negative effect of Kickfire’s lack of market or marketing traction. Customers were extremely reluctant to buy from Kickfire because they perceived, correctly, that Kickfire’s survivability was far from assured.
• While the InfiniDB forums suggest that there are at least a couple of production users of Calpont’s free InfiniDB, Calpont seemingly has a long way to go to be even as successful as Kickfire. But Calpont does have a bit of money to spend on lead generation; maybe some day they’ll even have actual customers.
• In a response to a question I messaged over, XtremeData tells me they have actual customers now. Press releases to follow.
• The admiration for the job Mark Hurd did at HP is in my opinion overstated. Sure, the financial/operational management appeared to work, but HP did little on Hurd’s watch to strengthen its reputation or customers’ loyalty. In particular:
  • HP’s analytics efforts have accomplished little.
  • HP’s data warehouse appliance efforts have failed pathetically.
  • From what I hear, HP’s execution in its Exadata partnership was not good.
  • HP’s server business in general is distinguished mainly by HP being a big company.
  • HP’s EDS acquisition has been rocky, not that EDS was sailing so smoothly on its own beforehand.
  • HP’s success in PCs amounts to “arguably, HP sucks a little less than the other guys”.
  • HP’s elite reputation is long gone (admittedly, for the most part that predates Hurd).
• Doug Henschen evidently favors really strong intellectual property protection for software, even forbidding plug-compatible reverse engineering. I agree with Doug up to the point that it should be forbidden to copy proprietary software, but I don’t see why he (or a court) would view such behavior as copying.

• Subsecond load latency is substantially impossible. Doing that amounts to OLTP.
• 5 seconds or so is doable with aggressive investment and tuning.
• Several minute load latency is pretty easy.
• 10-15 minute latency or longer is now very routine.

There’s generally a throughput/latency tradeoff, so if you want very low latency with good throughput, you may have to throw a lot of hardware at the problem.

I’d expect to hear similar things from any other vendor with reasonably mature analytic DBMS technology. Low-latency load is a problem for columnar systems, but both Vertica and ParAccel designed in workarounds from the getgo. Aster Data probably didn’t meet these criteria until Version 4.0, its old “frontline” positioning notwithstanding, but I think it does now.

Related link

• Just what is your need for speed anyway?

• How quickly and cost-effectively does it execute SQL?
• What analytic functionality, SQL or otherwise, does it do a good job of executing?

And so, in undertaking such a selection, you need to start by addressing three issues:

• What does “speed” mean to you?
• What does “cost” mean to you?
• What analytic functionality do you need anyway?

Key elements of cost* include:

• Software license and maintenance
• Hardware purchase cost, maintenance, electric power, and computer room burden
• Database and system administration
• (For some uses cases) Programming

*Assuming a classical in-house IT shop, where products are typically bought rather than leased/rented. With outsourced and/or monthly-fee structures, the details change but the principles remain the same.

Most of that can be evaluated pretty well via a spreadsheet, although things can get a bit tricky when you get to people costs, which are a large fraction of the whole. In particular, different analytic DBMS product suites have great, high-performance support for different (and often rapidly growing) sets of functionality – basic and advanced SQL, statistics, and more. Figuring out which ones will be best for your programmers, and how significant the differences are — well, that’s a lot like any other programming language evaluation, and those are rarely neat or clean-cut.

But when it comes to evaluating speed, there’s no substitute for a well-designed proof of concept (POC). Many analytic DBMS and appliance vendors are happy to let you do a POC, on your own premises (or remotely if you prefer), under your control, at no cost to you. And that’s great. It is crucial that a POC be run either by you, by a consultant* answerable to you, or – if you decide the vendor must run it for you – at least with you watching every step of the way and knowing exactly what is being done. Appliance vendors do find it cheaper to run POCs on their own premises, so a certain reluctance to ship you a box is understandable. But make no compromises about the transparency of a POC, or about your control of exactly what it is that gets tested.

*Since I sell consulting services for users evaluating analytic DBMS, I naturally am biased to think that consultants can be very useful in the process. But whether you should use them a little (sanity check), a medium amount (work with you through the process), or heavily (actually drive the process for you and/or execute the POCs) is very dependent upon your specific situation.

So far as I’ve been able to tell:

• Netezza loves to ship boxes to prospects for POCs, and have them set up the boxes and do POCs themselves. That’s a big reason why Netezza wants to call attention to this subject.
• Oracle has generally been pretty reluctant to ship Exadata boxes out for POCs. That’s the other reason Netezza wants to call attention to the issue.
• Open source vendors make it easy for you to download and test at least their community editions.
• Vertica makes it pretty easy for you to test its software too (download or cloud).
• ParAccel has generally insisted on running POCs itself, although it will do so on your premises if you insist.
• Teradata naturally tries to do POCs on its own premises, but doesn’t insist too hard. (Edit: Randy Lea of Teradata says that Teradata is now doing over half its POCs onsite.)
  

Most of the criticisms I’ve heard of vendors’ POC practices have been directed at Oracle or ParAccel.

For most POCs, it’s a good conceptual template to form and then test a hypothesis to the effect of:

• For a given technology product assemblage (brand of DBMS, number of nodes, etc.), and
• For a given level of human effort (e.g., administrative effort), you can
• Run a given a workload, with
• Satisfactory and satisfactorily consistent response times

Sometimes absolute throughput and price/performance are important secondary considerations; sometimes they’re less germane. But either way, it’s almost always right to focus primarily on the questions of “What do I want this system to do?” and “What do I think we’re going to have to invest in it?” By way of contrast, it’s often misleading to focus too much on questions like “What’s the one number that best describes the performance of this system?” — even if you customize that calculation for your environment – or, even worse, “How much speed-up can I get on my single worst Query from Hell?”

The fundamental rule of POC construction is: Model your entire use case as best you can. That means you need to consider, at a minimum:

• Your whole concurrent query, other analytic, and low-latency update workload (peak).
• Your whole query, analytic, load, backup, and maintenance workload (ongoing).
• Partial-failure scenarios.
• Your core SLAs (Service-Level Agreements).

Of course, that’s not as easy as it sounds. Presumably, the main reason you’re getting a new analytic DBMS is that you want to do new kinds of analysis. By the very nature of analytics, you won’t know what analytic operations are most useful until you try them out and see what their results are. On the other hand – if you haven’t done considerable thinking about how you’re going to use your new analytic database, how did you ever get funding for the project in the first place?

Seriously, I could write multiple posts, each as long as this one (but more application-oriented), about how to upgrade your analytic capabilities (and which fool’s gold to avoid). But this has gotten pretty long already, so for now I’ll just stop here.

Note: My clients at Netezza asked me to write something short about POCs they could use as a kind of foreword to some collateral, where by “short” they meant single-paragraph or something like that. They’re great clients, so I said yes, under the condition I could also use it as a blog post. Except … this post didn’t turn out to be nearly as short as they envisioned. Oops.

Related links

• My February, 2009 slide deck on how to select an analytic DBMS is in many parts still pretty current

• Sybase’s main businesses are:
  • Classic OLTP DBMS (Sybase ASE, for Adapative Server Enterprise, unless I’ve missed yet another name change).
  • Analytic technology — mainly Sybase IQ, but more generally Sybase RAP.
  • Mobile technology. (The frequently renamed small DBMS SQL Anywhere was the foundational product of and still is included in the mobile division.)
• SAP’s thoughts on in-memory database management are interesting. However, I think SAP’s oft-repeated claim that it has a lot of important in-memory database technology to bring to Sybase (or for that matter SAP customers) is mainly smoke and mirrors. Cool data access methods, good niche database products, and broadly applicable multi-domain DBMS innovations are three different things. Granting that SAP probably has the first and thinks it has the second is not the same as giving it much credence for having the third.
• SAP claims that, 15 years after its refusal to support Sybase turned Sybase into a DBMS also-ran, it by now is “relatively simple” to port SAP’s apps to Sybase ASE, and that they will make that happen. I actually believe that SAP’s apps will soon run on Sybase ASE, where by “soon” I mean “in a couple of years for no-apologies general availability.” (Certifying a DBMS for SAP is a long process.) The main missing features — e.g., row-level locking — were already put into Sybase back in the last millenium. Nor could there be fundamental architectural problems that keep SAP from supporting Sybase ASE, or else SAP couldn’t have supported Microsoft SQL Server (which, long ago, was a Sybase fork).
• I don’t see any market or competitive dynamics that would lead the SAP acquisition to hurt Sybase’s ASE or mobile businesses. General merger management mishegas is, of course, always a possibility.
• SAP Business Objects partners with Sybase IQ’s competitors. That could be a problem. However, coopetition is pretty strong in the business intelligence market. I don’t think any of SAP Business Objects, IBM Cognos, or Oracle Business Intelligence are much held back from partnering by competitive dislike of their parent companies.
• The rest of SAP might be able to drum up some extra business for Sybase IQ.
• It would be natural for IBM/Cognos to now buy a columnar DBMS of its own. Vertica is an obvious first choice. ParAccel would surely come much cheaper. Since ParAccel has little chance of surviving as an independent company — too immature and too little differentiation to overcome that — I’d expect ParAccel’s board to jump at the chance to sell out.
• It would be interesting if SAP Business Objects would revive the CEP-based BI idea.
• I gather Sybase’s AnswersAnywhere concept network/object model-based natural language/speech recognition technology never went anywhere. Unsurprising (it seemed like it needed too much hand-building to scale semantically), but regrettable even so.
• I don’t see anything in this acquisition that would revive PowerBuilder (Sybase’s Visual Basic competitor), Sybase’s CASE (Computer-Aided Software Engineering) tools, and so on.
• And on the personal side — I’ll probably lose Sybase as a customer due to this merger, but it could have been worse. A lot of vendors smaller than Sybase are bigger customers for Monash Research.

Edit: Right after I posted this, I saw email from Sybase clarifying that Sybase’s in-memory technology, while slightly influenced by some ANTs IP Sybase bought non-exclusive rights to, is essentially home-grown. That’s what I thought, but the call sounded like it was saying something different.

Further coverage of SAP/Sybase:

• SAP believes in database proliferation
• More quick reactions to SAP/Sybase
• Technical basics of Sybase IQ
  

• The X Company (XCo) has a <1 TB database.
• 100s of XCo’s customers log in at once to run reports. 50-200 concurrent queries is a good target number.
• XCo had been “suffering” with Oracle and wanted to upgrade.
• XCo didn’t have a lot of money to spend. Netezza pulled out of the sales cycle early due to budget (and this was recently enough that Netezza Skimmer could have been bid).
• Greenplum didn’t offer any references that approached the desired number of concurrent users.
• Ultimately the evaluation came down to Vertica and ParAccel.
• Vertica won.

Notes on the Vertica vs. ParAccel selection include:

• ParAccel sent an engineer on-site to do a proof-of-concept (POC), and generally competed very hard for the deal.
• Vertica dropped by for a sales call once, and let XCo do the Vertica POC itself.
• Not surprisingly, XCo got the impression that Vertica was easier to set up and administer than ParAccel.
• Also, when ParAccel emphasized architectural features such as custom “backplane” and compiled queries, XCo got the impression – right or wrong – that ParAccel’s performance was more brittle or situational than Vertica’s.
• ParAccel was modestly faster than Vertica in the POC. (I think — Vertica’s numbers were described as being “very competitive.”)
• In multiple ways, Vertica gave the impression of greater product and vendor maturity than ParAccel.

My contact continues to be interested in all things Greenplum, and has recommended Greenplum Single-Node Edition to his analyst colleagues.

• Vertica had a “killer” Q4 and is doing very well in Q1 again.
• Vertica burned hardly any cash last year; i.e., it was close to cash-flow neutral in 2009.
• Vertica is hiring aggressively, e.g., in sales.
• Vertica is well down the path with several CEO candidates who Jerry regards as outstanding. He is hopeful there will be a new CEO in April. (But I bet that would be late April, given what Jerry mentioned about his own travel plans.)
• Absent a full-time CEO, Jerry and Andy Palmer are spending a lot more time with Vertica.
• One Vertica customer is approaching a petabyte of user data. The last time Vertica had checked, that customer had been more in the ¼ petabyte range.
• Other multi-hundred terabyte Vertica databases were mentioned, including one where Vertica claims to have beaten Teradata and perhaps other competitors in a head-to-head competition (it sounds like that one’s too recent to be deployed yet).
• Vertica sees Aster and Greenplum competitively more often than it sees ParAccel.
• Vertica sees Sybase IQ competitively a lot in financial services (in new-name accounts for Sybase as well as where some kind of Sybase DBMS is an incumbent), and more occasionally in other sectors.

NDA parts of the conversation also gave me the impression that Vertica is moving forward just as eagerly as its peers. I.e., I didn’t uncover any reason to think that Ralph’s departure is a sign of trouble, of the company being shopped, etc.

On the other hand, I didn’t uncover any other clear reason for Ralph’s departure either. The party line is that Ralph left for “personal reasons”. It was his decision to leave. He did a great job while at Vertica. Basically, Vertica is saying that, even though all was going swimmingly, Ralph just up and quit, leaving some very valuable unvested stock options on the table at what had been his first CEO gig ever.

Contacting Ralph didn’t add any immediate insight. He responded quickly via a Twitter DM, but was in the airport for a trip to his home country of South Africa, and deferred discussion until after his return.

Incidentally, Infobright and ParAccel have both also had recent CEO turnover. Stated reasons in each case were of the “Right person to lead the next stage of the company’s growth” variety.

Anyone who has met with us knows ParAccel shies away from hype.

But like many things ParAccel says, that is not true.

The latest whoppers came in the form of several customers ParAccel listed on its website who hadn’t actually bought ParAccel’s DBMS, nor even decided to do so. It is fairly common to to claim a customer win, then retract the claim due to lack of permission to disclose. But that’s not what happened in these cases. Based on emails helpfully shared by a ParAccel competitor competing in some of those accounts, it seems clear that ParAccel actually posted fabricated claims of customer wins.

Another thing that was both technically and substantively false was ParAccel’s claim to be CERTIFIED price-performance leader. Obviously, this was meant to give the impression that ParAccel had been “certified” as the leader in price/performance, when the closest thing to that that was remotely true was that ParAccel had a leading position in the category of “price/performance measurements that happen to have a certification process.” At least, that was true for a short time; then ParAccel’s certification was found to have been erroneous, and got revoked, which did not however inspire ParAccel to immediately take the claim off the front page of its website.

ParAccel’s website also reflects a lot of praise from flagship customer LatiNode. What it perhaps understandably neglects to mention is that LatiNode is in a dormant state, placed there by acquirer Elandia due to LatiNode’s criminally corrupt customer acquisition practices.

I also don’t believe ParAccel’s endlessly-repeated claim that is has never lost a benchmark on performance. However, I must in fairness note that while I’ve been given names of customers who are supposed counterexamples to this claim by somebody I trust, I’ve never been able to actually verify those supposed ParAccel losses.

]]> http://www.dbms2.com/2009/09/30/facts-and-rumors/feed/ 2 http://www.dbms2.com/2009/07/08/progress-in-figuring-out-what-paraccel-is-doing/ http://www.dbms2.com/2009/07/08/progress-in-figuring-out-what-paraccel-is-doing/#comments Wed, 08 Jul 2009 23:46:05 +0000 Curt Monash http://www.dbms2.com/?p=835 (Oops: Thought I’d posted this before I went out for the afternoon …)

Barry Zane of ParAccel has — finally! — started a blog.  Barrry’s first post, probably in connection with ParAccel’s recent TPC-H submission and subsequent brouhaha, consisted mainly of metaphor + very elementary and well-known arguments for column stores. Barry’s second post, however, was in direct response to Daniel Abadi’s speculation about ParAccel’s architecture.  That post also promises a follow-up addressing the TPC-H in a more substantive way.

Barry’s points include:

• ParAccel never used the row-oriented Postgres execution engine. This is contrary to Daniel’s speculation.
• ParAccel previously used an adaption of the Postgres cost-based optimizer, but now has written a new one from scratch.
• ParAccel has designed its optimizer to handle lots and lots of joins. One reason Barry offers is that ParAccel wants to run customers’ old schemas unaltered, whether or not those are really optimal for the ParAccel DBMS.  That approach is somewhat in contrast to Vertica, which originally focused entirely on star schemas.   And it goes well with ParAccel’s interest in appealing to customers who at least think they want to run ParAccel in Oracle or SQL Server emulation mode.

Also in the post, Barry:

• Makes an extremely silly marketing exaggeration by referring to ” the only other vendor that was able to run the 30TB TPC-H” (emphasis mine).
• Makes the more excusable marketing exaggeration “Publishing the benchmark with unmatched performance is simply one way to demonstrate robustness and flexibility.  Nothing more, nothing less.”
• Makes the very clear marketing claim “For customers, the real test will be their own bake-offs, where our performance has never been beaten.” (Emphasis mine.) That last one directly contradicts what I’ve been told by at least two ParAccel competitors, so I’ll be curious to see what they come up with to substantiate their version of the story.

Anyhow, it’s great to see ParAccel retreating from its obsessive secrecy, which in my opinion has been even worse than Netezza’s used to be.

But now Daniel Abadi has done a brilliant, detailed, speculative analysis of ParAccel’s publications.  Here’s the meat, emphasis mine:

(1) Why did they configure their TPC-H application with such a high amount of disk I/O throughput capabilty when they are a column-store? (Stonebraker’s question)
(2) Why did queries spend seemingly 6X more time doing I/O than a column-store should have to do?
(3) Why are they worried about queries with thousands of joins?
(4) Why do they think TPC-H/TPC-DS queries have 42 joins?

And then a theory that answers all four questions at the same time came to me. Perhaps ParAccel directly followed my advice (see option 1) on “How to create a new column-store DBMS product in a week“. They’re not a column-store. They’re a vertically partitioned row-store (this is how column-stores were built back in the 70s before we knew any better). Each column is stored in its own separate table inside the row-store (PostgreSQL in ParAccel’s case). Queries over the original schema are then automatically rewritten into queries over the vertically partitioned schema and the row-store’s regular query execution engine can be used unmodified. But now, every attribute accessed by the query now adds an additional join to the query plan (since the vertical partitions for each column in a table have to be joined together).

This immediately explains why they are worried about queries with hundreds to thousands of joins (questions 3 and 4). But it also explains why they seem to be doing much more I/O than a native column-store. Since each vertical partition is its own table, then each tuple in a vertical partition (which contains just one value) is preceded by the row-store’s tuple header. In PostgreSQL this tuple header is on the order of 27 bytes. So if the column width is 4 bytes, then there is a factor of 7 extra space used up for the tuple header relative to actual user data. And if the implementation is super naive, they also will need an additional 4 bytes to store a tuple identifier for joining vertical partitions from the same original table with each other. This answers questions 1 and 2, as the factor of 6 worse I/O efficiency is now obvious.

It will be interesting to see whether ParAccel comments, but even it does, I wouldn’t necessarily take ParAccel’s statements as dispositive.  For example — and illustrative of my view of ParAccel’s trustworthiness — I believe ParAccel’s competition who tell me that ParAccel’s claim to have won or at least tied all POCs on performance is flat-out untrue.