Vertica’s paying customer count
In a recent Computerworld article, Andy Ellicott of Vertica was cited as saying Vertica has 50 paying customers total. That’s very much on par with Greenplum’s figure, leaving aside any questions of deal size. (Greenplum runs a number of databases much larger than Vertica’s biggest. However, I believe Greenplum also charges a lot less per terabyte of user data.)
Previous Vertica paying customer count figures include:
| Categories: Data warehousing, Greenplum, Vertica Systems | 3 Comments |
Three approaches to parallelizing data transformation
Many MPP data warehousing vendors have told me their products are used for ELT (Extract/Load/Transform) instead of ETL (Extract/Transform/Load). I.e., needed data transformations are done on the MPP system, rather than on the — probably SMP — system the data comes from.* If the data transformation is being applied on a record-by-record basis, then it’s automatically fully parallelized. Even if the transforms are more complex, considerable parallel processing may still be going on.
*Or it’s some of each, at which point it’s called ETLT — I bet you can work out what that stands for.
| Categories: Aster Data, Data integration and middleware, Data warehousing, EAI, EII, ETL, ELT, ETLT, MapReduce, Parallelization, Pervasive Software | 4 Comments |
Why MapReduce matters to SQL data warehousing
Greenplum and Aster Data have both just announced the integration of MapReduce into their SQL MPP data warehouse products. So why do I think this could be a big deal? The short answer is “Because MapReduce offers dramatic performance gains in analytic application areas that still need great performance speed-up.” The long answer goes something like this.
The core ideas of MapReduce are:
| Categories: Analytic technologies, Data warehousing, MapReduce, Parallelization | 10 Comments |
Known applications of MapReduce
Most of the actual MapReduce applications I’ve heard of fall into a few areas:
- Text tokenization, indexing, and search
- Creation of other kinds of data structures (e.g., graphs)
- Data mining and machine learning
That covers all MapReduce apps I recall hearing about via commercial companies and users, and also includes most of what’s in the two big sources I found online.
| Categories: MapReduce, RDF and graphs, Text | 7 Comments |
MapReduce links
For whatever reason, I seem to be making the peripheral posts about MapReduce tonight before getting to the meat of the issues. So be it. There’s a rich set of links out there about MapReduce, and here are some of the best of them:
- Aster Data introduced MapReduce integrated into its SQL data warehouse DBMS tonight. Aster’s site features an excellent white paper.
- Exactly the same is true of Greenplum.
- Google Labs offers the seminal MapReduce research paper. It also has a broken link to an associated slide presentation, which fortunately is available here.
- One can get a good sense of MapReduce by reading up on the open source implementation Hadoop.
- In particular, this list of Hadoop applications is the longest list of MapReduce applications I know of (ahead even of Google’s long internal list).
- Joel Spolsky explained the core MapReduce concept a couple of years ago.
| Categories: MapReduce, Parallelization | 3 Comments |
MapReduce sound bites
Last Thursday, both Greenplum and Aster Data — the two most recent of my numerous data warehouse specialist customers — both told me of the same major innovation. Both were rushing to announce it first, before anybody else did. This led to considerable tap dancing, with the upshot being that both are releasing the information tonight or tomorrow morning.
What’s going on is that Aster Data and Greenplum have both integrated MapReduce into their respective MPP shared-nothing data warehouse DBMS. Read more
| Categories: Analytic technologies, Aster Data, Greenplum, MapReduce, Parallelization | 7 Comments |
Greenplum’s single biggest customer
Greenplum offered a bit of clarification regarding the usage figures I posted last night. Everything on the list is in production, except that:
- One Greenplum customer is at 400 terabytes now, and upgrading to >1 petabyte “as we speak.”
- Greenplum’s other soon-to-be >1 petabyte customer isn’t in production yet. (Greenplum previously told me that customer was in the process of loading data right now.)
| Categories: Data warehousing, Greenplum, Specific users | 1 Comment |
Greenplum is in the big leagues
After a March, 2007 call, I didn’t talk again with Greenplum until earlier this month. That changed fast. I flew out to see Greenplum last week and spent over a day with president/co-founder Scott Yara, CTO/co-founder Luke Lonergan, marketing VP Paul Salazar, and product management/marketing director Ben Werther. Highlights – besides some really great sushi at Sakae – start with an eye-opening set of customer proof points, such as: Read more
| Categories: Analytic technologies, Data warehouse appliances, Data warehousing, Greenplum, PostgreSQL | 4 Comments |
My current customer list among the data warehouse specialists
One of my favorite pages on the Monash Research website is the list of many current and a few notable past customers. (Another favorite page is the one for testimonials.) For a variety of reasons, I won’t undertake to be more precise about my current customer list than that. But I don’t think it would hurt anything to list the data warehouse DBMS/appliance specialists in the group. They are:
- Aster Data
- Calpont
- DATAllegro
- Greenplum
- Infobright
- Netezza
- ParAccel
- Teradata
- Vertica
All of those are Monash Advantage members.
If you care about all this, you may also be interested in the rest of my standards and disclosures.
| Categories: About this blog, Aster Data, Calpont, DATAllegro, Data warehousing, Greenplum, Infobright, Netezza, ParAccel, Teradata, Vertica Systems | 1 Comment |
Kevin Closson doesn’t like MPP
Kevin Closson of Oracle offers a long criticism of the popularity of MPP. Key takeaways include:
- TPC-H benchmarks that show Oracle as somewhat superior to DB2 are highly significant.
- TPC-H benchmarks in which MPP vendors destroy Oracle are too unimportant to even mention.
- SMP did better than MPP the last time he was in a position to judge (which evidently was some time during the Clinton Administration), so it surely must still be superior for all purposes today.
| Categories: Data warehousing, Oracle, Parallelization | 18 Comments |
The Explosion in DBMS Choice
If there’s one central theme to DBMS2, it’s that modern DBMS alternatives should in many cases be used instead of the traditional market leaders. So it was only a matter of time before somebody sponsored a white paper on that subject. The paper, sponsored by EnterpriseDB, is now posted along with my other recent white papers. Its conclusion — summarizing what kinds of database management system you should use in which circumstances — is reproduced below.
Many new applications are built on existing databases, adding new features to already-operating systems. But others are built in connection with truly new databases. And in the latter cases, it’s rare that a market-leading product is the best choice. Mid-range DBMS (for OLTP) or specialty data warehousing systems (for analytics) are usually just as capable, and much more cost-effective. Exceptions arise mainly in three kinds of cases:
- Small enterprises with very limited staff.
- Large enterprises that have negotiated heavily-discounted deals for a market-leading product.
- Super-high-end OLTP apps that need absolute top throughput (or security certifications, etc.)
Otherwise, the less costly products are typically the wiser choice.
| Categories: Database diversity | 3 Comments |
Three happy 100 terabyte-plus customers for DATAllegro
Over on my Network World blog, I asked the question “So who are DATAllegro’s actual current customers?” As regular readers know, that’s a fairly hard question to answer. TEOCO is widely known as DATAllegro’s flagship reference, but after that the list gets thin in a hurry.
As a by-the-by to other discussions, DATAllegro Stuart Frost undertook to respond in part himself. Specifically, he gave me two names of two other happy customers that are or imminently will be running DATAllegro against 100+ terabytes of user data. Read more
| Categories: DATAllegro, DBMS product categories, Data warehouse appliances, Data warehousing | Leave a Comment |
Exasol technical briefing
It took 5 ½ months after my non-technical introduction, but I finally got a briefing from Exasol’s technical folks (specifically, the very helpful Mathias Golombek and Carsten Weidmann). Here are some highlights: Read more
| Categories: Analytic technologies, Columnar database management, Data warehousing, Exasol, In-memory DBMS, Memory-centric data management | Leave a Comment |
A NoteWorthy win for Intersystems Cache’
A small Microsoft SQL Server-based medical application vendor called NoteWorthy Medical Systems bought a small Intersystems Cache’-based medical application vendor called Mars Medical Systems. NoteWorthy then decided to rebuild its product line on Intersystems Cache’. A press release ensued.*
*In general, my criticisms of Intersystems’ stealth marketing are beginning to be relaxed. On the other hand, if you want to be technical, I still haven’t actually talked with the company for years …
I spoke briefly with Mark Conner, founder of Mars Medical and now EVP of NoteWorthy, about why he so loves Cache’. (I asked what he disliked about the product; his response was an emphatic “Nothing”.) It basically boils down to two reasons:
-
Mark thinks hierarchical data models are a great fit for medical applications. For example, the application’s UI (and local schema) look quite different depending on which particular complaints or diagnoses apply to particular patient visits.
-
Cache’ just runs and runs w/o DBA intervention. Mark cited a figure of two support engineers for Mars Medical, supporting over 1,000 medical (largely group) practices, almost none of which have DBAs.
The latter feature is crucial to small ISVs selling application software to even smaller users, and is a big part of why Progress and Intersystems have large share in that market. More generally, it’s the most important and common technical advantage that mid-range database management systems generally enjoy versus the market leaders. (The other big advantage, of course, is pricing.)
| Categories: Intersystems and Cache', Microsoft and SQL*Server, Mid-range | 2 Comments |
Patent nonsense in the data warehouse DBMS market
There are two recent patent lawsuits in the data warehouse DBMS market. In one, Sybase is suing Vertica. In another, an individual named Cary Jardin (techie founder of XPrime, a sort of predecessor company to ParAccel) is suing DATAllegro. Naturally, there’s press coverage of the DATAllegro case, due in part to its surely non-coincidental timing right after the Microsoft acquisition was announced and in part to a vigorous PR campaign around it. And the Sybase case so excited a troll who calls himself Bill Walters that he posted identical references to it on about 12 different threads in this blog, as well as to a variety of Vertica-related articles in the online trade press. But I think it’s very unlikely that any of these cases turn out to much matter. Read more
| Categories: Columnar database management, DATAllegro, Data warehousing, Database compression, Sybase, Vertica Systems | 5 Comments |
Compare/constrast of Vertica, ParAccel, and Exasol
I talked with Exasol today – at 5:00 am! — and of course want to blog about it. For clarity, I’d like to start by comparing/contrasting the fundamental data structures at Vertica, ParAccel, and Exasol. And it feels like that should be a separate post. So here goes.
- Exasol, Vertica, and ParAccel all store data in columnar formats.
- Exasol, Vertica, and ParAccel all compress data heavily.
- Exasol and Vertica operate on in-memory data in compressed formats. ParAccel decompresses the data when it gets to RAM. Exasol, Vertica, and ParAccel all — perhaps to varying extents — operate on in-memory data in compressed formats.
- ParAccel and Exasol write data to what amounts to the in-memory part of their basic data structures; the data then gets persisted to disk. Vertica, however, has a separate in-memory data structure to accept data and write it to disk.
- Vertica is a disk-centric system that doesn’t rely on there being a lot of RAM.
- ParAccel can be described that way too; however, in some cases (including on the TPC-H benchmarks), ParAccel recommends loading all your data into RAM for maximum performance.
- Exasol is totally optimized for the assumption that queries will be run against data that had already been previously loaded into RAM.
Beyond the above, I plan to discuss in a separate post how Exasol does MPP shared-nothing software-only columnar data warehouse database management differently than Vertica and ParAccel do shared-nothing software-only columnar data warehouse database management. 
| Categories: Columnar database management, Data warehousing, Database compression, Exasol, ParAccel, Vertica Systems | 9 Comments |
EnterpriseDB update
I had lunch today with CTO Bob Zurek of EnterpriseDB, who turns out to live in almost the same town I do (they technically separated in 1783, but share a high school today). DBMS-related highlights included:
- EnterpriseDB thinks PostgreSQL training and certification are a big deal for increasing PostgreSQL adoption.
- EnterpriseDB’s business focus right now (at least, one of them) is moving developers from interest to download to deployment and payment — i.e., the standard funnel for open source and open-source-inspired products.
- EnterpriseDB finds it important to be a good PostgreSQL community citizen. This makes a lot of sense, as EnterpriseDB doesn’t control the core PostgreSQL engine, even if it does employ some of the core PostgreSQL developers.
- But “open source” is not the same as “free”.
- I got the impression that the GridSQL technology EnterpriseDB acquired is being used to go after general read-mostly, horizontally-scaling applications (i.e., MySQL’s sweet spot). I did not get the impression, by way of contrast, that EnterpriseDB is out to play catch-up — e.g., with GreenPlum — in MPP data warehousing.
- Bob pointed out that something like “Vacuum” to clean up the database periodically is needed in a MVCC (MultiVersion Concurrency Control) engine. He thinks PostgreSQL’s autovacuum is good but not ideal.
- Bob draws this as yet another two-dimensional positioning graph, but in essence he thinks PostgreSQL and Postgres Plus are well-suited for a large space that’s above MySQL and below Oracle. I don’t think he really contradicted Kee Kwan’s opinion that there are good times to use PostgreSQL and good times to use MySQL.
- I was wrong when I previously said EnterpriseDB now offers MySQL portability. It just offers MySQL migration.
- The Elastra/EnterpriseDB cloud offering isn’t generally available yet.
- Stay tuned for developments in replication/high availability.
| Categories: EnterpriseDB and Postgres Plus, Mid-range, Open source, PostgreSQL | 1 Comment |
Netezza update
In my usual dual role, I called Phil Francisco of Netezza to lay some post-Microsoft/DATAllegro consulting on him late on a Friday night — and then took the opportunity of being on the phone with him to get a general Netezza update. Netezza’s July quarter just ended, so they’re still in quiet period, so I didn’t press him for a lot of numerical detail. More generally, I didn’t find a lot out that wasn’t already covered in my May Netezza update. But notwithstanding all those disclaimers, it was still a pretty interesting chat.
My strongest takeaway was that Netezza sees concurrency as a significant competitive advantage. This is reflected in POCs, where Netezza guides prospects to simulate real-life mixed workloads. It also reflects the Netezza customer base. Phil says Netezza has “busy” warehouses with up to 80 terabytes of user data, with lots of busy ones in the single-digit to 20ish terabyte range. Multiple Netezza references have 100s of concurrent users, and the 1000 mark has been crossed.
Speaking of concurrency, Phil had a clear opinion of the typical Sybase IQ installation — a small reporting mart, supporting hundreds or thousands of users, but probably not a lot of ad hoc query. On the other hand, he recalls outright competing against Sybase only twice in the past year.
The vendor Netezza does see the most is, no surprise, Oracle. He put Oracle at 60ish percent, with most of the rest divided among Teradata and DB2 (only a few Microsoft SQL Server). Among the other new data warehouse specialists, Greenplum comes up the most often. (There was some confusion between “competitor” and “incumbent” in our discussion, and the sample sizes are small anyway, so fine levels of detail shouldn’t be taken too seriously.)
On the advanced analytics side, it sounds as if SAS integration akin to Teradata’s will happen sooner than any significant integration of Netezza’s own NuTech acquisition.
| Categories: Data warehouse appliances, Data warehousing, Greenplum, Netezza, Sybase | 2 Comments |
Database compression coming to the fore
I’ve posted extensively about data-warehouse-focused DBMS’ compression, which can be a major part of their value proposition. Most notable, perhaps, is a short paper Mike Stonebraker wrote for this blog — before he and his fellow researchers started their own blog — on column-stores’ advantages in compression over row stores. Compression has long been a big part of the DATAllegro story, while Netezza got into the compression game just recently. Part of Teradata’s pricing disadvantage may stem from weak compression results. And so on.
Well, the general-purpose DBMS vendors are working busily at compression too. Microsoft SQL Server 2008 exploits compression in several ways (basic data storage, replication/log shipping, backup). And Oracle offers compression too, as per this extensive writeup by Don Burleson.
If I had to sum up what we do and don’t know about database compression, I guess I’d start with this:
- Columnar DBMS really do get substantially better compression than row-based database systems. The most likely reasons are:
- More elements of a column fit into a single block, so all compression schemes work better.
- More compression schemes wind up getting used (e.g., delta compression as well the token/dictionary compression that row-based systems use too).
- Data-warehouse-based row stores seem to do better at compression than general-purpose DBMS. The reasons most likely are some combination of:
- They’re trying harder.
- They use larger block sizes.
- Notwithstanding these reasonable-sounding generalities, there’s a lot of variation in compression success among otherwise comparable products.
Compression is one of the most important features a database management system can have, since it creates large savings in storage and sometimes non-trivial gains in performance as well. Hence, it should be a key item in any DBMS purchase decision.
Some Elastra numbers
GigaOm reports that Elastra just raised $12 million, and that it has 40 paying customers, up from 13 around the time of Elastra’s March launch.
| Categories: Cloud computing, Elastra | Leave a Comment |
Column stores vs. vertically-partitioned row stores
Daniel Abadi and Sam Madden followed up their post on column stores vs. fully-indexed row stores with one about column stores vs. vertically-partitioned row stores. Once again, the apparently useful way to set up the row-store database backfired badly.* Read more
Extensive QlikView coverage from a big fan and reseller
David Raab is a reseller and great fan of QlikTech’s QlikView. His recent lengthy post about the product (I hesitate to call it “detailed” only because he rightly complains that QlikTech is in fact stingy with technical detail) is positive enough to have been recommended by the company itself. Specifically, it was cited in the comment thread to my recent post on QlikTech, where David himself also addressed some of my questions.
But of course, no technology is perfect, not even one as great as David thinks QlikView is. Read more
Daniel Abadi and Sam Madden on column stores vs. indexed row stores
Daniel Abadi and Sam Madden — for whom I have the highest regard after our discussions regarding H-Store — wrote a blog post on Vertica’s behalf, arguing that column stores are far superior to fully-indexed row stores for not-very-selective queries. They link to a SIGMOD paper backing their argument up, provide some diagrams, and generally make a detailed case. As best I understand, here are some highlights: Read more
| Categories: Columnar database management, Vertica Systems | 7 Comments |
QlikTech/QlikView update
I talked with Anthony Deighton of memory-centric BI vendor QlikTech for an hour and a half this afternoon. QlikTech is quite the success story, with disclosed 2007 revenue of $80 million, up 80% year over year, and confidential year-to-date 2008 figures that do not disappoint as a follow-on. And a look at the QlikTech’s QlikView product makes it easy to understand how this success might have come about.
Let me start by reviewing QlikTech’s technology, as best I understand it.
