• The most important software on the Oracle Big Data Appliance is a full set of Cloudera Enterprise code. Oracle will do Tier 1 Cloudera/Hadoop support, while Cloudera handles Tiers 2 and 3.
• The key spec ratios are 1 core/4 GB RAM/3 TB raw disk. That’s reasonably in line with Cloudera figures I published in June, 2010.
• This is really Oracle’s multi-structured big data appliance. Oracle’s relational big data appliance is Exadata, which has been out for years and has comparable capacity to Oracle’s new “Big Data Appliance.” (Chris Preimesberger made a similar point.)
• The Oracle Big Data Appliance list price is $450,000 for 18 12-core servers, plus $54,000/year maintenance.
  • That’s around $25,000 per server (and associated storage).
  • That’s also around $2,000/core.
  • That’s also around $500/TB of spinning disk, before compression.
  • None of those per-unit figures sounds ridiculous …
  • … but because of Oracle’s appliance configuration there’s indeed a hefty minimum initial purchase.

Peter Goldmacher argues that, because of size and price point, the Oracle Big Data appliance is targeted for high-end deployments rather than starter/test/development set-ups. To first approximation, that makes sense, in that:

• The Oracle Big Data Appliance is in the petabyte range for data capacity, and …
• … the number of petabyte-scale Hadoop deployments is in the low tens, and …
• … many of those aren’t at Oracle shops anyway.

Surely the Oracle Big Data Appliance isn’t designed for the 4-8 node play-with-Hadoop crowd.

On the the other hand, if you’re at a big, committed Oracle shop, and you want to do your first serious Hadoop deployment, why not go with the Oracle Big Data Appliance? You probably could save money with an alternative approach — but if your employers are committed to Oracle, saving money is surely not their greatest concern. Overpay by a bit; make your management happy with the Oracle logo; get Hadoop on your resume; prosper. That seems like a winning plan all the way around.

Top-level points included:

• Exasol’s technical philosophy is substantially the same as before, albeit not with as extreme a focus on fitting everything in RAM.
• Exasol believes its flagship DBMS EXASolution has great performance on a load-and-go basis.
• Exasol has 25 EXASolution customers, all in Germany.*
• 5 of those are “cloud” customers, at hosting providers engaged by Exasol.
• EXASolution database sizes now range from the low 100s of gigabytes up to 30 terabytes.
• Pretty much the whole company is in Nuremberg.

*That excludes some money from Hitachi. Exasol’s Hitachi partnership is still in limbo, an apparent casualty of the world economic crisis.

On the technical side:

• As noted in my 2008 post, EXASolution is a columnar, no-head-node MPP (Massively Parallel Processing) DBMS.
• The main way EXASolution compresses data is via dictionary/tokenization. 5:1 is a typical compression ratio before mirroring and so on, out of a 2-10:1 range.
• EXASolution writes data to blocks in memory that are smaller than what is otherwise its preferred size (1/2 to 5 megabytes). These are sent to disk, where merge eventually happens. Exasol insists that write performance has always been fully satisfactory to customers to date.
• EXASolution doesn’t have much in the way of performance tuning knobs. Exasol says they aren’t needed, and says that one really can start an EXASolution POC (Proof of Concept) in a day or so.
• EXASolution doesn’t have much in the way of workload management capabilities, except what’s automagic (e.g., short query bias). However, it does collect statistics you can query via your favorite BI tool.
• EXASolution doesn’t have much in the way of analytic platform capabilities, although there is some Lua-based scripting. However, there’s something NDA in the analytic platform area Coming Soon.*

In general, the whole thing sounds somewhat like ParAccel, at least at a high level.

*Exasol is not and never has been our client, but we can keep secrets for them even so.

Naturally, Exasol believes EXASolution has fine concurrency, with at least one customer routinely running 2000 concurrent users, 200 concurrent sessions (via connection pooling), and 5-10 concurrent queries. Another customer has 3500 Cognos users. 1-200 concurrent queries appears to be the record peak load. Anyhow, Exasol says that plans to offer real workload management could be accelerated if a need were discovered.

Exasol says it almost never loses POCs, but admits that it competes fairly rarely against Vertica and ParAccel, no doubt for reasons of geography. Exasol boasts one visible Sybase IQ replacement (Sony Music).

While Exasol’s sales to date have been in Germany, there are plans to change that soon. At least one sales cycle is well underway in Eastern Europe. Offices in other Germanic countries are planned. Existing customers are planning to deploy additional copies outside Germany. Discussions are underway regarding other geographies, e.g. English-speaking ones.

• Vertica Community Edition has all of regular Vertica’s features. However …
• … HP Vertica reserves the right to open a feature gap in future releases.
• The license restriction on Vertica Community Edition is that you’re limited to 1 terabyte of data, and 3 nodes. I imagine that’s for one production copy, and you’re perfectly free to also set up mirrors for test, development, disaster recovery, and so on. However …
• … HP Vertica would be annoyed if you stuck a free copy of Vertica on each of 50 nodes and managed the whole thing via, say, Hadapt.
• HP Vertica plans to be very generous with true academic researchers, suspending or waiving limits on database size and node count. Not coincidentally, Vertica Community Edition is being announced at XLDB, where Vertica is also a top-level sponsor. (I introduced Vertica and XLDB’s Jacek Becla to each other as soon as I heard about Vertica’s Community Edition plans.)
• The only support available for Vertica Community Edition is through forums. This could change.

I’m a big supporter of the Vertica Community Edition idea, for four reasons:

• It should now be easier to download and evaluate Vertica.
• Vertica Community Edition could be a big help to academic researchers.
• Vertica could now be more appealing to some of the “Omigod, we’re outgrowing Oracle Standard Edition and we don’t want to pay up for Oracle Enterprise Edition/Exadata” crowd.
• People are under the impression that what Vertica actually charges today resembles its long-ago list prices. This announcement may help puncture Vertica’s outdated pricing image.

• salesforce.com. This is the parent company, and the runaway leader in the SaaS (Software as a Service) enterprise application market, especially in the area of CRM (Customer Relationship Management).
• force.com. This is salesforce.com’s application development stack split out for other SaaS vendors to use, both inside and outside the CRM segment. It can be referred to as a PaaS offering (Platform as a Service). force.com relies on a proprietary salesforce.com language called APEX, which has a strong stored procedure/ database trigger orientation.
• database.com. This is the database part of force.com, spun out separately in general availability as of Dreamforce two weeks ago.
• data.com. Also launched at Dreamforce (and based, if I understand correctly, on an acquisition), this is a provider of 3rd-party data you might use as inputs to your CRM systems.
• Heroku. Another salesforce.com acquisition, Heroku is in essence a PaaS competitor to force.com. Heroku is focused on Ruby and Java, and supports a number of DBMS, SQL and NoSQL alike.
• AppExchange. This is a marketplace for things designed to integrate with salesforce.com (and perhaps also apps built on force.com). The latest claim is that there are 1200+ AppExchange offerings.
• The complete set of SaaS apps built on force.com. A 2008 white paper refers to 47,000 organizations being “supported” by force.com. Recently I’ve heard a figure just under 100,000. I’m not clear as to what that metric measures — aggregate users of SaaS apps built  via force.com? Clearly there are a lot of SaaS apps built on force.com, with actual customers, but I don’t know how big “a lot” is. (Perhaps a salesforce.com person could chime into the comment thread with some clarity.)

The pricing for force.com and database.com is clearly designed for enterprise SaaS applications whose users will be inside customer organizations. If you want to do something public-facing, prices are prohibitive without a special deal. That’s the bad news. The good news is that salesforce.com says, publicly and privately, that it’s indeed open to cutting such volume pricing deals.

When I talked with CTOs and the like at some Dreamforce-exhibiting SaaS vendors, on the whole they seemed very happy with force.com. The one repeated complaint was that force.com imposed unpleasant rate limits (e.g., number of API calls). Working around those limits involves unnatural acts of coding, phones calls to helpful salesforce.com staffers to get the limits raised, or both. When I talked with salesforce.com cofounder Parker Harris, he seemed painfully aware of the problem, and indicated that relaxing the limits is an important technical goal.

The other force.com weakness I uncovered was expected — while it may be great as long as your application matches its implicit assumptions, there are some things it can’t easily do. This has been a recurring issue since database-oriented 4GLs (Fourth-Generation Languages) came around in the 1980s. For example, one firm wanted a Flash UI — I’m not sure why — and went outside force.com for that part of the application.

Related link

• The database architecture of salesforce.com, force.com, and database.com

• A Kaminario K2 DRAM-only appliance costs $100K.
• A Kaminario K2 flash-only appliance costs $30K (but nobody buys that configuration).
• A typical Kaminario K2 hybrid DRAM/flash appliance might cost $35K (which tells us that there’s a lot more flash than DRAM).

Kaminario positions DRAM as where you focus your most write-intensive/ bottlenecking loads, such as logging or temp space, with the primary benefit being performance and a secondary benefit being slowing the wear on your flash.

If you want even your mirrors to be on flash — which Kaminario says greatly reduces the temporary performance hit in case of a failure — there will be an additional charge. Perhaps Kaminario will dig up a price number and post it in the comment thread.

The flash comes in via Fusion-io cards. Kaminario stresses that it sells a SAN (Storage Area Network) kind of offering, as opposed to the shared-nothing way one might otherwise use Fusion-io cards in servers’ PCIe slots. Kamanario further asserts its built-in high availability is both smoother and less costly than Texas Memory Systems or Violin Memory alternatives; Kaminario is generally proud of its high availability features, down to redundant uninterruptible power supplies. Apparently the sweet spot of Kaminario’s market is single-chassis 5-6 TB systems, but Kaminario asserts seamless elasticity even if you grow into a second chassis.

Price resistance seems to have gotten strongly in the way of Kaminario’s growth, although the company was evasive about customer counts and the like. But it does now have 60+ employees and an aggressive hiring plan, vs. <50 when I wrote about Kaminario a year ago. I do believe that many enterprises would benefit from throwing solid-state storage at certain performance problems, at least as a band-aid, while they contemplate software changes.* But evidently Kaminario has had difficulties — especially at the DRAM-only price point — getting customers to agree, or at least to agree that Kaminario K2 was a sufficiently cost-effective way to address the issue.

*If you like, you can regard this as deferring repayment of your technical debt.

Kaminario’s comments about how its technology is or will be applied are all over the place (again, I think part of this is due to having a small number of customers overall, and wanting to conceal how small that number is). But in general Kaminario has seen more OLTP (OnLine Transaction Processing) than analytic uptake, which contributes to them thinking that low latency is a bigger deal than raw IOPS (Input/Output Per Second). Certainly Kaminario is focused on database applications of some kind or other, generally running on big-name DBMS such as Oracle or Microsoft SQL Server

Let’s try eight categories instead. While no categorization is ever perfect, these each have at least some degree of technical homogeneity. Figuring out which types of analytic database you have or need — and in most cases you’ll need several — is a great early step in your analytic technology planning. 

Enterprise data warehouse (Full or partial)

• Kinds of data likely to be included: All, but especially operational
• Likely use styles: All
• Canonical example: Central EDW for a big enterprise
• Stresses: Concurrency, reliability, workload management

The enterprise data warehouse (EDW) ideal says that you copy all your data into one place, and drive all decision-making from there. Full EDWs are pipedreams. Still, a partial EDW makes sense for most large enterprises, and many indeed already have one. The first product lines to consider for classical EDWs are Teradata, DB2, Exadata, and maybe Microsoft SQL Server, especially if you’re going to stress concurrency and/or operational use cases.

Traditional data mart

• Kinds of data likely to be included: All
• Likely use styles: Business intelligence, budgeting/consolidation, investigative
• Examples: Reporting servers, planning/consolidation servers, anything MOLAP, etc.
• Stresses: Performance, concurrency, TCO

Whether or not you have something like an enterprise data warehouse, it’s common to have lighter-weight data marts as well. A traditional data mart might drive reports and dashboards. Or it might be specialized for budgeting, planning, and/or consolidation.  Some investigative analytics may be in the mix as well.

Any DBMS that can support an EDW can also support a data mart, but it may not be the most cost-effective way to do so. Columnar DBMS might have more attractive performance and TCO (Total Cost of Ownership); the same goes for Netezza. Some of them — e.g. Sybase IQ and Vertica — have excellent track records in concurrent usage as well. Ted Codd pushed what amounts to MOLAP (Multidimensional OnLine Analytic Processing) systems for these use cases. But relational DBMS commonly do a better job, which is one reason most major MOLAP products have wound up at RDBMS companies.

Investigative data mart — agile

• Kinds of data likely to be included: All, especially customer-centric
• Likely use styles: Investigative
• Canonical example: A few analysts getting a few TB to examine
• Stresses: Ease of setup/load, ease of admin, price/performance

Besides the traditional data mart, there are at least two other kinds. Both are focused on investigative analytics, but they’re differentiated by database size.

If you have just a few analysts,* looking at no more than a few terabytes of data (perhaps even just some gigabytes) — and if that data is “single-subject” and fairly homogenous — your watchwords should be “cheap”, “easy”, and “fast”. You don’t need to invest in much hardware, in expensive software, in much administrative effort (the analysts can be their own DBAs),  nor should you endure much set-up time. Just grab a product, grab some data, and start running queries (or extracts into the statistical tool of your choice).

*If you have dozens or even hundreds of analysts hitting the same database, you’re probably back to the more concurrency-oriented scenarios outlined above.

Infobright is often cost-effective among columnar analytic DBMS. Other vendors might cut you a price break as well. If you have multiple terabytes of data, don’t rule out Netezza’s lowest-end products (even if they’d really rather sell you something bigger). Or, if you’re in the sub-terabyte range, maybe you can get by with an in-memory BI tool such as QlikView, and not do anything special on the DBMS side at all.

Investigative data mart — big

• Kinds of data likely to be included: All, especially customer-centric, logs, financial trade, scientific
• Likely use styles: Investigative
• Canonical example: Single-subject 20 TB – 20 PB relational database
• Stresses: Performance, scale-out, analytic functionality

But if you’re looking at tens of terabytes of relational data, or even more, you really do have a “big data” problem. Performance and scalability are major challenges, usually best addressed by MPP (Massively Parallel Processing) systems, such as Netezza, Vertica, Aster Data, ParAccel, Teradata, or Greenplum. Performance POCs (Proofs Of Concept) are a big part of the buying process. Vendor price negotiations are crucial too.

Actually, in the low tens of terabytes you might be able to get away with a shared-disk system that has excellent compression — e.g., columnar products like Sybase IQ, Infobright, or SAND, rather than just Vertica and ParAccel.

Assuming you have affordable, scalable query performance, the competitive differentiator can switch to additional analytic functionality. Aster, Netezza, ParAccel, Vertica, and Greenplum either offer full analytic platforms, or seem to be on the path to doing so. Teradata, which now owns Aster Data, offers substantial built-in analytic capability in its traditional products as well, and the same goes for Sybase IQ.

Continued in Part 2, where we cover some of the more difficult use cases.

Oracle disputes some common claims about its cost and pricing. In particular, Oracle software maintenance costs a fixed 22% of your annual license price, so if you get a discount on your licenses, it ripples through to your maintenance. This is true even if you have an all-you-can-eat ULA (Unlimited License Agreement).

• Based on that, Oracle contends that Exadata isn’t all that expensive if you have a suitable ULA. You have to buy the hardware and the storage software, but the database server software is effectively free. (Whether your use of additional licenses affect the price of your ULA when it comes up for renewal might, of course, be a different matter.)
• Nothing in that discussion obviates the point that if you’re just using Oracle Standard Edition, upgrading to Oracle Enterprise Edition, associated chargeable options, and/or Exadata can be seriously expensive.

Back in April, Steve Miller suggested that approximate BI could be a growing trend, gaining speed at the expense of (often false anyway) precision. That idea of course goes well with Infobright’s recent released Rough Query feature, and also with Datameer’s year-earlier pitch.

Aster Data (now a Teradata company) is positioning itself as analyzing multi-structured data — which is my second-choice term, behind the more precise but odder-sounding “poly-structured.” I hope “poly-structured” wins, and plan to keep using it myself; but I recognize that “multi-structured” may actually be the one that prevails.

Barbara Darrow wrote a great piece on Oracle Exadata pricing. Highlights include:

• Routine Oracle software discounts are high.
• Exadata discounts are higher.
• Big/referenceable customers get the best Exadata discounts. The term “extremely deep” was used. (I’ve also heard that from Oracle competitors, with the term “free” even coming up, hyperbolically or otherwise.)
• Oracle’s hardware maintenance pricing is forcing users to trash Sun gear, even when it’s working. One guy told the story of literally crying as the Sun boxes were taken away.
• Oracle’s 22% of license maintenance fee goes up to 27% after two years. I didn’t know that.

Oracle has been making considerable messaging fuss around a win in Japan, where Softbank replaced years-old Teradata systems with vastly less new Exadata gear. I blogged that this is hardly an apples-to-apples comparison. During my visit last April, Oracle pushed back, in particular pointing out that the Softbank division that awarded the deal was very separate from the one that was an Oracle reseller. But Monday Teradata shared with me a counter-pushback, asserting that during the recent worldwide recession, Softbank assigned its underemployed systems integration division to do internal projects — including the data warehouse upgrade. I.e., Teradata stands by its claim that this replacement was strongly influenced by the Softbank/Oracle partnership.

If you’re analytically inclined, Kx Systems has some interesting ideas, manifested in kdb+ and so on. A 2009 ACM article seems as good a starting point as any, the company’s website probably aside. Confusingly, Kx is small company that evidently does most of its selling through a couple of much larger partners. Also, 1010data happens to be built on an older version of Kx’s technology.

• Hadoop nodes today tend to run on fairly standard boxes.
• Hadoop nodes in the past have tended to run on boxes that were light with respect to RAM.
• The number of spindles per core on Hadoop node boxes is going up even as disks get bigger.

A key input comes from Cloudera, who to my joy delegated the questions to Omer Trajman, who wrote:

Most Hadoop deployments today use systems with dual socket and quad or hex cores (8 or 12 cores total, 16 or 24 hyper-threaded). Storage has increased as well with 6-8 spindles being common and some deployments going to 12 spindles. These are SATA disks with between 1TB and 2TB capacity. The amount of RAM varies depending on the application. 24GB is common as is 36GB – all ECC RAM. HBase clusters may have more RAM so they can cache more data. Some customers put Hadoop on their “standard box” which may not be perfectly balanced (e.g. more RAM, less disk) and needs to be altered slightly to meet the above specs. The new Dell C2100 series and the HP SL170 series are both popular server lines for Hadoop.

For a year ago perspective, see this post: http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/

Bullet points from that year-ago link include:

4 1TB hard disks in a JBOD (Just a Bunch Of Disks) configuration

2 quad core CPUs, running at least 2-2.5GHz

16-24GBs of RAM (24-32GBs if you’re considering HBase)

Gigabit Ethernet

So basically we’re talking in the range of 2-3 GB of RAM per core — and 1 spindle per core, up from perhaps half a spindle per core a year ago.

Meanwhile, a 2009 Yahoo slide deck refers to “500 nodes, 4000 cores, 3TB RAM, 1.5PB disk”; that divides out to 8 cores, 6 GB of RAM, and 3 TB of disk per node, all on “commodity hardware.” By 2010 Yahoo was evidently up to 2 GB of RAM per core.

There are lots of data points on the Apache Hadoop wiki, but many seem a few years old, and I don’t immediately see how to time-stamp them. Overall, they seem consistent with the trends I noted at the top of the post.

One thing I haven’t done is attempted to price any of these systems.

Contributions in the comment thread would be warmly appreciated.

• Product discounts are down, effectively creating a price increase.
• Service prices are way up for some customers, because cheaper service options have been eliminated.
• Service quality is unsatisfactory.
• Oracle is difficult to do business with.

So far, I haven’t seen any comments to the effect “I don’t know what you guys are talking about; we’re perfectly happy with Sun”, but surely those will come too.