• Workday has forward-thinking ideas about SaaS enterprise applications and the integration of business intelligence into same.
• Workday has highly innovative ideas in how it manages data.
• Companies founded by Dave Duffield tend to feature smart, likeable people who talk to one pleasantly and forthrightly. Workday is no exception; CTO Stan Swete and the other Workday folks present were a delight to talk with.
• I’d invited Merv Adrian to come along with me. He asked great questions, and I could gather myself a bit despite how sleep-deprived I was for the first part of that trip.

Workday kindly allowed me to post this Workday slide deck. Otherwise, I’ve split out a quick Workday, Inc. company overview into a separate post.

The biggie for me was the data and object management part. Specifically: 

• Workday’s applications run entirely in-memory, in a highly object-oriented structure. Persistence is mainly for the sake of data safety …
• … but not entirely. In earlier releases, Workday kept absolutely everything in RAM. However, certain things are kept only on disk, such as:
  • Audit files.
  • Certain documents (notably resumes).
• Workday’s whole database – data and metadata alike – is persisted to disk in <10 MySQL/InnoDB tables. MySQL is basically just being used as a key-value store, albeit one with ACID transactional support.
  • There are 3 main tables: attributes, relationships, instances.
  • When I suggested this might be like an entity-attribute-value model, Workday said it would be even better to think in terms of instanceID-attribute-value.
  • As you might expect for a database that simple, its schema doesn’t change much.
  • By way of comparison, Workday estimates that if its software were written relationally, there would be 1000s of tables, which would take up 10-100X as much disk space.
• All write transactions are banged immediately into the MySQL database. I.e., RAM and disk are never allowed to get out of sync.
• Workday’s database is append-only. This is exploited for effective dating (pretty heavily, it seems, perhaps because that’s a useful concept in human resources) and snapshotted reporting.
• Workday’s built-in BI doesn’t have a lot of choice but to do scans, traversing the object model. This turns out to be fast enough.

Other notes on Workday’s data and object management strategy include:

• Workday is object-oriented through and through – no object-relational mapping – turtles all the way down. On average, a class has about 2 attributes.
• 94% of requests are reads, traversing the object hierarchy.
• Workday databases are pretty small.
  • The biggest database Workday supports uses 17 gigabytes of RAM.
  • Workday databases are much smaller on disk than in RAM.
• Workday’s “dream” is to move from disk to solid-state memory.
• Workday uses GPLed MySQL/InnoDB. So there’s no software license reason to ever move away (e.g., to a pure key-value store).
• Disaster recovery is based on local and remote MySQL slaves.

Obviously, serious apps have been built before in object-oriented and/or key-value ways, with the resulting objects then being banged to disk (or in some cases kept in memory). Examples include:

• Numerous applications are built on object-oriented DBMS. Generally they go against disk, although memory-centric implementations can save a lot of pointer-chasing. Often they’re queried via SQL.
• Basho’s website says that its key-value store Riak was originally conceived in connection with a planned salesforce automation product, but I don’t think that the application part of that plan ever got built.
• SAP has longstanding doubts about relational dogma, although not nearly to Workday’s extreme.
• Obviously, some major internet applications just bang data into key-value stores.

Still, perhaps because it wholly object-oriented yet doesn’t even bother with anything like a real object-oriented DBMS, Workday’s approach seems particularly cool.

Other highlights of Workday, Inc.’s technical story include:

• Workday has settled into a schedule of three releases per year, and has pretty much lived up to that for >2 years.
  • Every user is always on the latest Workday release.
  • You can delay turning on significant new Workday software functionality if you want to.
  • Pure UI changes to the Workday software are handled much as they are on various websites today. Sometimes you have no choice but to live with them; sometimes the prior version of the UI remains available to you for a while.
• Workday’s navigational approaches look pretty cool.
  • The core concept is a list of actions you can perform now, rather than more standard menus.
  • Roles/permissions are of course central to this.
  • Reports have lots of actionable links in them. (More than just drilldown, although specific examples have slipped my memory.)
  • Alternatively, you can navigate via a search box, searching both on names of objects (e.g. users, divisions) or on names of tasks. This is somewhat reminiscent of an approach SAP was considering a few years ago.
• Workday says it has four key design premises:
  • Web-Familiar Experience. I’d say that’s true to to the extent it makes sense. In many ways, the web needs to catch up to Workday.
  • Enterprise Reporting. The idea is that you get a report, then take actions based on it. Hence the report-centric options for navigation.
  • Integration On-Demand. That’s a fancy way of saying “Plays nicely with others.”
  • Configurable Business Processes. Duh. That’s pretty essential if you want to do serious SaaS applications.
• Workday maintains a strong separation between application logic and UI development. Developer do no screen layouts. Instead, Uis are automatically generated for:
  • Flash/FLEX
  • iPhone
  • Mobile HTML
  • PDF export
  • Excel export
• Workday only talks to the outside world via web services.
  • Workday is heavily into SOAP (Simple Object Access Protocol).
  • The acquisition of OEM partner CapeClear gave Workday an Integration Service (i.e., enterprise service bus) that translates SOAP into whatever else might be needed for integration, and also does reliable delivery.
  • All that said, Stan Swete sees integration among various SaaS offerings as an area needing significant future attention.
• Workday’s business intelligence ideas are interesting, but I think there’s a long way for that technology still to go.
  • Workday’s BI seems to be focused on report/drilldown kinds of functionality.
    • You can slice by up to 2 dimensions at once.
    • Then you can keep slicing, however, by more dimensions, as many times as you like.
  • While you can take actions straight from reports, some of the specific BI/app integration ideas we discussed are still futures. (E.g., analyzing spend at the time of expense report data entry or approval.)
  • Of course, Workday’s web services interface lets you export Workday data into 3rd-party tools. Indeed, if you want to integrate data from Workday and some other source(s), that’s your only choice.
• Workday offers a clever metaphor to illustrate that your data may be more secure offsite than on – the bank vault. (I have no idea whether that’s a SaaS industry standard, but I hadn’t heard it before.) Of course, that metaphor does beg some issues specific to the remote data case, such as:
  • When your data is on premises, you know whether the government has insisted on looking at it.
  • More than cash, data keeps traveling back and forth to the remote location, which creates at least a theoretical risk of interception.
• Workday says the toughest part of globalization is the issue of which personal data is or is not maintained. For example, in the US you’re not allowed to not ask a job applicant’s religion, but in the UK you’re not only permitted but indeed required to.

This post is part of a three-post series

• Workday Inc. company overview (brief)
• Workday Inc. technology overview (detailed)
• Workday Inc. CTO Stan Swete’s comments on database strategy

• If you are or want to be a serious MapReduce user – and you’re past the “play around over the weekend” stage — you probably should have either:
  • A serious non-DBMS MapReduce distribution.
  • MapReduce integrated into your analytic DBMS.
  • Both.
• The obvious choice for non-DBMS MapReduce is Hadoop.
• The obvious choice for a Hadoop distribution is Cloudera Enterprise.
• Cloudera Enterprise has three main aspects, in an inseparable bundle:
  • Distributions for a double-digit number of open source projects. It’s nice having all that in one package – unless, of course, you like playing with Tinkertoys.
  • Proprietary Cloudera code.
  • Cloudera support.
• Cloudera says its proprietary code is and in the future is planned to be concentrated – at least in large part — on integrating open source technology with closed source products. This has the virtue of being targeted directly at that segment of the market which has proven it’s actually willing to pay money for software.
• Cloudera Enterprise areas of focus, now and in the presumed future, include:
  • Core Hadoop engine, which Cloudera says is quite predictably and appropriately evolving more slowly than the tools around it.
  • Development, management and administrative tools, including:
    • Pig and Hive. Cloudera says >70% of Facebook Hadoop jobs are initiated through Hive, and the same is true of Yahoo and Pig.
    • Connectivity to commercial tools.
    • The product formerly known as “Cloudera Desktop.”
  • Workflow, which in this context refers to letting you create a Hadoop application as a sequence of small steps, rather than forcing you to kluge it into being one unwieldy thing. At the moment, this is much less widely adopted than Pig and Hive, but Cloudera has high hopes for it, because of its obvious benefits in modularity and manageability.
  • Quasi-DBMS technology. Besides Hive and Pig, this includes HBase. Cloudera says there has been considerable demand for HBase, and it is pleased that project is now mature enough to ship. Cloudera stresses that it intends HBase not for OLTP, but as an adjunct to analytic processing. E.g., Cloudera suggests HBase would be a fine vehicle for replicating dimension tables across each node of a cluster.
  • Data connectivity, e.g. to MySQL or to sensor log files.
• Cloudera Enterprise pricing is well below DBMS prices – not by a full order of magnitude, if I’m right about everybody’s quantity discount policies, but even so by a lot. Details are NDA.

Cloudera sometimes sends confusing signals about its beliefs and strategies. For example, one can get different stories depending on whether one talks to:

• Somebody at Cloudera who comes primarily from the user and open source communities.
• Somebody at Cloudera who has actually worked at a software company before.

But I predict that Cloudera will now stick for a while with more or less the strategy outlined above.

Naturally, we also talked about Hadoop adoption. Highlights of that part – no doubt somewhat biased towards Cloudera’s own customer base — included:

• Notwithstanding eBay’s prior skepticism about MapReduce, it is quoted saying nice things in a Cloudera press release, and has apparently become quite a large Hadoop user, starting out with a search-quality use case.
• Typical Hadoop deployment sizes are 10 nodes or so when experimenting, 80-500+ in production.
• 10 terabytes/node – I’m pretty sure Cloudera meant of user data — is not inconceivable, so a cost-conscious 500-node user could have 5 petabytes of data managed by Hadoop.
• Cloudera has half a dozen customers at the 75+ node production level.
• Web and financial services are the two vertical markets moving most aggressively into Hadoop production. The government is also in significant Hadoop production, but the details of that are classified.
• Web uses for Hadoop include:
  • Clickstream – sessionization, etc. – that’s a super-mainstream use.
  • Search – analyzing search attempts in conjunction with structured data.
  • Machine learning (for ad serving, etc.).
• Financial services uses for Hadoop include:
  • Internal trading rule enforcement/fraud detection.
  • Complex ETL.
  • Portfolio risk assessment (typically overnight).

None of this is inconsistent with previous surveys of Hadoop use cases.

Various users talked at the Hadoop Summit this week. I wasn’t there, and won’t write about their stories for now. That said, Twitter’s slide deck from same has some interesting stuff, including:

• 7 TB/day ETLed from MySQL.
• Petabytes-being-stored accordingly coming soon.
• Open sourcing their ETL tool Crane.
• 3-4X LZO compression at little CPU cost.
• HBase is a more usable for them than HDFS, which isn’t mutable enough.
• Pig = 5% of code and coding effort vs. vanilla Hadoop at 30% or less performance hit.

*The comment threads are the most informative parts of those posts.

I’m a little unclear as to the Netezza Migrator details, not least because Netezza folks don’t seem to care too much about Netezza Migrator themselves. That said, the core ideas of Netezza Migrator are: 

• Netezza Migrator is an enhanced (?) version of EnterpriseDB’s Postgres Plus Advanced Server DBMS. (Recall that Postgres Plus is PostgreSQL-based and fairly Oracle-compatible.)
• Netezza Migrator does not run on Netezza appliances, but rather on conventional computers off to the side.
• Netezza Migrator generally farms out queries to Netezza appliances, but can also manage data itself. (That latter part could supposedly come in handy for small tables one might want to execute stored procedures against.)
• Netezza Migrator does a better job of farming out queries (and also inserts/updates/loads) to Netezza appliances than an Oracle DBMS would. The two biggest examples of that are:
  • Oracle will farm out SELECTs, but not JOINs.
  • Oracle won’t invoke Netezza’s parallel/bulk load capabilities.

• Netezza now intends to come out with a Flash-based appliance earlier than it originally expected.
• Indeed, Netezza has suspended — by which I mean “scrapped” — prior plans for a RAM-heavy disk-based appliance. It will use a RAM/Flash combo instead.*
• Tim Vincent of IBM told me that customers seem ready to adopt solid-state memory. One interesting comment he made is that Flash isn’t really all that much more expensive than high-end storage area networks.

Uptake of solid-state memory (i.e. Flash) for analytic database processing will probably stay pretty low in 2010, but in 2011 it should be a notable (b)leading-edge technology, and it should get mainstreamed pretty quickly after that. 

*So far as I can tell, that’s one of the two significant roadmap changes between the 2009 and 2010 editions of Enzee Universe. The other one is that the robust form of appliance-to-appliance replication technology is coming out later than Netezza had originally planned and hoped.

There also is increasing reason to think that the issues with Flash memory wearing out are overwrought.  And by the way, the entire history of enterprise solid-state memory use is basically shorter than the time in which these products supposedly will wear out, so it’s not as if there have been a lot of real-life failures out there.)

• First, clever things are being done in the area of error correction codes, although for the most part I defer that part of the discussion to Petascan’s Camuel Gilyadov. E.g., this seems to be the idea behind Anobit.
• Second, analytic DBMS are pretty much an ideal use case for Flash reliability. Suppose, as is the case for many products and implementations, you only write things in big blocks. Then you are, ipso facto, resetting the Flash bits only in big blocks. Thus, at least in theory, you automatically have pretty perfect wear leveling.

• VoltDB is based on the H-Store technology, which I wrote about in February, 2009. Most of what I said about H-Store then applies to VoltDB today.
• VoltDB is a no-apologies ACID relational DBMS, which runs entirely in RAM.
• VoltDB has rather limited SQL. (One example: VoltDB can’t do SUMs in SQL.) However, VoltDB guy Tim Callaghan (Mark Callaghan’s lesser-known but nonetheless smart brother) asserts that if you code up the missing functionality, it’s almost as fast as if it were present in the DBMS to begin with, because there’s no added I/O from the handoff between the DBMS and the procedural code. (The data’s in RAM one way or the other.)
• VoltDB’s Big Conceptual Performance Story is that it does away with most locks, latches, logs, etc., and also most context switching.
• In particular, you’re supposed to partition your data and architect your application so that most transactions execute on a single core. When you can do that, you get VoltDB’s performance benefits. To the extent you can’t, you’re in two-phase-commit performance land. (More precisely, you’re doing 2PC for multi-core writes, which is surely a major reason that multi-core reads are a lot faster in VoltDB than multi-core writes.)
• VoltDB has a little less than one DBMS thread per core. When the data partitioning works as it should, you execute a complete transaction in that single thread. Poof. No context switching.
• A transaction in VoltDB is a Java stored procedure. (The early idea of Ruby on Rails in lieu of the Java/SQL combo didn’t hold up performance-wise.)
• Solid-state memory is not a viable alternative to RAM for VoltDB. Too slow.
• Instead, VoltDB lets you snapshot data to disk at tunable intervals. “Continuous” is one of the options, wherein a new snapshot starts being made as soon as the last one completes.
• In addition, VoltDB will also spool a kind of transaction log to the target of your choice. (Obvious choice: An analytic DBMS such as Vertica, but there’s no such connectivity partnership actually in place at this time.)

I should also note that when Tim Callaghan described architectural options to get around 2PC performance issues, they sounded a lot like eventual consistency. Maybe tunable RYW consistency isn’t in the cards, but at least there’s a NoSQL-like possibility with VoltDB.

VoltDB’s open source strategy is:

• VoltDB will be open sourced.
• Community VoltDB will be GPLed. Professional Edition VoltDB has a non-GPL license.
• The VoltDB Professional Edition won’t start out with features beyond the Community Edition ones, but will gain such later on. I didn’t get the sense the plans for those features were completely baked yet, but ideas mentioned included:
  • Management/monitoring tools.
  • Integration with expense closed-source enterprise software products, such as ones in the management/monitoring area.
  • Yet more “extreme”/edge-case performance.
• Before VoltDB decided for sure that it wasn’t selling licenses, it sold a license to Getco, which also seems to be an investor in the company.

VoltDB had a beta test with about 150 participants. None is in production yet, although at least a few are clearly headed there. Most VoltDB beta testers are in some kind of online business, with a particular concentration in everybody’s new favorite market, online gaming. Most of the rest are in investment/trading — a major target market for at least three different Mike Stonebraker companies — and a few are in telecom. VoltDB assures me that some of the beta users are companies one actually has heard of before, but VoltDB is not in a position to name any of those.

VoltDB is not ideally suited for a classic order management system, since you’d want to partition both on CustomerID and SKU, the latter because you’d constantly updating inventory stock levels. However, this argument doesn’t apply in the case of virtual goods. Virtual goods that are sold for real money — and hence need ACID levels of transaction integrity — are thus a clear target market for VoltDB. (The example that came up was in, you guessed it, online gaming.) The other interesting use case that Tim highlighted was low-latency analytics/ELT. For reasons I didn’t totally grasp, Tim likes to call this “Stateful ELT.” (Given that the data goes into the VoltDB database before much else happens to it, I’m pretty sure I heard “ELT” correctly. But I guess I might have been mishearing “ETL”.)

VoltDB company highlights include:

• VoltDB has about a dozen employees, all but two of whom are technical. (However, I’m not sure they’re counting Andy Ellicott against the two. But then, last I heard he wasn’t full time at VoltDB.)
• VoltDB’s venture funding status is, if I may paraphrase, “Mumble mumble.”
• Although long separate from Vertica, VoltDB is still located in Vertica’s offices.

The bulleted list in my notes is:

• Individual-human

  • Expense of expertise

  • Limited numeracy

• Organizational

  • Limited budgets

  • Legacy systems

  • General inertia

• Political

  • Obsolete systems

  • Clueless lawmakers

  • Obsolete legal framework

I shall explain.

The expense of expertise. Highly skilled Oracle DBAs are expensive. The same can be said for many other categories of people, whether in IT or business units, needed to exploit the opportunities of analytic technology. Newer, simpler approaches to analytic database management can clearly help. The verdict is more mixed so far on newer business intelligence or data mining technologies.

Limited numeracy. If you’re reading this, you’re very likely more numerate (i.e., more capable with numbers, mathematics, and analysis) than the average person, or indeed than the average knowledge worker. A typical knowledge worker can understand an analytic claim, on some level – but can he think critically about it? Can he make valid analytic arguments of his own? That’s less clear. Maybe it’s possible to build a new company in which analytic competence is a prerequisite for employment (Google comes to mind as one candidate.) But uniform analytic ability is almost inconceivable at an established enterprise. That’s one of the biggest reasons converting an ongoing concern into a top-to-bottom “analytic enterprise” is a lot harder than business intelligence gurus sometimes suggest.

Actually, I do think practical numeracy goes up each decade. Go, for example, to a message board discussing sports, and the chances are high you’ll pretty soon see concepts like “small sample size” used perfectly correctly. That would have been much less likely 30 years ago, even aside from the fact that in those days no such things as “message boards” happened to exist. I credit this favorable trend to a multitude of factors, from the available of BI and related technologies, to the promotion of probability and statistics at multiple levels of at least the United States educational curriculum. If nothing else – when I was in school, it was very rare for a girl to be serious about learning calculus or any kind of advanced mathematics.* Thankfully, that seems to have changed.

*I do recall my main college girlfriend cursing her way through a physical chemistry course, which implies some working knowledge of partial differential equations. But she was pretty unusual for her day.

Even so, that’s not enough. Get me in a discussion about politics or charity, and I’ll argue that few things are more important than boosting numeracy through the education system. But that’s hardly an answer in a business-today time frame. If your enterprise is trying to deploy analytics universally across the organization today – well, that’s a very hard challenge.

Limited budgets. For every benefit there is a cost. And when it comes to analytics, the costs (at least some of them) can be a lot easier to quantify than the benefits. This can make it hard to get the budget to do proper analytics.

Legacy systems. In many cases, the best and most cost-effective analytic products are fairly new ones. But companies tend to have the older and costlier ones already in place. Yes, I’m seeing a reasonable number of “escape from Oracle” or even “escape from DB2” projects. But a lot more enterprises just try to work within what they have.

And that’s even before we start to talk about stovepipes, silos, or data integration. It’s also before we consider the difficulty of modernizing OLTP systems to either gather more data for analysis or incorporate more analytics into operational business processes.

General inertia. Budget limitations and legacy limitations can both be excuses for doing nothing. But something nothing gets done even without the aid of such excuses.

Obsolete systems. In the particular case of government, the legacy systems problem can be ridiculously bad. In large part, this is due to broken procurement processes. Computer systems are bundled up into specific government contracts, which are then bid and awarded in a long process, whose results can be challenged, making the process even longer. When the contract is finally awarded, a long implementation cycle begins. Systems are years out of date before they are put in. By the time they are replaced, they are a lot more ancient than that. And, as big as they are, they are isolated projects, with little cross-government standardization, and with little effort put into facilitating data integration across systems.

The consequences are horrific. Better collaboration tools might have averted the 9/11 attacks. Weak analytics allow all kinds of fraud to continue. And considering just the costs of data integration projects, untold billions of dollars are directly spent due to government computer failings.

Obsolete legal framework. The procurement process is just one major example of obsolete technology-related laws. The liberty and privacy problem is even more profound. Intellectual property rights and censorship,* of course, are other major areas of nonsense, even in countries that are basically pretty free. Part of the problem is that legal structures for a less advanced world are sometimes ill-suited for the power of the modern technology.

*Of course, those areas are not particularly tied to analytics. But I thought I’d throw them in anyway while I was on a roll.

Technology-challenged lawmakers. These issues are hard, and even technologically savvy lawmakers would struggle with them. At least, I think so; to my knowledge, that hypothesis has never come close to being tested. What I do know is that when laws meet technology, nonsense commonly ensues.

Cast Iron sold an integration appliance, most focused on integrations that involved SaaS applications such as Salesforce, with an option for doing all this purely in the cloud. IBM is accordingly spinning Cast Iron as a major cloud player, which is something of an exaggeration.

IBM will surely get value from whatever specific connectors Cast Iron does a better job at than IBM’s current offerings do. What I’m more curious about is whether Cast Iron’s core technology will survive in a form that continues it’s core message of “simplicity, simplicity, simplicity.”

• ITA Software builds huge OLTP systems that it runs itself on behalf of airlines.
• Very, very unusually, ITA Software builds these huge OLTP systems in LISP.
• ITA Software is an Oracle shop (see Dan Weinreb’s comment).
• ITA Software is run by a techie (again, see Dan Weinreb’s comment).
• ITA Software has an interesting screen-scraping/web ETL project called Needlebase

ITA’s software does both price/reservation lookup/checking and reservation-making. I’ve had trouble keeping it straight, but I think the lookup is ITA’s actual business, and the reservation-making is ITA’s Next Big Thing. This is one of the ultimate federated-transaction-processing applications, because it involves coordinating huge OLTP systems run, in some cases, by companies that are bitter competitors with each other. Network latencies have to allow for intercontinental travel of the data itself.

Indeed, airline reservation systems are pretty much the OLTP ultimate in themselves. As the story goes, transaction monitors were pretty much invented for airline reservation systems in the 1960s.

A really small project for ITA Software is Needlebase. I stopped by ITA to look at Needlebase in January, and what it is is a very smart and hence interesting screen-scraping system. The idea is people publish database information to the web, and you may want to look at their web pages and recover the database records it is based on. Applications of this to the airline industry, which has 100s of 1000s of price changes per day — and I may be too low by one or two orders of magnitude when I say that — should be fairly obvious. ITA Software has aspirations of applying Needlebase to other sectors as well, or more precisely having users who do so. Last I looked, ITA hadn’t put significant resources behind stimulating Needlebase adoption — but Google might well change that.

Edit: I just re-found an old characterization of (some of) what ITA Software does by — who else? — Dan Weinreb:

I am working on our new product, an airline reservation system.  It’s an online transaction-processing system that must be up 99.99% of the time, maintaining maximum response time (e.g. on www.aircanada.com).  It’s a very, very complicated system.  The presentation layer is written in Java using conventional techniques.  The business rule layer is written in Common Lisp; about 500,000 lines of code (plus another 100,000 or so of open source libraries).  The database layer is Oracle RAC.  We operate our own data centers, some here in Massachusetts and a disaster-recovery site in Canada (separate power grid).

Related links

• ITA Software and Needlebase websites
• More about LISP

• Datameer offers a business intelligence and analytics stack that runs on any distribution of Hadoop.
• Datameer is still building a lot of features that it talks about, for target release in (I think) the fall.
• Datameer’s pride and joy is its user interface. Very laudably for a software start-up, Datameer claims to have spent considerable time with professional user interface designers.
• Datameer’s core user interface metaphor is formula definition via a spreadsheet.
• Datameer includes 124 functions one can use in these formulae, ranging from math stuff to text tokenization.
• Datameer does some straight BI, with 4 kinds of “visualization” headed for 20 kinds later. But if you want to do hard-core BI, use Datameer to dump data into an RDBMS and then use the BI tool of your choice. (Datameer’s messaging does tend to obscure or even contradict that point.)
• Rather, Datameer seems to be designed for the classic MapReduce use cases of ETL and heavy data crunching.
• Datameer’s messaging includes a bit about “Datameer is real-time, even though Hadoop is generally thought of as batch.” So far as I can tell, what that boils down to is …
• … Datameer will let you examine sample and/or partial query results before a full Hadoop run is over. Apparently, there are three different ways Datameer lets you do this:
  • You can truly query against a sample of the data set.
  • You can query against intermediate results, when only some stages of the Hadoop process have already been run.
  • You can drill down into a “distributed index,” whatever the heck that means when Datameer says it.
• Datameer will let you import data from 15 or so different kinds of sources, SQL, NoSQL, and file system alike.

]]> http://www.dbms2.com/2010/04/16/introduction-to-datameer/feed/ 2 http://www.dbms2.com/2010/04/12/greenplumchorus/ http://www.dbms2.com/2010/04/12/greenplumchorus/#comments Mon, 12 Apr 2010 11:54:39 +0000 Curt Monash http://www.dbms2.com/?p=1887 Greenplum is making two product announcements this morning. Greenplum 4.0 is a revision of the core Greenplum database technology. In addition, Greenplum is announcing Greenplum Chorus, which is the first product release instantiating last year’s EDC (Enterprise Data Cloud) vision statement and marketing campaign.

Greenplum 4.0 highlights and related observations include:

• For the most part, Greenplum 4.0 is focused on general robustness catch-up and Bottleneck Whack-A-Mole, much like the latest releases from fellow analytic DBMS vendors Vertica and Aster Data.
• Greenplum has switched its replication approach from logical (execute transactions against two different disks) to block-level (just ship over the blocks that were changed by the original transaction). This seems to increase a Greenplum database’s robustness/performance/uptime in the face of disk/node failure. It also provides Greenplum with an ongoing performance advantage in that data only has to be compressed once in total for both disk writes.
• The Greenplum DBMS now has something called “tablespaces,” which sounds as if it extends Greenplum’s “polymorphic storage” to accommodate different kinds of storage device. Everybody has to do and for the most part is doing this, e.g. Teradata and Sybase. At least for now, you need to have the same mix of storage technology at every Greenplum node. That said, while Greenplum’s customers will surely want solid-state storage in the future, that’s not quite yet a major current issue.
• The timetable on Greenplum 4.0 is a salami-thin-slicer’s delight:
  • Greenplum 4.0 has been used in POCs (Proofs of Concept) for a while.
  • Greenplum 4.0 has been in early access for a few weeks.
  • Greenplum 4.0 controlled availability is planned for the end of April.
  • Greenplum 4.0 general availability is planned around the end of May or early June.
  • (Note: Everything in Greenplum 4.0 has been built, and is undergoing QA).
• Greenplum has put together a nice list of big-name customers, including Fox/MySpace, eBay, Sears, and T-Mobile. While Fox/MySpace never got to the predicted 1-petabyte level of user data, T-Mobile is loosely projected to indeed get there. The same 1-petabyte projection is made more confidently about another Greenplum telecom customer (unnamed), which seems to be in the process of acquiring a 300-node Greenplum system.

The really interesting part of this announcement, however, is Greenplum Chorus. Greenplum agrees with my assertion that Greenplum Chorus is a new kind of data integration/ETL technology. In particular, Greenplum Chorus is designed around a stance I agree with, namely it’s unrealistic to put everything into a single enterprise data warehouse (EDW); you need to manage data marts as well, preferably in a coordinated way. Mainstream data integration/ETL (Extract/Integration/Load) vendors such as Informatica or Ab Initio would surely say “That’s often quite true, and our technology can handle such scenarios just as it handles single-EDW-data-sink environments.” But Greenplum Chorus offers three capabilities not generally found in traditional data integration products (and offers only those three capabilities), namely:

• Spin out data marts, whether by recopying the data or by creating a virtual data mart inside another data warehouse/mart.
• Find/discover data in databases across your enterprise.
• Do social networking around databases/data marts.

Greenplum Chorus is heading into early access soon, with general availability slated around midyear. Also in the mix is a Greenplum “Hypervisor” that can somehow relate to an almost unlimited number of nodes or databases; however, I didn’t get a lot of details on the Greenplum Hypervisor technology or on the target dates for delivering and integrating the Hypervisor with other parts of Greenplum’s technology.

When Greenplum first talked about about the enterprise data cloud (EDC) idea, it emphasized the spinning out of physical data marts in an easy way, as opposed to the virtual data marts pushed by Oliver Ratzesberger and Teradata. Greenplum Chorus, however, supports both kinds (as, at least directionally, does Teradata), specifically letting you choose between:

• “Independent sandboxes” – physical copies of the data, in a separate Greenplum database instance.
• “Satellite sandboxes” – virtual data marts, of course managed by the same Greenplum database instance.

Actually, if you want to recopy data in the same Greenplum database instance, you can do that too, via something called “data sets,” but that’s not the main focus. Either option, I presume, can be configured to provide either or both of the two main benefits of spun-out data marts, namely:

• Control over the performance and SLAs (Service-Level Agreements) of your analytic workload
• Ability to mix in new raw data and/or new aggregations

in either case without messing up the performance, SLAs, security, or “one truth-ness” of the existing database.

To provide those capabilities in an analytic DBMS, you need sufficiently robust parallel data movement (for the physical sandboxes) and workload management (for the virtual ones). Greenplum obviously believes it has both. Teradata makes the same claim. Other vendors would make similar assertions, and presumably will offer similar capabilities soon. You also want some kind of ability to ingest data from foreign databases, but that can be pretty routine stuff; e.g., in Release 1 of Chorus, Greenplum is content to offer ODBC access to Oracle, SQL Server, et al.

The “data discovery” and “social networking” aspects of Greenplum Chorus seem to be quite Release 1 as well. Basically, Greenplum lets people post discussion threads about databases and data marts, discussing what value can be derived from them. I guess somebody could include links to web-technology reports based on those databases, but otherwise there’s no integration with business intelligence tools and their collaboration capabilities. Even so, Greenplum reports that business executives liked this capability in early access testing.

Greenplum Chorus is ETL without a lot of T, and without a lot of performance optimizations either. That may not be much of a problem in its paradigmatic use case, spinning out a data mart quickly for some analysis to see if valuable conclusions can be drawn. Presumably, in the most successful cases, business and technical processes would emerge after the fact to pipe up-to-date versions of that analysis into operational systems, mooting any ETL deficiencies in the initial exploration moot. In a world where “data exploration” is becoming an increasingly important concept, something like Greenplum Chorus may suffice to provide significant customer value. But whether Greenplum Chorus’s capabilities are eventually co-opted by more fully-featured data integration suites remains an open question for the future.