I would say thousands. The object model for our applications consists of over 2000 classes. On average these classes have multiple relationships with other classes so that would have some kind of multiplicative effect when it came to using tables.

One example of where you’d be proliferating tables (and not getting as satisfactory of a solution relationally) is worktags. Currently we have a class for worktags. Instances of this class can point to various instances of detail lines (expense lines, po lines, invoice lines, etc…). A detail line can have many worktags pointing to it. To model this relationally you’d need either a separate table for each type of detail line in the system to store the tags associated with it or a single worktag for detailed line table that could be foreign keyed for all types of detail lines that would store their worktag. Either way involves more tables and more clunkiness.

Another example of where our oo designs wouldn’t directly translate is our ability to describe to shared part of a detail line in one class and have all instances of detail lines inherit the fields that are shared. To do this relationally you’d probably replicate the shared fields in each table representing the various kinds of transactional details (again lines, po lines, invoice lines, etc…). You’d lose the ability to maintain and change the shared fields (and the processing logic for those fields) in one place.

Anyway, I’d go with “thousands” as our answer. I do think this is an interesting question and wish we had more time to figure out a more accurate answer.

This post is part of a three-post series

• Workday Inc. company overview (brief)
• Workday Inc. technology overview (detailed)
• Workday Inc. CTO Stan Swete’s comments on database strategy

• Workday has forward-thinking ideas about SaaS enterprise applications and the integration of business intelligence into same.
• Workday has highly innovative ideas in how it manages data.
• Companies founded by Dave Duffield tend to feature smart, likeable people who talk to one pleasantly and forthrightly. Workday is no exception; CTO Stan Swete and the other Workday folks present were a delight to talk with.
• I’d invited Merv Adrian to come along with me. He asked great questions, and I could gather myself a bit despite how sleep-deprived I was for the first part of that trip.

Workday kindly allowed me to post this Workday slide deck. Otherwise, I’ve split out a quick Workday, Inc. company overview into a separate post.

The biggie for me was the data and object management part. Specifically: 

• Workday’s applications run entirely in-memory, in a highly object-oriented structure. Persistence is mainly for the sake of data safety …
• … but not entirely. In earlier releases, Workday kept absolutely everything in RAM. However, certain things are kept only on disk, such as:
  • Audit files.
  • Certain documents (notably resumes).
• Workday’s whole database – data and metadata alike – is persisted to disk in <10 MySQL/InnoDB tables. MySQL is basically just being used as a key-value store, albeit one with ACID transactional support.
  • There are 3 main tables: attributes, relationships, instances.
  • When I suggested this might be like an entity-attribute-value model, Workday said it would be even better to think in terms of instanceID-attribute-value.
  • As you might expect for a database that simple, its schema doesn’t change much.
  • By way of comparison, Workday estimates that if its software were written relationally, there would be 1000s of tables, which would take up 10-100X as much disk space.
• All write transactions are banged immediately into the MySQL database. I.e., RAM and disk are never allowed to get out of sync.
• Workday’s database is append-only. This is exploited for effective dating (pretty heavily, it seems, perhaps because that’s a useful concept in human resources) and snapshotted reporting.
• Workday’s built-in BI doesn’t have a lot of choice but to do scans, traversing the object model. This turns out to be fast enough.

Other notes on Workday’s data and object management strategy include:

• Workday is object-oriented through and through – no object-relational mapping – turtles all the way down. On average, a class has about 2 attributes.
• 94% of requests are reads, traversing the object hierarchy.
• Workday databases are pretty small.
  • The biggest database Workday supports uses 17 gigabytes of RAM.
  • Workday databases are much smaller on disk than in RAM.
• Workday’s “dream” is to move from disk to solid-state memory.
• Workday uses GPLed MySQL/InnoDB. So there’s no software license reason to ever move away (e.g., to a pure key-value store).
• Disaster recovery is based on local and remote MySQL slaves.

Obviously, serious apps have been built before in object-oriented and/or key-value ways, with the resulting objects then being banged to disk (or in some cases kept in memory). Examples include:

• Numerous applications are built on object-oriented DBMS. Generally they go against disk, although memory-centric implementations can save a lot of pointer-chasing. Often they’re queried via SQL.
• Basho’s website says that its key-value store Riak was originally conceived in connection with a planned salesforce automation product, but I don’t think that the application part of that plan ever got built.
• SAP has longstanding doubts about relational dogma, although not nearly to Workday’s extreme.
• Obviously, some major internet applications just bang data into key-value stores.

Still, perhaps because it wholly object-oriented yet doesn’t even bother with anything like a real object-oriented DBMS, Workday’s approach seems particularly cool.

Other highlights of Workday, Inc.’s technical story include:

• Workday has settled into a schedule of three releases per year, and has pretty much lived up to that for >2 years.
  • Every user is always on the latest Workday release.
  • You can delay turning on significant new Workday software functionality if you want to.
  • Pure UI changes to the Workday software are handled much as they are on various websites today. Sometimes you have no choice but to live with them; sometimes the prior version of the UI remains available to you for a while.
• Workday’s navigational approaches look pretty cool.
  • The core concept is a list of actions you can perform now, rather than more standard menus.
  • Roles/permissions are of course central to this.
  • Reports have lots of actionable links in them. (More than just drilldown, although specific examples have slipped my memory.)
  • Alternatively, you can navigate via a search box, searching both on names of objects (e.g. users, divisions) or on names of tasks. This is somewhat reminiscent of an approach SAP was considering a few years ago.
• Workday says it has four key design premises:
  • Web-Familiar Experience. I’d say that’s true to to the extent it makes sense. In many ways, the web needs to catch up to Workday.
  • Enterprise Reporting. The idea is that you get a report, then take actions based on it. Hence the report-centric options for navigation.
  • Integration On-Demand. That’s a fancy way of saying “Plays nicely with others.”
  • Configurable Business Processes. Duh. That’s pretty essential if you want to do serious SaaS applications.
• Workday maintains a strong separation between application logic and UI development. Developer do no screen layouts. Instead, Uis are automatically generated for:
  • Flash/FLEX
  • iPhone
  • Mobile HTML
  • PDF export
  • Excel export
• Workday only talks to the outside world via web services.
  • Workday is heavily into SOAP (Simple Object Access Protocol).
  • The acquisition of OEM partner CapeClear gave Workday an Integration Service (i.e., enterprise service bus) that translates SOAP into whatever else might be needed for integration, and also does reliable delivery.
  • All that said, Stan Swete sees integration among various SaaS offerings as an area needing significant future attention.
• Workday’s business intelligence ideas are interesting, but I think there’s a long way for that technology still to go.
  • Workday’s BI seems to be focused on report/drilldown kinds of functionality.
    • You can slice by up to 2 dimensions at once.
    • Then you can keep slicing, however, by more dimensions, as many times as you like.
  • While you can take actions straight from reports, some of the specific BI/app integration ideas we discussed are still futures. (E.g., analyzing spend at the time of expense report data entry or approval.)
  • Of course, Workday’s web services interface lets you export Workday data into 3rd-party tools. Indeed, if you want to integrate data from Workday and some other source(s), that’s your only choice.
• Workday offers a clever metaphor to illustrate that your data may be more secure offsite than on – the bank vault. (I have no idea whether that’s a SaaS industry standard, but I hadn’t heard it before.) Of course, that metaphor does beg some issues specific to the remote data case, such as:
  • When your data is on premises, you know whether the government has insisted on looking at it.
  • More than cash, data keeps traveling back and forth to the remote location, which creates at least a theoretical risk of interception.
• Workday says the toughest part of globalization is the issue of which personal data is or is not maintained. For example, in the US you’re not allowed to not ask a job applicant’s religion, but in the UK you’re not only permitted but indeed required to.

This post is part of a three-post series

• Workday Inc. company overview (brief)
• Workday Inc. technology overview (detailed)
• Workday Inc. CTO Stan Swete’s comments on database strategy

Workday, Inc. was founded by Dave Duffield and Aneel Bhusri, who’d previously worked together at PeopleSoft. It is generally the case that the companies Dave starts: 

• Develop application software for large or fairly large enterprise customers.
• Build those applications in/on their own platform technology, which is meant to be cutting-edge in its day. (For example, PeopleSoft was early in building an RDBMS-based client/server application suite, and did so with the help of a clever technology called PeopleTools I nonetheless helped talk PeopleSoft out of further commercializing.)
• Focus first on human resources software (Dave had another HR company before PeopleSoft).
• Move fairly early into non-profit/higher-education accounting (Dave had a company in that area before PeopleSoft, and PeopleSoft was fairly active in the area too).
• Emphasize a pleasant corporate culture.

All these things seem true of Workday Inc., although the non-profit/higher-ed move is just underway now. Specifically:

• Workday was founded in 2005, starting with an asset buy of some platform software a key PeopleTools developer had been working on for years.
• Workday has had multitenant SaaS offerings from the getgo. (And that’s all Workday does.)
  
• Workday has around 150 customers.
• Workday sells mainly to multinational corporations, generally based in North America. Efforts in the UK are beginning to ramp up.
• Workday has six core application modules, among which are:
  • Workday Human Capital Management (almost all the customers).
  • Workday Payroll (a little under 50 customers). Workday is partnered with local providers for payroll in 20 countries, and is building its second inhouse version (Canadian) now.
  • Workday Financial Management (a little under 20 customers, for what is far from a complete system).
  • Workday Benefits Network, providing connectivity to benefits providers (that’s the only Workday module that isn’t straight software).
• Workday, Inc. has around 500 employees, mainly in Pleasanton, CA. About 20 are in Dublin, Ireland, courtesy of the acquisition of CapeClear. About 1/3 are in development.

Workday prices its services based on metrics for the overall client business, not per-Workday-user. (Actually, the metric is basically headcount, which makes sense given Workday’s application focus.)

Some of these points are covered in more detail in a Workday Inc. slide deck.

This post is part of a three-post series

• Workday Inc. company overview (brief)
• Workday Inc. technology overview (detailed)
• Workday Inc. CTO Stan Swete’s comments on database strategy

Edit: Also, there’s a Workday blog with only a few posts, which nonetheless seems to flesh out a few of the ideas in this post series.

• Akiban is basically in the business of making OLTP (OnLine Transaction Processing) DBMS.
• That said, Akiban does not necessarily aspire to offer the DBMS that has the best update efficiency or throughput. In particular, the Akiban DBMS stores every datum twice, even before replication (and indexes) are taken into account.
• Akiban wants to store everything as third normal form relational databases. I didn’t ask whether 3NF is a hard requirement, a Really Good Idea if you want Akiban to run fast (that’s the one I’d guess), or merely a general design assumption.
• Akiban characterizes its core differentiators/value proposition as being:
  • Scale-out
  • No need to pay the traditional cost of joins
• Thus, Akiban is telling something like a NoSQL story.
• However, Akiban offers SQL.
• Specifically, Akiban offers SQL through a MySQL front end. However, the choice of front-end could change (Drizzle?), and non-relational front-ends (object?)* could eventually also be offered.
• Akiban’s first target market is SaaS providers, specifically ones that have true multitenancy issues. More generally, Akiban is pursuing cloud/private cloud applications with lots of tables. (Ori talks of a few thousand tables as being a small number.) At least at first, Akiban is conceding the market for huge-volume, scale-out, no-expensive-join web databases to the NoSQL contenders.
• Akiban has been in prototyping/development of some kind for several years. However, Akiban got its first angel funding early last year and its first venture funding late in 2009, so development only ramped up recently.
• Ori tells a version of the rather common “Everything I need to know in life I learned in the Israeli Army, and now I’m commercializing it” story. However, I didn’t get the sense that Akiban is necessarily a direct extension of a specific Israeli military project.

* A lot of Boston-area DBMS developers have significant non-relational experience. E.g., Jack Orenstein was an Object Design founder, and Peter Beaman used to work for Intersystems, both object-oriented DBMS vendors.

Akiban technical highlights include:

• Somewhat confusingly, Akiban databases are divided into “groups” of tables. The point of Akiban groups is:
  • Many-to-many relationships exist only within Akiban groups, not among tables in different groups.
  • Tables within Akiban groups are kind of pre-joined; more precisely, data is organized physically in a way that anticipates joins.
  • Thus, most Akiban joins can be executed without the cost of traditional join algorithms.
• One copy of the data Akiban stores is, in effect, clustered by object. E.g., a customer and her orders are stored together, or a patient and the records of her doctor visits. That’s how Akiban anticipates most joins.
• The other copy of Akiban data is stored in columns (I’m not sure if this part is strictly columnar or more hybrid row/column), which are ordered consistently. In particular, they’re in an order dictated by the organization of the other copy of the data, whatever that means. Akiban’s goal is for this copy of the data to support reporting, operational BI, etc.
• Akiban relies heavily on its optimizer to determine data layout, probably more than conventional DBMS do.
• In essence, Akiban has a MySQL front-end and a storage engine back end, each running on its own hardware cluster, with each node of one cluster talking to each node of the other.
• I gather that Akiban distributes data among nodes clustered according to, in effect, object identifier. Presumably, inter-node joins are rare. But we didn’t discuss distribution, replication, or other scale-out issues in any detail. Indeed, I gathered that significant parts of all that weren’t built yet, and perhaps even not yet architected.
  

• Direct improvement in user interface or functionality. Examples (again overlapping) include:
  • True UI enhancements.
  • Application functionality that just lets you do more.
  • Most modern mobile, web, and/or social software efforts, in which a relatively small amount of coding effort produces features that may or may not lead to rapid viral adoption.
  • Ease or functionality not just for end users, but also for administrators. In particular, SaaS, cloud, private cloud and/or appliance benefits are commonly concentrated in this area.
  • Languages and other programmer aids too.
• Performance/efficiency improvement. Overlapping examples include:
  • Anything that directly purports to improve response time, hardware cost or utilization, or power/floor space consumption.
  • Anything to do with parallelization or scale-out.
  • Many, many under-the-covers enhancements to make data more protected (against theft or loss alike), user features snazzier, and so on. With a few exceptions – which are generally regarded as unsolved artificial intelligence problems – almost anything can be hacked together quickly in some high-level programming tool, assuming performance is no of no concern. It’s getting the performance remotely right that can often slow market introduction.*
• New or enhanced logical data model. Examples of innovation via data model – either truly new or else just newly implemented in a performant way — include:
  • A huge fraction of application innovation, in “traditional” functionality and workflow alike. In several technological eras, just about everything about applications has been a commodity except the data model, but the data model alone was enough to provide long-lasting product differentiation. Indeed, it probably is true today, although that may finally change as business intelligence integration becomes a large part of application software technology.
  • Most things that are called knowledge representation.
  • Many things that are described by terms like “unstructured” or “semi-structured” data.
  • Most innovations described by terms such as metadata management.

To check that I’m not being too glib here, let’s consider a few categories of software technology.

• MPP analytic DBMS are all about performance/efficiency improvement (whether of SQL queries or other analytics), except when they’re about ease of administration and the like.
• Hadoop is about scaling out cheap machines in a way that is (for some purposes) easy to program.
• The core of NoSQL is about efficient scale-out; easier programming also plays a big role.
• Disruptive small vendor business intelligence innovation has a lot to do with better and more useful user experiences, except when it’s about ease of programming and/or administration. The BI industry is also moving to in-memory analytics, which harnesses better performance to provide more interactive user experiences.
• SAS, which has long competed on the basis of superior functionality for statistical programmers, is now also on a big performance kick via MPP analytic DBMS partnerships.
• Oracle’s DBMS efforts have long been focused on performance and administrative usability.
• As noted above, enterprise application functionality is usually all about the data model. Exceptions arise when there is a major generation of UI functionality, such as interactivity (long ago), GUIs (ditto), or BI integration (in its early days now). SaaS is also pitched as an ease-of-everything play.
• Administrative tools are usually about making administration easier. In a few cases (e.g., backups), they’re more about performance.

I’d say my proposed trichotomy is holding up pretty well.

So what set me off on this line of reasoning? Well, Stephen O’Grady wrote

The reason I am against software patents is … very simple. … I am against software patents because it is not reasonable to expect that the current patent system, nor even one designed to improve or replace it, will ever be able to accurately determine what might be considered legitimately patentable from the overwhelming volume of innovations in software. Even the most trivial of software applications involves hundreds, potentially thousands of design decisions which might be considered by those aggressively seeking patents as potentially protectable inventions. If even the most basic elements of these are patentable, as they are currently, the patent system will be fundamentally unable to scale to meet that demand. As it is today.

In addition to questions of volume are issues of expertise; for some of the proposed inventions, there may only be a handful of people in the world qualified to actually make a judgment on whether a development is sufficiently innovative so as to justify a patent. None of those people, presumably, will be employed by the patent office. … Nor will two developers always come to the same conclusions as to the degree to which a given invention is unique.

In considering whether I agreed, I realized that the analysis is different for each of my three categories of innovation mentioned above.

• In the case of a logical data model, O’Grady is almost surely right. Many of those are just copied from the real world anyway, and hence don’t meet any kind of “novel and non-obvious” test. The rest are so general and abstract it’s really hard to say what – if anything – is new and non-obvious about them vs. well-established, often academic prior art.
• In the case of performance enhancements, the core ideas can usually also be found in well-established computer science publications. What’s more, the true innovations may be such simple algorithms that they’re not patentable. What’s left over is incremental enhancement. Once again, O’Grady is right.
• But the case of user interface/experience enhancements is not so clear. Inventor comes up with a useful idea for something that hasn’t been built before. Inventor builds and patents it. I’m not sure how that’s different from the case of building physical devices of various kinds, which have been patented for centuries. Determining what’s novel or non-obvious doesn’t seem to require specialized technical knowledge, at least not above and beyond that required in other disciplines.

Bottom line: There are many other reasons to oppose software patents, but Stephen O’Grady’s “It’s impossible to adjudicate them fairly” argument remains unproven, at least when it is applied to software enhancements whose essence is better designs for user experiences.

Related links:

• My negative comments about patents in the areas of MapReduce and columnar DBMS
• Three standpoints from which to view a software product strategy

• The most important issue in database and analytic technology, in my opinion, isn’t technological at all – rather, it’s the legal and political steps needed to preserve liberty in the face of advancing, intrusive technology.
• Another important issue for society – and this one does involve a lot of technology – is scientific number crunching. In particular, database technology for scientific computing needs to be developed much further. I’ll have more to say on all this soon.
• More generally, technology needs to keep advancing for parallel analytics. Fortunately, it is. Watch this space over the next few weeks.
• Oracle has said, in effect, that its most important technological challenge of the decade is getting solid-state memory right. I agree.
• Data volumes will keep going up, up, up. Technology needs to keep evolving accordingly. Much of what I write is on that subject.
• Data needs to be processed and analyzed at very different latencies. And there’s much further to go in integrating disparate latencies.
• Analytic database management in the cloud hasn’t been solved yet, especially for Big Data. Among the reasons are the difficulty of moving data into the cloud (unless it originated there), the slowness of moving it from node to node in shared-nothing architectures (which reduces the elasticity benefit), and above all the long and unpredictable latencies of interprocessor communication while queries are running (a key subject of discussion at the Boston Big Data Summit).
• Better business intelligence user interfaces are increasingly available. I’m thinking particularly of approaches with buzzwords like visualization/interactive exploration or faceted. But they aren’t well-integrated into the overall analytic stack, as big BI vendors are trailing the smaller ones in this regards. (Part of the problem relates to my previous point.)
• Application development over text search isn’t in the same league as application development over relational DBMS. The choices are mainly XML (e.g., MarkLogic), SQL for text integrated into RDBMS (limited by the weakness of those integrations), and something like Attivio’s Java SDK. There’s a major conceptual barrier in building those apps, namely the unpredictability of query results. Still, it should be possible to do better.
• Similarly, text analytics and conventional analytics exist well side by side. They can even be in the same database and/or dashboard, although in practice that is limited by the strong SaaS focus of text mining vendors and users. But analytic integration of them is really hard. Linguistic imprecision is, in my opinion, only the #2 reason for this difficulty. The #1 reason is that trends detected by text analytics are much less precise than trends on tabular data – e.g., a 50% increase in a certain kind of complaint may be no more significant than a 5% change in a revenue variable.
• I’m increasingly persuaded that graph analytics can be handled without a graph-centric data model. But right now, it isn’t being handled well at all. Lots more needs to be done – although when it is, it will just exacerbate the privacy/liberty dangers that so concern me.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Interesting trends in database and analytic technology

Robert Hodges of Continuent offers a great outline of Continuent’s clustering story, with a lot of “Now we got right what we previously didn’t know/admit we got wrong.” Continuent now claims to have a strong clustering offering, both paid and free/open-source, for both MySQL and PostgreSQL, with Oracle support perhaps coming really soon.

Merv Adrian, who has overrated the importance of TPC benchmarks in the past, seems to have become more skeptical.

Interim CEO Mark Burton laid out Infobright’s focus pretty clearly when he took over:

… the focus must be in building products that fit market segments where ease-of-use and easily attainable performance are valued.  This doesn’t sound like the high end of Data Warehousing to me where highly complex MPP architectures and teams of DBAs spend their time.  It sounds like the realm of Departmental IT and SMB where business leaders are in a hurry to gain access to data and answers without the lead time and pain of complex architectures and high costs.

I’m hearing about a SaaS focus from a lot of companies. The Continuent link above mentions one. So does RainStor’s latest blog post. Gooddata, a SaaS vendor itself, seems focused on analyzing data that was originally created via SaaS. I haven’t talked with Cast Iron or Pervasive for a while, but when I did, their ETL market targeting was all about SaaS. And of course, I hear dumber SaaS-focus ideas as well. I think the biggest substantive reason for this trend is — if you don’t have the broadest feature set, and fear large enterprises therefore won’t want your stuff, going after SMBs makes sense. And SMBs are presumed to be going SaaS. Also in the mix, of course, are a single platform to support, a small number of large SaaS vendors to sell to or partner with, and/or general trendiness.

Roman’s latest venture is business intelligence firm Gooddata. Gooddata was founded in 2007 and has been soliciting and getting attention for a while, so I was surprised to learn that Gooddata officially launched just a few weeks ago. Anyhow, some less technical highlights of the Gooddata story include:

• Gooddata believes it makes BI easy to adopt, unlike every other BI vendor on the planet — not excluding the many other BI vendors who say the same thing about themselves.
• Gooddata is entirely cloud-based, specifically in the Amazon cloud. I.e., Gooddata is selling SaaS-based BI.
• Gooddata wants to sell to enterprises that are large enough to have more than a couple of BI users, and small enough not to be well served by the BI market leaders.
  • In revenue terms, this is the ever-popular $100 million – $1 billion market.
  • Specifically, Gooddata believes that those enterprises may have decent “back office” BI, but don’t have much in the front office. Gooddata wants to provide them with front office BI, which seems to basically mean CRM analytics. Gooddata sees this as a market in which QlikTech is the major player. Generally, Gooddata wants to emulate and go after QlikTech.
  • Even more specifically, Gooddata wants to sell to Salesforce.com customers, who it believes are not well-served by what passes for built-in analytics at Salesforce. Partnering with NetSuite didn’t work as well, since NetSuite’s customers turn out to be smaller firms than are in Gooddata’s target market.
• Something I heard from both Jaspersoft and Gooddata is that there’s a hot market in providing cloud-based BI to online gaming companies. I gather these are mainly games running on mass communication platforms such as Facebook or the iPhone. Surely not coincidentally, it seems likely that:
  • These are small companies whose success – and hence data intake – can suddenly explode.
  • The data originates in cyberspace, with no particular need ever to come to the game companies’ own premises.
• Gooddata has 50 production customers.
• Gooddata had 2500 “projects” at the end of beta in June, and is adding 100 more per month. (Those numbers look weird together.) A “project” is a lot like a database, with associated reports, security privileges, etc.
• Gooddata has close to 40 people, mainly in development.
• I didn’t detect much of a sales strategy, nor much of a marketing strategy beyond the impressive early buzz generation. Perhaps that’s a partial explanation as to why the rate of Gooddata adoption fell even before the company officially launched.
• I forgot to ask what those 50 customers were actually paying, but considering Gooddata’s price list, it appears a typical price range for Gooddata’s stuff would be $500-$2,000/month.

Gooddata technical highlights include:

• Gooddata is building an entire BI stack – reporting, dashboards, ETL, in-memory database management, everything. I doubt Gooddata would claim that the pieces are best-of-breed in many ways other than BI ease of adoption and use.
• So far I’ve seen three Gooddata ease-of-use features or feature groups that strike me as differentiated – reusability (of metrics and/or reports), collaboration, and tag clouds. More on those below. Gooddata is also building toward an agility pitch, but those features aren’t all baked yet.
• Gooddata is MySQL-based today, but plans to move to a memory-centric compressed column store in 2010. Roman doesn’t reject analogies to SAP’s BI/BW/whatever Accelerator. Yes, folks – Gooddata is yet another BI vendor doing some form of memory-centric OLAP. That’s a big trend.
• I’m guessing that a big reason Gooddata is reinventing so many technical wheels is to ensure that the Gooddata stack is seamlessly multi-tenant from top to bottom. (Hasso Plattner of SAP’s comments on a similar idea suggest a similar emphasis.)
• Gooddata has its own multidimensional query language called MAQL (the A doesn’t seem to stand for anything). Today MAQL generates SQL for MySQL. The future columnar memory-centric data store will — I think – understand MAQL natively.

Now we get to the good stuff. When I wrote about reinventing business intelligence back in May, I focused on some interesting developments I see as actually underway — at least on an experimental basis and/or from small vendors – namely:

• Text-search interfaces. Well, while I didn’t see true text search in the Gooddata demo, I did see tag clouds, which have some of the same benefits.
• Collaboration tools. Well, Gooddata has a nice-looking approach to BI collaboration, heavily reflected in its UI metaphors. (That said, I haven’t really compared Gooddata to Microsoft SharePoint or SAP’s Portal/Rooms/whatever.)
• Memory-centric analytics (for speed of exploration). As noted above, Gooddata has that coming soon.
• Data exploration that tries to ignore fixed relational schemas, ala Attivio or Splunk. Roman says Gooddata is interested in or working on that, but offers no timetable.

Meanwhile, something I’ve been seeking for years, but haven’t seen much progress on since enhancement stopped on Cognos Metrics Manager, is more user-friendly metrics management. Well, it doesn’t have a lot of bells and whistles, but at least Gooddata has the basics – a list of already-defined metrics, and a reasonable way of compounding them into other metrics. I think that kind of thing will be a major BI feature going forward, to the point that a few years from now we’ll be worrying about how to port them from one BI vendor’s tool from another.

Bottom line: If you’re interested in BI, you should look at a Gooddata demo.

]]> http://www.dbms2.com/2009/12/27/introduction-to-gooddata/feed/ 12 http://www.dbms2.com/2009/11/23/boston-big-data-summit-keynote-outline/ http://www.dbms2.com/2009/11/23/boston-big-data-summit-keynote-outline/#comments Mon, 23 Nov 2009 06:25:50 +0000 Curt Monash http://www.dbms2.com/?p=1227 Last month, Bob Zurek asked me to give a talk on “Big Data”, where “big” is anything from a few terabytes on up, then moderate a panel on cloud computing. We agreed that I could talk just from notes, without slides. So, since I have them typed up, I’m posting them below.

The top two points from Q&A probably were:

• Big Data and the cloud actually have relatively little to do with each other, a few exceptions notwithstanding, especially if the data is in a shared-nothing DBMS (as opposed to, say, a MapReduce-oriented file cluster). Two principal reasons are:
  • Redistributing data from node to node is a little slow, undermining some of the elasticity benefits of the cloud.
  • Getting data into the cloud in the first place is a lot slow.
• The NoSQL movement is a lot like the Ron Paul campaign — it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don’t have a whole lot in common with each other.

Anyhow, here are my notes for the talk, edited in just a couple of places for readability or linkage.

Quick introduction

• Big Data vs. cloud
• How big is Big Data?
• At the low end of that range, there’s little you can’t do with conventional technology if you have:
  • An unlimited budget for hardware
  • An unlimited budget for software
  • An unlimited budget for people, especially Oracle DBAs

Big Data in OLTP

• Hard-core OLTP
  • Focus of DBMS technology for a long-time
  • Big budgets because each transaction has significant value
  • Tough to get users to change technologies
• Lighter-weight OLTP
  • Classic example = web companies
    • Big ones — retail-oriented ones (eBay, Amazon) partially excepted — rolled their own technology stacks
    • Reluctant to give money to anybody
      • Open source, etc.
  • Difficulty finding market
    • Product vs. feature
      • Clustering/HA/DR/whatever
      • Ditto cloud enablement
    • True products haven’t found much traction yet

Analytic Big Data use cases

• Kinds of data for analytics
  • More of same != big
  • More detail and/or new kinds
    • Complete data sets
    • Transactions
    • Call details
    • Tick/trade history
    • Web clickstreams
    • Network event logs
    • Other machine-generated data
    • CAM bottom line
      • Anything human-generated should and will be retained in its entirety
      • Quantities of machine-generated data retained should and will grow roughly in line w/ computing cost reductions (Moore’s Law, etc.)
• Analytic uses of Big Data
  • Analytics is mainly about three things
    • Problem detection
    • Customer relationship improvement
      • (Those overlap when the customer relationship is bad)
    • Financial statements on steroids
  • Main kinds of analytics
    • What BI vendors traditionally sell
      • General reporting and dashboards
      • Ad-hoc query (now driven from those reports and dashboards)
      • Planning (allegedly integrated with BI)
    • Research
      • Ad hoc relational query (worth mentioning twice because it drives so much of the market)
      • Data mining
      • Most web search and web mining
    • Operational/near-real-time
    • Archiving/compliance
  • What gets Big?
    • Mainly research and archiving
    • But when reporting or operational get Big, you have really interesting computing problems

Technology issues and trends

• Moore’s Law
  • CPUs — All about cores, hence parallelism is key
  • RAM
  • SSDs – hence replace disks
  • Sensors – hence generate lots more data
• Kryder’s Law
  • But rotational speeds up only 12.5X since Eisenhower Administration
  • Hence solid-state memory (or RAM) will soon take over
• In the mean time, I/O bottlenecks have had to be beaten
  • Hence sequential scans
  • Hence index-light architectures
  • Hence columnar
• DBMS “overhead”
  • Raw license and maintenance fees – software increasing fraction of total
  • OLTP vestiges – locking and all that
  • DBAs
    • People costs = huge fraction of total
    • Index-lightness addresses
    • So does appliance
  • Many people don’t really know how to write SQL
• Configuration
  • Appliance/tightly-balanced
    • Netezza
    • Teradata earlier
    • Greenplum/Sun
    • Oracle
    • IBM
    • Microsoft/Madison
  • Commodity/do what you want
    • Vertica
    • Greenplum now
    • Infobright, Aster and others
    • MapReduce-oriented file systems
  • Extreme rigidity is silly
    • Teradata, Oracle have both signaled moving to more modularity
    • Big driver of that = heterogeneous storage
      • Cheap disk
      • Expensive disk
      • Solid-state
      • RAM
    • CPU/storage ratio is even more of a driver

Theoretically defensible ways to segment the market

• Latency requirements
  • High availability and low latency go together
• Query types
  • Simultaneous users for same
• Database size
• Budget

Actual segments right now

• Utter ADW/EDW
• Data mart
  • Size
  • Naturally columnar vs. naturally row-based
• Operational/frontline
• Less dramatic/smaller EDW

Aster Data is officially announcing the 4.0 release of nCluster. There are two big pieces to this announcement:

• Aster is offering a slick vision for integrating big-database management and general analytic processing on the same MPP cluster, under the not-so-slick name “Data-Application Server.”
• Aster is also offering a sophisticated vision for workload management.

In addition, Aster has matured nCluster in various ways, for example cleaning up a performance problem with single-row updates.

Highlights of the Aster “Data-Application Server” story include:

• At its core, the Aster “Data-Application Server” is the Aster nCluster MPP analytic DBMS, enhanced with basic application server functionality (I didn’t ask for details of that part), running on the same nCluster worker nodes that answer SQL queries.
• Thus, Aster is eliminating a lot of the data movement that plagues three-tier architectures and other less-integrated approaches.
• The Aster “Data-Application Server” further offers integrated workload management for applications and queries; more on that below.
• The Aster “Data-Application Server” requires applications to be parallelized and invoked via Aster’s SQL/MapReduce.
• As befits a MapReduce-based system, the Aster “Data-Application Server” lets you write your applications in lots of different languages (the usual suspects, and it also does .NET).
• The Aster “Data-Application Server” runs applications in their own process spaces, protecting the DBMS server from crashes and other damaging behavior.
• The Aster “Data-Application Server” allows applications to manage memory themselves, persistently, and not just via relational constructs. Thus, if you want your application to maintain a graph, mini rules engine, and/or finite state machine, you can, without doing SQL contortions.

In a compelling proof point for the Aster Data-Application Server’s slickness, Aster has leapfrogged Teradata and Netezza in the extent to which SAS functionality is integrated into Aster’s DBMS. (Aster and SAS both say that you can do full SAS modeling in parallel on Aster, but even so I wouldn’t be surprised to discover there were some parts of SAS’ system that turned out to be exceptions.) Of course, Aster is hardly the only analytic DBMS vendor to have the idea of explicitly enhancing general analytic processing; that’s why we see lots of MapReduce announcements, and it’s also why Teradata enhanced its UDFs (User-Defined Functions) to have some kind of persistent memory.* But I don’t know of anybody else whose approach is quite so elegant and general at this time.

*Unfortunately, I don’t yet know much about Teradata’s UDF enhancements. I neglected to drill down on Global Persistent Memory when it was mentioned a couple of times at Teradata Partners last week, and Teradata was unable to accommodate my request this week for a rapid follow-up briefing on the subject.

Aster’s approach to workload management is similarly stylish. The idea is:

• Lots of variables are available to be taken into account (e.g., user role, expected query duration, actual duration of a running query, etc.)
• SQL statements can be written against any of these variables.
• The SQL statements serve as rules to set query/task priorities.
• There seem to be a few different ways to measure priority, including explicit allocation of CPU or I/O resources, as well as more conventional “This group of queries gets higher priority than that one” kinds of metrics.
• The whole thing provides integrated workload management for queries, applications, load jobs, data redistribution, and so on.

Right now the interface is – well, you’re manipulating a SQL table. A more conventional workload management GUI is slated for the second quarter of 2010.

Discussing subjects such as mirroring and ILM (Information Lifecycle Management) with Aster can be tricky, as Aster uses the word “partition” in confusing ways. Anyhow, Aster has a few different levels of compression, and the ability to apply different levels of compression to different partitions, to change compression levels via ALTER TABLE, and to alter (presumably increase) compression on the fly when doing online backup. Aster is also part of a growing trend to eschew RAID, instead doing mirroring in its own software. (Other examples of this strategy would be Vertica, Oracle Exadata/ASM, and Teradata Fallback.) Prior to nCluster 4.0, this caused a problem, in that the block sizes for mirroring were so large as to create a lag in transactional updating. But Aster says this problem is now solved, and indeed claims that nCluster 4.0 is superior to most rivals in transactional efficiency.

And finally, while I was talking w/ Aster Data anyway, I checked up on cloud and MapReduce customer penetration. The answers were:

• Aster has two serious production cloud users, both of which have been disclosed for a while, namely:
  • ShareThis, which runs Aster nCluster on Amazon EC2
  • Didit, which runs Aster nCluster on AppNexus
• Outside of those two, Aster sees some cloud use for test, development, prototyping, etc.
• Every single Aster customer uses SQL/MapReduce — i.e., they invoke MapReduce via Aster nCluster SQL queries.
• Some of those customers use MapReduce for ETL, some use it for actual analytics.

]]> http://www.dbms2.com/2009/10/30/aster-data-application-server-ncluster/feed/ 7