• As one might expect, QlikView 11 contains fairly leading-edge stuff, but also some “better late than never” features.
• The leading-edge stuff is concentrated in the general area of “collaboration”.
• Additionally, QlikTech is always pushing the QlikView user interface ahead in various ways.
• The “Well, it’s about time!” feature list starts with the ability to load QlikView via third-party ETL tools (Informatica now, others coming).
• QlikTech is generally good at putting up pretty pictures of its product. You can find some in the “What’s New in QlikView 11″ document via a general QlikView resource page.*
• Stephen Swoyer wrote a good article summarizing QlikView 11.

*One confusing aspect to that paper:  non-standard uses of the terms “analytic app” and “document”.

As QlikTech tells it, QlikView 11 adds two kinds of collaboration features:

• Integration with social media, which QlikTech calls “asynchronous integration.”
• Direct sharing of the QlikView UI, which QlikTech calls “synchronous integration.”

I’d add a third kind, because QlikView 11 also takes some baby steps toward what I regard as a key aspect of BI collaboration — the ability to define and track your own metrics. It’s way, way short of what I called for in metric flexibility in a post last year, but at least it’s a small start.

That direct sharing of user interfaces is a cool feature, which every business intelligence vendor should offer. In an era of distributed workforces, when people can’t be assumed able to huddle around the same desk, it has value even for use among close coworkers. But it also should prove useful in a variety of more naturally remote use cases, multiple examples of which can be found in each of the areas of:

• Support (internal or external).
• Faceoffs — I mean collaborations — between two or more enterprise departments. Examples might include: manufacturing and purchasing, manufacturing and sales, or accounting and anybody else.

As for social media being used for BI collaboration — that’s generally in the air. For example:

• salesforce.com is pushing enterprise social media use broadly, and will surely increase its emphasis on the social media/BI intersection now that Dave Kellogg is there.
• Spotfire has announced similar features in its latest release.
• The more cumbersome side of the feature set (portal-based collaboration, emailing of individual reports) has been available from multiple vendors for years.
• eBay open-sourced a more dataset-centric version of the idea, just as Oliver Ratzesberger left the firm.*

*Umm — does anybody have a link to the project, or at least a name for it?

BI has been a communication tool since the first green paper report was dumped on the first desk. And there’s been collaboration in doing analysis at least since it’s been possible to email .XLS file attachments. Still, BI is too often used as bludgeon rather than binocular. Hopefully, the current generation of technology will finally serve to change that.

The explanation was led by Oliver Ratzesberger, late of eBay*and progenitor of eBay’s Singularity project. In simplest terms, one event can spawn a lot of event attribute information, perhaps in the form of name-value pairs, which it then makes sense to store together in some way. The example Oliver dwelled on was that, on any given web page, there can be 100+ pieces of information to record, including:

• All 50 search results you were shown, and their positions in the search rankings.
• Every ad, image, or graphical element.
• An ID as to which test you were participating in (every page you see on eBay has some element being tested).

*Oliver is leaving eBay for a still-secret large company. I would conjecture that Michael McIntire is on the move too, either to replace Oliver or to go with him, but Oliver did a very good job of not commenting on the matter.

There are several reasons why one might wish to store this information in ways that grieve relational purists. First, reconstructing all this information via joins would be brutally expensive. What’s more, reconstructing all this information via joins could be impractical. Some comes from third party ad servers, which might not reproduce the same ads upon demand. Other is in the form of rankings, which can’t always be reliably reproduced from one query to the next. (That’s just one of several reasons text search and relational DBMS are an awkward fit.)

Also, there’s a strong dynamic schema flavor to these databases. The list of attributes for one web click might be very different in kind from the list for the next page. Forcing that kind of variability into a fixed relational schema, while theoretically possible, doesn’t necessarily make a lot of sense.

2. That said, the Hadoop part of EMC ‘s story is based on MapR, which so far as I can tell is actually a pretty good Hadoop implementation. More precisely, MapR makes strong claims about performance and so on, and Apache Hadoop folks don’t reply “MapR is full of &#$!” Rather, they say “We’re going to close the gap with MapR a lot faster than the MapR folks like to think — and by the way, guys, thanks for the butt-kick.” A lot more precision about MapR may be found in this M. C. Srivas SlideShare.

3. On its latest earnings call, Oracle clearly said it would introduce a Hadoop appliance, versus just hinting at a Hadoop appliance the prior quarter. The money quote was: 

Finally, big data or the searching of large amounts of data using Hadoop. After Hadoop finishes filtering the data, the place you want to put that data is an Oracle Database, and that’s what a lot of our customers are doing. And we are exploiting the trend, the big data technology and the big data trend, if you prefer, by building a Hadoop appliance that attaches to the Oracle Exadata database or any Oracle Database for that matter. But you don’t have to buy our Hadoop appliance if you can use whatever servers you want running Hadoop, and we provide the interface between Hadoop and the Oracle Database.

In other words, Oracle is saying “We’d like to sell you a Hadoop appliance, but you can run Hadoop in some other way and we’ll coexist with it just fine.” That makes sense; refusing to coexist with Hadoop is not exactly a realistic option.

4. Back in June, I expressed great skepticism about the idea of a Hadoop appliance. There was at least partial pushback in the comment thread from both Amr Awadallah and Eric Baldeschwieler. Oops.

Their reasoning seems to be centered around matters of installation, administration, and general packaging.

5. A month ago I noted aggressive near-term plans for Apache Hadoop evolution. As noted above, one reason this is needed is competition from folks like MapR. Also, I note that:

• Three years ago, Oliver Ratzesberger’s group at eBay complained that CPU utilization running Hadoop was at 18%.
• Now Oliver uses a figure of 10-15%., and attributes an even lower figure to — I’m guessing here — Yahoo. (Another possibility might be Facebook.)
• In between eBay became one of the biggest and most prominent users of Hadoop.

The moral of eBay’s Hadoop adventures, as I see it, is neither “Hadoop sucks!” nor “Hadoop doesn’t suck!”; rather, it’s that there’s a lot of scope for Hadoop to operate differently in the future than it does today.

Similarly, whatever throughput Yahoo does or doesn’t get, it clearly has adopted Hadoop at the expense of the columnar-in-Postgres system it previously was so proud of.

Also, there has been a claim going around that — notwithstanding NameNode’s status as a single point of Hadoop failure —  no Hadoop installation has ever lost data due to a NameNode failure. The folks at MapR beg to differ, and sent over some links that sure seem to say the opposite.

6. Since we’ve just established that Hadoop will change, rapidly and pretty fundamentally, what exactly is the benefit of an appliance that is “balanced” for Hadoop usage today?

Evidently Mike Olson of Cloudera is still telling the machine-generated data story, exactly as he should be. The Information Arbitrage/IA Ventures folks said something similar, focusing specifically on “sensor data” …

… and, even better, went on to say: 

Privacy is dead.
What do we consider to be the boundaries of privacy, especially with respect to items like medical data? In a data privacy-free world, should we be regulating data usage instead? How do we deal with asymmetric access to our personal data, e.g., how is it that insurance companies claim the right to our personal information?

Obviously, my answer to the second question is Yes!!!!

Also from Hadoop World — Dave Menninger, now an analyst, reports on some Hadoop metrics:

How big is “big data”? In his opening remarks, Mike shared some statistics from a survey of attendees. The average Hadoop cluster among respondents was 66 nodes and 114 terabytes of data. However there is quite a range. The largest in the survey responses was a cluster of 1,300 nodes and more than 2 petabytes of data. (Presenters from eBay blew this away, describing their production cluster of  8,500 nodes and 16 petabytes of storage.) Over 60 percent of respondents had 10 terabytes or less, and half were running 10 nodes or less.

That eBay comment was particularly interesting.

A while back, Doug Henschen noted that Netezza flagship reference Catalina Marketing is now at 2.5 petabytes. Most of that is in one 600 billion row table. Oddly, the article talks of the Netezza/SAS partnership accelerating model-building via in-database scoring (not modeling) technology. Doug also wrote of a lot of analytic DBMS replacements, including:

• Microsoft by ParAccel
• Oracle by Aster Data, IBM, Oracle Exadata, probably Netezza, and probably Hadoop
• Netezza by Greenplum
• IBM by Teradata

Carl Olofson pointed out on Twitter that DataScaler was an in-memory database technology just bought by Oracle. This inspired me to google on them, and I found a sparse DataScaler CEO blog. I link it because of an amusing juxtaposition — the second-to-last post says, in effect, “We make appliances and we recommend all these awesome technology design partners who helped us design the hardware,” while the very last post says “Designing our own hardware was a mistake.”

Fred Holahan is now VP of Marketing at VoltDB, which is a lesson to me about giving free consulting … Anyhow, Fred tells me that VoltDB has about a dozen users on their way to production, some of whom are headed to being VoltDB paying customers, some of whom are not.

• eBay has thrown out Greenplum. (Edit: As per the comments below, eBay wouldn’t endorse that wording itself.) eBay’s 6 ½ petabyte Greenplum database has turned into a >10 petabyte Teradata database, which will grow 2 1/2x further in size soon.
  • Specifically, Oliver told me there are 8 petabytes of spinning disk, with 80% compression. So that’s 40 petabytes before you multiply by a reducing factor to cover mirroring, temp space, and so on. My low end for that factor would be 25-28%; my high end would be 35-40%; either way, we’re talking about >10 petabytes of true user data.
  • The 8 petabytes of spinning disk are headed to 20 petabytes next year.
  • Oliver gave the impression that Greenplum got thrown out more for reliability reasons than performance. (While eBay saw a major performance difference between Teradata and Greenplum, Oliver previously indicated he was inclined to attribute this more to specific Sun Thumper hardware/storage choices than to software.)
• That database, called “Singularity,” has some interesting aspects – notably, a character field that’s a string of name-value pairs – on which you can do views and so on for virtual tables — in a table that otherwise has dozens of conventional relational columns.
  • The system ingests log data in the form of lots and lots of name-value pairs.
  • The most commonly found ones go into columns in the usual way.
  • The rest are strung together into, well, a character string.
  • Teradata has developed some features for eBay that make it easier to index, query, etc. on that character string of name-value pairs.
• eBay’s more EDW-like (Enterprise Data Warehouse) multi-petabyte Teradata database continues to grow, with the main system apparently up to 4 ½ petabytes from the previous 2 ½.
• I took the opportunity to ask what kinds of data marts (virtual or otherwise) were spun out in practice.
  • In Oliver’s ranking,
    • #1 was derived data based on other data already in the data warehouse.
    • #2 was other data within eBay that had never been put into the data warehouse in the first place.
    • #3 was data truly from outside data.
  • Todd Walter chimed in to point out that at other Teradata customers who perhaps didn’t have as fully fleshed out an EDW, #1 and #2 could be reversed.
• eBay sees Hadoop as an interesting tool for certain special purposes.
  • eBay likes Hadoop for certain tasks such as image analysis. (Edit: And analysis of search results.)
  • eBay doesn’t like Hadoop for anything that requires data movement, such as a join.
  • Similarly, eBay doesn’t like HBase.
• eBay is enamored of the idea to do “social networking around analytics.”
  • This is something that has been built but not rolled out yet.
  • It seems more focused on actual business intelligence than on the underlying data, unlike Greenplum Chorus, which seems more focused on the databases themselves.
  • Since it hasn’t been rolled out yet, we don’t know which (if any) of activity streams, forums, or whatever will actually get significant adoption.

• Google has published about a project called Dremel. Per Tasso Agyros, one of Dremel’s key concepts is nested data structures.
• Those arrays that the XLDB/SciDB folks keep talking about are meant to be nested data structures. Scientific data is of course log-oriented. eBay was very interested in that project too.
• Facebook’s log files have a big nested data structure flavor.

I don’t have a grasp yet on what exactly is happening here, but it’s something.

• If you are or want to be a serious MapReduce user – and you’re past the “play around over the weekend” stage — you probably should have either:
  • A serious non-DBMS MapReduce distribution.
  • MapReduce integrated into your analytic DBMS.
  • Both.
• The obvious choice for non-DBMS MapReduce is Hadoop.
• The obvious choice for a Hadoop distribution is Cloudera Enterprise.
• Cloudera Enterprise has three main aspects, in an inseparable bundle:
  • Distributions for a double-digit number of open source projects. It’s nice having all that in one package – unless, of course, you like playing with Tinkertoys.
  • Proprietary Cloudera code.
  • Cloudera support.
• Cloudera says its proprietary code is and in the future is planned to be concentrated – at least in large part — on integrating open source technology with closed source products. This has the virtue of being targeted directly at that segment of the market which has proven it’s actually willing to pay money for software.
• Cloudera Enterprise areas of focus, now and in the presumed future, include:
  • Core Hadoop engine, which Cloudera says is quite predictably and appropriately evolving more slowly than the tools around it.
  • Development, management and administrative tools, including:
    • Pig and Hive. Cloudera says >70% of Facebook Hadoop jobs are initiated through Hive, and the same is true of Yahoo and Pig.
    • Connectivity to commercial tools.
    • The product formerly known as “Cloudera Desktop.”
  • Workflow, which in this context refers to letting you create a Hadoop application as a sequence of small steps, rather than forcing you to kluge it into being one unwieldy thing. At the moment, this is much less widely adopted than Pig and Hive, but Cloudera has high hopes for it, because of its obvious benefits in modularity and manageability.
  • Quasi-DBMS technology. Besides Hive and Pig, this includes HBase. Cloudera says there has been considerable demand for HBase, and it is pleased that the project is now mature enough to ship. Cloudera stresses that it intends HBase not for OLTP, but as an adjunct to analytic processing. E.g., Cloudera suggests HBase would be a fine vehicle for replicating dimension tables across each node of a cluster.
  • Data connectivity, e.g. to MySQL or to sensor log files.
• Cloudera Enterprise pricing is well below DBMS prices – not by a full order of magnitude, if I’m right about everybody’s quantity discount policies, but even so by a lot. Details are NDA.

Cloudera sometimes sends confusing signals about its beliefs and strategies. For example, one can get different stories depending on whether one talks to:

• Somebody at Cloudera who comes primarily from the user and open source communities.
• Somebody at Cloudera who has actually worked at a software company before.

But I predict that Cloudera will now stick for a while with more or less the strategy outlined above.

Naturally, we also talked about Hadoop adoption. Highlights of that part – no doubt somewhat biased towards Cloudera’s own customer base — included:

• Notwithstanding eBay’s prior skepticism about MapReduce, it is quoted saying nice things in a Cloudera press release, and has apparently become quite a large Hadoop user, starting out with a search-quality use case.
• Typical Hadoop deployment sizes are 10 nodes or so when experimenting, 80-500+ in production.
• 10 terabytes/node – I’m pretty sure Cloudera meant of user data — is not inconceivable, so a cost-conscious 500-node user could have 5 petabytes of data managed by Hadoop.
• Cloudera has half a dozen customers at the 75+ node production level.
• Web and financial services are the two vertical markets moving most aggressively into Hadoop production. The government is also in significant Hadoop production, but the details of that are classified.
• Web uses for Hadoop include:
  • Clickstream – sessionization, etc. – that’s a super-mainstream use.
  • Search – analyzing search attempts in conjunction with structured data.
  • Machine learning (for ad serving, etc.).
• Financial services uses for Hadoop include:
  • Internal trading rule enforcement/fraud detection.
  • Complex ETL.
  • Portfolio risk assessment (typically overnight).

None of this is inconsistent with previous surveys of Hadoop use cases.

Various users talked at the Hadoop Summit this week. I wasn’t there, and won’t write about their stories for now. That said, Twitter’s slide deck from same has some interesting stuff, including:

• 7 TB/day ETLed from MySQL.
• Petabytes-being-stored accordingly coming soon.
• Open sourcing their ETL tool Crane.
• 3-4X LZO compression at little CPU cost.
• HBase is a more usable for them than HDFS, which isn’t mutable enough.
• Pig = 5% of code and coding effort vs. vanilla Hadoop at 30% or less performance hit.

The main new activity I know of has been in the open source SciDB project.  

• A company called Zetics has been started to commercialize SciDB. As of now, the entire staff seems to be CEO Marilyn Matz, techie Paul Brown, and part of Mike Stonebraker. Marilyn says Zetics has some venture capital, but even under NDA didn’t tell me who it was from. Zetics does not have its own web site.
• Marilyn tells me there are 20-25 contributors to SciDB, led by Paul Brown and Mike Stonebraker. Brown is full-time. Persistent Systems has been donating the efforts of a few of its employees. Some LSST folks have been doing SciDB work backed by grant money. Most or all of the rest seem to be purer volunteers. Some Russians have been particularly active.
• Release 0.5 of SciDB is expected in June. Release 1.0 is expected in September. This is a rewrite; prior demo code has been scrapped. Perhaps not coincidentally, it’s also a small slip from prior project plans.
• The array data model is an example of what’s being implemented first. (Duh — you can’t have a DBMS without a data model.) Support for uncertainty is an example of what’s been deferred until later.
• As has been clear since XLDB3 last August, one major target market for SciDB is genomic research.
• It’s obvious that the oil and gas industry, with all its geospatial data, should be interested in SciDB. But there’s not much activity in that regard; outreach is evidently needed. If you can think of somebody in that sector (or anywhere else) who should be alerted to SciDB, please ping them.
• Interest from web analytics users in SciDB seems to have receded a bit from the days when eBay almost funded the project.

In other scientific data management news,

• Microsoft put out a book called The Fourth Paradigm on scientific database management. The whole thing can be downloaded, very officially, as a giant PDF. I think it’s worth skimming. I don’t think it’s worth actually reading. (I did read it.)
• XLDB4 will be at Stanford October 5-7. Unlike prior XLDBs, it will have an open (i.e., no invitation required) part.

Finally, you surely are aware of the whole “Climategate” mess, in which major climate researchers’ email was hacked and many unkind conclusions were drawn. Well, one of the most technical parts of the disclosure was in a long series of Read Me files, in which an unfortunate programmer lamented about the difficulty of reconstructing published results from files at hand. These turned out to illustrate a classic problem that SciDB or alternatives are meant to solve:

• Raw data was impossible to use without various adjustments to regularize it (the word “regridding” comes up a lot, for example). Massaging was needed before analytics could be done on it.
• The raw data was thrown out or lost, and could not be reconstructed (why they couldn’t have asked the suppliers of the data to give it to them again was unclear in this case, since it wasn’t original experimental data).
• It was thus impossible to massage the data in any new or improved way.

• What Teradata calls the Teradata Agile Analytics Cloud, which is a combination of previously existing technology plus one new portlet called the Teradata Elastic Mart(s) Builder. (Teradata’s Elastic Mart(s) Builder Viewpoint portlet is available for download from Teradata’s Developer Exchange.)
• Teradata Data Mover 2.0, coming “Soon”, which will ease copying (ETL without any significant “T”) from one Teradata system to another.
• Teradata Express DBMS crippleware (1 terabyte only, no production use), now available on Amazon EC2 and VMware. (I don’t see where this has much connection to the rest of Teradata’s cloud strategy, except insofar as it serves to fill out a slide.)
• Unannounced (and so far as I can tell largely undesigned) future products.

Teradata openly admits that its direction is heavily influenced by Oliver Ratzesberger at eBay. Like Teradata, Oliver and eBay favor virtual data marts over physical ones. That is, Oliver and eBay believe that the ideal scenario is that every piece of data is only stored once, in an integrated Teradata warehouse. But eBay believes and Teradata increasingly agrees that users need a great deal of control over their use of this data, including the ability to import additional data into private sandboxes, and join it to the warehouse data already there.

The Teradata Elastic Mart(s) Builder Viewpoint portlet automates the inclusion of outside data. If you’re already an authorized Teradata data warehouse user, you can fill in a very short form (three or so fields) and add authorization to import outside data, e.g. from a .CSV file. No fuss, little bother. Trivial as that sounds, when you combine it with Teradata’s pre-existing robust workload management tools, it creates a pretty good virtual data mart story.

Spinning out and maintaining consistency with physical data marts is a different matter. Teradata doesn’t seem too sure it believes in those. And while Teradata is obviously planning to increase its capability in that regard anyway, I didn’t get a lot of detail beyond the reference to Data Mover 2.0.

Related links

• My Greenplum-inspired post on the future of data marts, outlining issues in “private cloud” data warehousing.
• eBay’s “Analytics as a Service” pitch (about 1 ½ years old)
• A post by Teradata’s Dan Graham explaining the Teradata Agile Analytics Cloud and Elastic Mart(s) Builder Viewpoint portlet
• Home page and complete screen shot for the Teradata Elastic Mart(s) Builder Viewpoint portlet

XLDB more or less comprises:

• A series of what have now been three workshops: XLDB1 in 2007, XLDB2 in 2008, and XLDB3 in 2009 (the closest thing to a master link is probably the XLDB3 site’s related link page). Participants have included, among others:
  • A lot of big-name database-oriented computer science researchers — Mike Stonebraker, Dave DeWitt, Martin Kersten, and numerous others
  • Academics responsible for scientific database management, especially but not only in the astronomy area
  • Some vendors (although vendor participation was cut back after XLDB1) — at XLDB3, which is the one I went to, the three vendor folks who actually talked were Stephen Brobst of Teradata, Luke Lonergan of Greenplum (who worked in scientific high performance computing earlier in his career), and Jeff Hammerbacher of Cloudera.
  • eBay and to some extent other large web companies
  • A European Union funding bureaucrat
  • Me
• An attempt to kick start a broader movement, perhaps comprising (it’s not totally clear yet):
  • Computer science researchers interested in database issues
  • Database technology vendors
  • Scientific researchers (academic) who have very large or otherwise difficult database management problems
  • Scientific researchers (commercial) who have very large or otherwise difficult database management problems
  • Other commercial users who have very large database management problems

The first result or spin-out from the XLDB effort seems to have been the SciDB project. This is an effort to build an open source DBMS called SciDB that will address some of the needs the XLDB effort is uncovering. (More on that in other posts.) Somewhat confusingly, all the use cases the XLDB group is collecting are currently being posted on SciDB’s website, apparently because it’s glitzier and healthier than, say, the excessively sparse XLDB wiki. Some SciDB development has happened, but no large sugar daddy has yet been found. (It’s a fairly open secret that eBay looked seriously and favorably at funding SciDB before the economic downturn.) hit.

Numerous big-name computer scientists are associated with SciDB, indeed more closely (it would seem) than with XLDB. That said, I’m guessing Dave DeWitt’s involvement in the open-source SciDB isn’t what it would be if he hadn’t gone to Microsoft. DeWitt actually skipped XLDB3, although he was in town for VLDB. (XLDB3 was back-to-back with VLDB 2009 in Lyon, France in late August.) Stonebraker just didn’t make the flight for either conference, due to the double-knee “upgrade” he had back in March.

There’s a lot more to be said about the cross-discipline or science-specific requirements that researchers place on data management, but I’ll leave that for later and just get this posted as a start — assuming, of course, that blog outages permit.

Related links

• Paper laying out the SciDB project
• One version of a SciDB overview page, with links to academic papers