• In this release, Vertica UDTFs can only be written in C++. Other UDTF languages are promised.
• Otherwise, Vertica UDTFs sound pretty flexible; in particular:
  • They can ingest and emit any number of rows.
  • Their assumed schemas can be defined programmatically (both input and output).
• Vertica UDTFs go in the SELECT clause, not the FROM clause. I must confess to not grasping Vertica’s argument as to why this provides great and important flexibility.
• UDTF syntax mirrors SQL 99 Analytics pretty closely.

*It looks like the “F” is in the official name, but will often be dropped colloquially.

Other Vertica analytic platform highlights include:

• Proper integrated UDT workload management is promised, and there’s a little bit of UDT workload management already.*
• Vertica is delivering some prebuilt functions for aggregation, statistics, etc.
• Vertica has cool temporal and time series features.
• Vertica’s geospatial support seems pretty basic (circles and rectangles).
• Vertica’s NDA plans moving forward are pretty much as one would hope.

*Vertica’s UDT workload management is RAM-only, and “honor system” — i.e., it assumes that the UDTFs declare their resource usage correctly, which Vertica says is the right way to handle in-process C++ routines.

Vertica also argues that fast-performing SQL in and of itself can amount to analytic functionality. For example, Vertica has tried to ensure that it offers great performance in the kinds of self-joins that are used in graph analysis. Since Vertica has plenty of customers among the kinds of Web and telco companies that use graph analysis today, I’m inclined to grant some benefit of the doubt here. That said, Vertica thinks 3 hops is plenty for most kinds of graph analysis people want to do, and I can think of applications (e.g. anti-terrorism) where that’s surely not the case.

• I’m posting draft slides.
• I’m encouraging comment (especially in the short time window before I have to actually give the talk).
• I’m offering links below to more detail on various subjects covered in the talk.

The talk concept started out as “advanced analytics” (as opposed to fast query, a subject amply covered in the rest of any Netezza event), as a lunch break in what is otherwise a detailed “best practices” session. So I suggested we constrain the subject by focusing on a specific application area — customer acquisition and retention, something of importance to almost any enterprise, and which exploits most areas of analytic technology. Then I actually prepared the slides — and guess what? The mix of subjects will be skewed somewhat more toward generalities than I first intended, specifically in the areas of investigative analytics and derived data. And, as always when I speak, I’ll try to raise consciousness about the issues of liberty and privacy, our options as a society for addressing them, and the crucial role we play as an industry in helping policymakers deal with these technologically-intense subjects.

Slide 3 refers back to a post I made last December, saying there are six useful things you can do with analytic technology:

• Operational BI/Analytically-infused operational apps: You can make an immediate decision.
• Planning and budgeting: You can plan in support of future decisions.
• Investigative analytics (multiple disciplines): You can research, investigate, and analyze in support of future decisions.
• Business intelligence: You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
• More BI: You can communicate, to help other people and organizations do these same things.
• DBMS, ETL, and other “platform” technologies: You can provide support, in technology or data gathering, for one of the other functions.

Slide 4 observes that investigative analytics:

• Is the most rapidly advancing of the six areas …
• … because it most directly exploits performance & scalability.

Slide 5 gives my simplest overview of investigative analytics technology to date: 

• Fast query
  • Persistent storage (any data volume)
  • RAM (10s -100s of gigabytes, or more)
• Fast analytics
  • Predictive modeling
  • Transformation/tagging
  • Graph

Slide 6 points out that this is all supported by cheap data creation and acquisition, specifically in the area of machine-generated data, which gets the full benefit of Moore’s Law.

Slides 7-13 point out how the example problem domain involves lots of analytic tasks performed on lots of kinds of data. Specific examples cited include text analytics and graph/relationship analytics.

Slide 14 contains the punch line, so I’ll quote it in full:

Derived data

• You can’t keep re-analyzing all that in raw form …
• … so don’t.

If you have one takeaway from this session, let it be the utter importance of derived data.

Slide 16 lists kinds of derived data that are important in the single application of reducing telco churn:

• Normalized data
  • Parsed/sessionized logs
  • Text/sentiment highlights
  • Social network graph(s)
  • Web de-anonymization
  • Household matching
• Scores and buckets
  • Demographic
  • Psychographic
  • Offer hot buttons
  • (Dis)satisfaction
  • Credit/fraud risk
  • Lifetime customer value
  • Influence on others!

And finally, Slide 17 is my first pass at best practices for dealing with derived data:

• Evolving data warehouse schema
• Data marts
  • Physical or virtual
  • Inputs/outputs to “EDW”
• “Data science”
  • Research != production
• Multiple processing pipelines
  • Log parsing
  • Text
  • Predictive analytics
  • Generic ETL
  • Streaming “ETL”

That last list looks like a starting point for a whole set of interesting future posts.

• Netezza i-Class was first foreshadowed in February, 2010.
• Netezza i-Class customer testing started in October, 2010 or so. Netezza i-Class evidently has been shipped to 4-5 partners and a single-digit number of end-user organizations, spread across some usual-suspect industries (financial services, telecom, and so on).
• Netezza i-Class 1.0 general availability is still in the (near) future.

My advice to Netezza as to how it should describe TwinFin i-Class boils down to: 

1.  The Netezza platform has been enhanced in two major ways:

• There’s a good way to run all kinds of analytic processes. This is very flexible and powerful, but tightly integrated with the SQL engine even so.
• You are supplying some specific high-performing, highly parallel, big-data analytic process building blocks. More precisely, you have greatly extended the set of such building blocks; you had some cool building blocks (notably Spatial) even before this.

2.   There are four main ways to get at this:

• Extended SQL.
• Programming, in a bunch of languages and paradigms, integrated into the SQL.
• Partner code, with them doing the programming for you.

Some of the rah-rah words aside, that’s a pretty fair overview. Here’s more detail.

To refresh your memory: Netezza TwinFin i-Class functionality basics include, as best I can tell (and there’s some more detail at the links above):

• You can run processes in a usual-suspect set of languages on Netezza i-Class (even Fortran).
• One notable example is R; indeed, there’s an R client for talking to Netezza TwinFin.
• Netezza provides its own Hadoop implementation, which differs from standard Hadoop implementations most notably in that it manages data relationally via the usual Netezza DBMS, not in anything like HDFS.
• Anything written in any language except C/C++ (or of course SQL) — and in particular anything involving Hadoop — runs out-of-process versus the Netezza DBMS. C/C++ can run in-process, for maximum performance.*
• There’s an assortment of parallelized mathematical analytic packages built into Netezza i-Class. The matrix algebra ones are called nzMatrix. Most of the rest are part of a collection called nzAnalytics. Often these are implemented as stored procedures, as they may make multiple passes through the data.
• Netezza has thoughtfully ported thousands of analytic procedures for you to the Netezza platform (in essence, the basic R/CRAN and GNU libraries). These are not promised to be parallel on their own, but you’re welcome to invoke an instance on each node and parallelize that way.

I forgot to check, but I’m guessing any extension of workload management to cover non-DBMS processes won’t be in the first release of Netezza i-Class.

*However, Netezza says that if you can batch requests to return even just 500-1000 records at a time, the out-of-process performance penalty — which is based on wait time for transferring data between processes — becomes insignificant.

None of that is particularly new information. But after a visit to Netezza on Tuesday, I’ve finally gotten some kind of handle on how i-Class is architected. Highlights of the Netezza i-Class architecture story, as I understand them, include:

• It all starts with UDtFs — User-Defined (table) Functions, which are subject to the usual limitations.
• To overcome the standard limitations of UDtFs, Netezza built:
  • A set of UDtFs that, taken together, execute command-line programs.
  • For each language (Java, Python, R, etc., and I think also C/C++), a library that talks to the command-line executor. This library can talk to multiple instances of the executor, so it’s not limited to a single data stream. Similarly, it can persist past the life of a query.
• Similarly, Netezza built a C/C++ library that talks to the command-line executor and also talks MPI (Message Passing Interface).
  • This has not yet been exposed outside Netezza.
  • Rather, MPI is used by nzMatrix, so that nzMatrix can invert (for example) really, really big matrices.
• There are two* main ways to invoke all this.
  • SQL. Any analytic process can be invoked via a SQL UDtF. Netezza tends to use the term UDAP (User-Defined Analytic Process) interchangeably for the process itself and for the SQL UDtF that encapsulates it.
  • Netezza’s (interfaces to an) R client. More on that below.
• Netezza’s version of Hadoop is an important special case. The mappers and reducers you write in Hadoop are UDAPs.

I didn’t delve far enough into Netezza’s UDAP syntax to understand how it compares to, say, Aster’s SQL/MR.

*From a marketing standpoint, Netezza might prefer to count partner code separately as a third way, but I’m focusing on the technology here, which is used by partners and end-user organizations alike.

Other Netezza/Hadoop notes include:

• Netezza has the usual kind of Cloudera partnership.
• Since Netezza’s owner IBM has a Hadoop implementation, it seems obvious there will be some partnership action with that too. But at this point it’s not so far along.

The Netezza TwinFin i-Class R story goes something like this:

• Assume you’re using R on a client. (I’m not sure whether Netezza has an R client to give or recommend to you.)
• There are three Netezza packages that change how R works, by letting it use stuff on the Netezza box.
  • nzR translates between logical R memory structures and Netezza tables. In particular, nzR allows R to run, not just in-memory, but against the data on the Netezza box.
  • nzMatrix lets you do R matrix algebra against the data on the Netezza box.
  • nzAnalytics lets you invoke various algorithms that run on the Netezza box, against Netezza data.

A recently announced Netezza partnership with Revolution Analytics is meant to lead to Revolution replacing Netezza’s ports of R libraries with its own preferred distribution, and then supporting same.

Finally, there’s Netezza Spatial.

• Netezza claims multiple orders of magnitude of performance advantage for Netezza Spatial vs. geospatial alternatives, which is always a nice thing to be able to say.
• Generally, Netezza Spatial is now regarded as being part of i-Class.
• However, the product timing and adoption comments above don’t apply to Netezza Spatial.
• Netezza Spatial has a couple of dedicated salespeople, and seems to be well-liked by retailers.
• Netezza surely wishes everybody would forget about some of rewrites and controversy associated with Netezza Spatial.

Perhaps there are yet more pieces of the Netezza TwinFin i-Class story I’m overlooking, but I hope I now have most of the major aspects at least partway right.

First and foremost, the fourth annual New England Database Summit (nee “Day”) is next week, specifically Friday, January 28. As per my posts in previous years, I think well of the event, which has a friendly, gathering-of-the-clan flavor. Registration is free, but the organizers would prefer that you register online by the end of this week, if you would be so kind.

The two things potentially wrong with the New England Database Summit are parking and the rush hour drive home afterwards. I would listen with interest to any suggestions about dinner plans.

One thing I hope to figure out at the Summit or before is what the hell is going on on Vertica’s blog or, for that matter, at Vertica. The recent Mike Stonebraker post that spawned a lot of discussion and commentary has disappeared. Meanwhile, Vertica has had three consecutive heads of marketing leave the company since June, and I don’t know who to talk to there any more. 

Speaking of blog problems, we’ve had performance/reliability glitches here again. Melissa Bradshaw determined that the problem was an apparently activated WP Super Cache not actually caching anything. We should be OK now, so please let me know if there are further difficulties. One interesting step — it turns out that there’s a WordPress plug-in that does automatic EXPLAINs (if you’re the blog administrator).

Another interesting Mike Stonebraker post can be found (at least for now) over on the VoltDB blog. He continued his assault on the CAP Theorem, arguing that availability is an exaggerated concern when there are bug- or other human-error-driven kinds of outages, and also arguing that the concept of “partition tolerance” is misguided. Commenters pushed back, pointing out that in geographically distributed scenarios, the CAP Theorem sense of partitioning is quite a legitimate concern.

When I posted an expansive definition of machine-generated data a few weeks ago, Daniel Abadi shot back advocating a narrower one (see the comment thread, which includes a link to his thoughtful post). The disagreement boils down to conflicting intuitions as to whether the machine-data/true-human-data ratio will keep growing rapidly, in hybrid cases such as web logs or social gaming.

Dave McClure recently offered a survey of hot startup investing themes. High on his list were location-based services, which is a reminder to us all that geo-spatial data is becoming much more important. Ray Wang is savvy enough to understand the privacy dangers location-based services cause, but influential though Ray is, his view will probably remain in the minority. Machine-generated data and video each also make appearances on Dave’s list.

And wait! I have even more links for you!  Several are taken from Thomas Houston’s choices for The Best Tech Writing of 2010. He chose well. I recommend sampling his list further.

• In an article about new electronic exchanges, the New York Times shared some numbers — 56% of trading volume “high speed” in stocks, 1/3 or so when looking at domestic futures, .1 milliseconds to do a NASDAQ trade, 13 milliseconds for a trade that involves Chicago/NYC communication, 60 milliseconds for NYC/Frankfurt. Slashdot offers photos and other context.
• James Taylor caught up with once-hot KXEN, and evidently got the impression KXEN was focusing a lot of its efforts on the tedious, time-consuming data-preparation side of modeling.
• Richard Tibbetts is being pretty funny on his blog.
• (Slashdot) The Russian government seems to be getting into open source software in a big way. Well, PostgreSQL is already big in Russia (close to 1 million installations, I was once told), so this might conceivably add some energy to its development.
• In Drupal 7, Drupal now has “a built-in test environment, version upgrade manager, and a database abstraction layer for use with MariaDB, SQL Server, MongoDB, Oracle, MySQL, PostgreSQL, and SQLite.” That may explain how MongoDB can hope to further penetrate the Drupal market.
• The Boston Phoenix argues that government lacks the manpower, budget, and expertise to keep up with its responsibilities in preserving and exposing information. Fixing that problem sounds like a pretty worthy open source development effort to me.

Finally:

• Clay Shirky reminded us that modern machine learning is what replaced old-style AI.
• Nominally reviewing a book he obviously disdains, Garry Kasparov — in my opinion the most admirable world chess champion ever — surveyed computer chess in quick, nontechnical way. The whole thing is a bit wordy even so, so I’ll quote one part:

In 2005, the online chess-playing site Playchess.com hosted what it called a “freestyle” chess tournament in which anyone could compete in teams with other players or computers. … The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

The privacy discussion has gotten more active, and more complicated as well. A year ago, I still struggled to get people to pay attention to privacy concerns at all, at least in the United States, with my first public breakthrough coming at the end of January. But much has changed since then.

On the commercial side, Facebook modified its privacy policies, garnering great press attention and an intense user backlash, leading to a quick partial retreat. The Wall Street Journal then launched a long series of articles — 13 so far — recounting multiple kinds of privacy threats. Other media joined in, from Forbes to CNet. Various forms of US government rule-making to inhibit advertising-related tracking have been proposed as an apparent result.

In the US, the government had a lively year as well. The Transportation Security Administration (TSA) rolled out what have been dubbed “porn scanners,” and backed them up with “enhanced patdowns.” For somebody who is, for example, female, young, a sex abuse survivor, and/or a follower of certain religions, those can be highly unpleasant, if not traumatic. Meanwhile, the Wikileaks/Cablegate events have spawned a government reaction whose scope is only beginning to be seen. A couple of “highlights” so far are some very nasty laptop seizures, and the recent demand for information on over 600,000 Twitter accounts. (Christopher Soghoian provided a detailed, nuanced legal analysis of same.)

At this point, it’s fair to say there are at least six different kinds of legitimate privacy fear. The first five I have in mind are:

• Governmental force. Your web browsing history can be used against you in a court of law. Profiling — perhaps based on big data analytics — might make you a target for law enforcement or anti-terrorism investigation as well. And between financial transaction records, communication records, physical movement data, and more, the government’s ability to gather and process information about people is effectively unlimited.
• Private sector discrimination. There are many ways private companies could use detailed profiles — or just incriminating photos — to your detriment. They could fire you, not hire you, deny you insurance or credit, or simply not give you their best possible price.
• Identity theft. Phishing happens.
• Social pressure and stalking. What your friends, neighbors, and classmates know about you can become a serious problem, especially if they gang up and cyber-bully you.  That your violent ex-boyfriend can track you could be a bigger problem yet.
• Embarrassment and creep-out. Some people REALLY don’t like being viewed naked in busy public places. Some people are bothered by weirdly (in)appropriate advertisements. Maybe there’s no obvious tangible harm other than their uncomfortable feelings — but their discomfort is serious even so.

The sixth is:

• Throwing out the baby with the bathwater in a backlash. I think medical privacy rules already cost lives, in research and treatment alike. (Forbes offers multiple examples of life-saving research being stopped by HIPAA.) I think there’s an inevitable tradeoff between intrusive physical security measures, such as the TSA’s various forms of security theater, and spooky behind-the-scenes profiling and surveillance. (Unfortunately, we can hardly live in a society without some kinds of security measures — and even if we could, voters would never allow it.) Proposals that would hamstring the internet advertising industry currently seem unlikely to pass — but how much uproar would it take before that changed?

You probably knew most of that already. Even so, here are a number of examples and links.

Slashdot has more on the Jensen case, in which a man was convicted of murder in no small part because his computer revealed that he had searched for information about murder methods, including one that duplicated the actual method of his wife’s death. The money quote in an appeals court decision is in Section 37.1, which characterizes “computer evidence” as “probably the most incriminating other evidence.”

Obviously, that particular trial had the correct outcome. But would you want a court to hear about the research you did about minimizing your taxes? What about when you considered changing jobs, something that might be of interest both in employment and child custody litigation? How about your viewing of sexy images other than those of your spouse? And by the way — just what kinds of online viewing or writing should get you on a terrorist watch list?

It’s getting ever more practical to track our actions and movements. The privacy implications are potentially grave — would you want THAT kind of information used in court, or for decisions about your insurance? Accordingly, the Electric Freedom Foundation coined the word “traitorware” to describe and call attention to consumer (mainly) devices that keep or even transmit records of your movements and doings. And mind-bogglingly, Forbes says that Sprint “turned customers’ GPS information over to law enforcement 8 million times in a year.”

But it’s not just your own devices. The New York Times recently wrote of uses for smart video technology. Enforcing good hand-washing procedure on doctors sounds great. But how about enforcing busy-ness on cubicle workers? Watching movie previewers’ emotional reactions to specific scenes and characters seems kosher. But what if all movie-goers are watched? Or what if this is done at supermarkets or vending machines, perhaps even repeatedly over time? Could be creepy, huh? And by the way, while it could be a way to get you great discounts and service, it could also be a way to categorize you as unworthy of same.

Meanwhile, the Washington Post reminds us that battlefield video surveillance technology is getting ever better. Obviously, such technology could be used for domestic law enforcement as well. More immediately, the Washington Post tells us of things like automatic recognition of license plates being used by police departments that repurpose anti-terrorism grants to general law enforcement. (Analogies to the 1980′s habit of tying every grant proposal to the Strategic Defense Initiative “Star Wars” project are probably not misplaced.)

But the benefits foregone if we don’t use this technology are also scary. The biggest current examples are probably the ones I cited above — medical research, anti-terrorism, and the whole online advertising industry. But even more examples are coming down the pike, of clever electronic aids with privacy implications. Most notably — as the population ages, and nursing homes continue to be miserable (and costly) places, revolutionizing elder care becomes ever more desirable. Well, sensor technology will soon be able to watch over old folks, keeping them safe(r) in their independence — but only if it is highly detailed or intrusive.

Also:

• Slashdot/New York Times had an article on a Marin County ordinance forbidding smart electrical meters. Part of the reason is a San Francisco area backlash against the privacy implications of having your electricity usage recorded moment to moment.
• Slashdot also points us at an article about technology you use for self-control in various ways. In principle, however, the “self” part is an optional feature.
• Computing Now offered an overview of on-body sensing technology. Mainly it talks about gaming and medical applications, but it also suggests that people receptive to putting tracking technology on themselves in crowds in return for certain emergency-response benefits.

Finally, privacy-threatening observations of our web use, postings, and other internet communications are much too numerous to list. But some high/lowlights include:

• AddThis now claims to track 1 billion unique individuals online.
• In its transparency reports, Google publishes a count of how often various governments ask it for data. For the first half of 2010, leaders included the US (4287), Brazil (2435), India (1420), the UK (1343), and France (1017). Also over 100 requests each were Argentina, Australia, Chile, Germany, Italy, Singapore, Spain, and Taiwan.
• Verizon had over 90,000 such requests in 2007.
• An internet service provider who successfully fought off an FBI information request nonetheless was kept under a gag order for six years about the case.
• The Obama Administration wants ever more new legal powers to get electronic information without court order or notice to the investigation’s subjects.
• The UK government, or parts thereof, wants to track everything you do online or telephonically, except — supposedly — the actual contents of your calls and messages. Similarly Australia. There’s an EU directive along those lines as well, although the EU also has some strong data retention safeguards.
• Various governments — India, the United Arab Emirates, and many more — have recently insisted that communication offerings such as BlackBerry provide them with back doors to snoop, on pain of being banned countrywide.
• Informal methods of data acquisition are also used. For example, police departments use fake identities (with photos of hot girls, of course) to friend young folks on Facebook to look for photos of underage drinking.
• A CNN survey article showed how extreme public data revelation can get, using the example of Louis Gray.
• For its own marketing reasons, DuckDuckGo offers a simple overview of how your web searches could/can be used against you.
• And as an example of how even petty internet uses can cause problems, one of many things that got former HP CEO Mark Hurd into trouble was looking at racy online pictures of business associate Jodie Fisher.

• HP hired a software guy, Leo Apotheker, as CEO, and a software guy with a liking for high-end services, Ray Lane, as chairman. Now a Leo Apotheker conference call suggests HP will increase its emphasis on software, and maybe high-end services as well. No surprise. The article suggests, however,  that HP at this point has no clear strategy along these lines. That’s no surprise either.
  • And then there’s Sarah Lacy’s take, of which the interesting part reads “Separately, Andreessen has said that he thinks enterprise software is ripe for disruption and his firm is going to fund a new generation of Oracle-killers.”
  • I added more on Ray Lane’s tenure at Oracle over on Software Memories.
• Netezza had a falling out with its original supplier of geospatial technology, Intelligent Integration Systems (IISi), and a lawsuit ensued over alleged copying. Now ISSi has upped the stakes, essentially alleging that Netezza’s new geospatial software doesn’t work, and that hence the CIA (evidently a Netezza user) is killing the wrong people via drone strikes. Netezza has wisely selected from its short list of acceptable responses, including versions of:
  • “All our classified customers are happy, and if we told you anything more than that, that would kind of defeat the purpose of being classified, wouldn’t it?”
  • “Copy, schmopy. A polygon is a polygon, and has been since Euclid.”
  • “We don’t have no steenking bugs.”
• OCZ, whoever they are, are trying to offer solid-state drives with PCIe-like bandwidth, which makes sense in that most observers except Teradata think the SAS interface isn’t fast enough for solid-state.
• Speaking of Teradata, I’d been wondering somewhat as to why they just shut down Kickfire’s product line after acquiring its assets. Well, somebody who tested a Kickfire box told me that — great TPC-H results notwithstanding — it turned out not to be nearly as fast as one might think, on real-life data sets that didn’t fit entirely into RAM. Hard though such a thing may be to imagine, it turns out that Kickfire’s TPC-H results were yet less significant than I thought they were.
• I haven’t been looking at High Scalability nearly as much as I should, and that’s an understatement. It’s an outstanding blog.
• A couple of Google execs offered some predictions about the future of online advertising, which might be of interest to anybody selling analytic (or text analytic) technology to the online/digital media market.
• The BBC shows us what a single 133-character tweet plus its metadata look like in JSON. (All 1582 characters.)
• Huffington Post’s CEO made some comments about influencers which are additive to what I’ve been saying about influencers over on Strategic Messaging. (If you don’t read that — well, it’s my blog about marketing.)
  
• Speaking of my other blogs, I’m not bothering to put up a separate post like this over on Text Technologies, where thee posts I have put up recently tend to be (at least by my standards) relatively link-heavy anyway, but I have a couple more to share even so:
  • Paul Carr’s 7 rules for TechCrunch/AOL employees are really funny.
  • Some major search engine marketing experts are sounding defeatist about web spam.

And it’s not going to get any better. Next week = the oft-postponed elder care trip. Then I’m back for a short week. Then I’m off on my quarterly visit to the SF area. Soon thereafter I’ve have a lot to do in connection with Enzee Universe. And at that point another month will have gone by.

Anyhow:

• Back in January, Oracle finally briefed me on Exadata 2. I also requested and got permission to post what I regarded as pretty interesting slides, then never got around to doing so. Well, here they are. (Pay no attention to the word “Confidential”.)
• Two people I have a lot of respect for, Cindi Howson and Doug Henschen, seem bullish on SAP’s in-memory NewDB efforts. But for a variety of execution reasons, I’m skeptical that this will matter for anything except SAP’s analytics suite. I.e., I don’t think anybody much except SAP will write OLTP apps to it, and I don’t think that without OLTP apps being written to it it’s much more than Business Objects’ answer to QlikView.
• I just learned that Netezza’s previous geospatial technology didn’t get ported to TwinFin. However, Netezza obviously found a geospatial alternative.

I ‘m beginning to make a habit of asking vendors for a postable version of their slide decks. Sybase IQ is another example.

• Google is doing something called BigQuery that is “SQL-like” for big data analytics. I don’t know anything about it.
• I also don’t know anything about IBM BigSheets yet. It sounds something like Datameer, but that could be way off the mark.

The main new activity I know of has been in the open source SciDB project.  

• A company called Zetics has been started to commercialize SciDB. As of now, the entire staff seems to be CEO Marilyn Matz, techie Paul Brown, and part of Mike Stonebraker. Marilyn says Zetics has some venture capital, but even under NDA didn’t tell me who it was from. Zetics does not have its own web site.
• Marilyn tells me there are 20-25 contributors to SciDB, led by Paul Brown and Mike Stonebraker. Brown is full-time. Persistent Systems has been donating the efforts of a few of its employees. Some LSST folks have been doing SciDB work backed by grant money. Most or all of the rest seem to be purer volunteers. Some Russians have been particularly active.
• Release 0.5 of SciDB is expected in June. Release 1.0 is expected in September. This is a rewrite; prior demo code has been scrapped. Perhaps not coincidentally, it’s also a small slip from prior project plans.
• The array data model is an example of what’s being implemented first. (Duh — you can’t have a DBMS without a data model.) Support for uncertainty is an example of what’s been deferred until later.
• As has been clear since XLDB3 last August, one major target market for SciDB is genomic research.
• It’s obvious that the oil and gas industry, with all its geospatial data, should be interested in SciDB. But there’s not much activity in that regard; outreach is evidently needed. If you can think of somebody in that sector (or anywhere else) who should be alerted to SciDB, please ping them.
• Interest from web analytics users in SciDB seems to have receded a bit from the days when eBay almost funded the project.

In other scientific data management news,

• Microsoft put out a book called The Fourth Paradigm on scientific database management. The whole thing can be downloaded, very officially, as a giant PDF. I think it’s worth skimming. I don’t think it’s worth actually reading. (I did read it.)
• XLDB4 will be at Stanford October 5-7. Unlike prior XLDBs, it will have an open (i.e., no invitation required) part.

Finally, you surely are aware of the whole “Climategate” mess, in which major climate researchers’ email was hacked and many unkind conclusions were drawn. Well, one of the most technical parts of the disclosure was in a long series of Read Me files, in which an unfortunate programmer lamented about the difficulty of reconstructing published results from files at hand. These turned out to illustrate a classic problem that SciDB or alternatives are meant to solve:

• Raw data was impossible to use without various adjustments to regularize it (the word “regridding” comes up a lot, for example). Massaging was needed before analytics could be done on it.
• The raw data was thrown out or lost, and could not be reconstructed (why they couldn’t have asked the suppliers of the data to give it to them again was unclear in this case, since it wasn’t original experimental data).
• It was thus impossible to massage the data in any new or improved way.

• Teradata is productizing “private cloud,” under names including “Teradata Enterprise Analytics Cloud,” “Teradata Agile Analytics Cloud,” and “Teradata Elastic Mart Builder.” I.e., Teradata hopes to leapfrog Greenplum in its “Enterprise Data Cloud” strategy. This is only fair, in that Greenplum lifted the idea from Teradata and eBay in the first place. It also provides major support for what I think is an extremely sensible trend. Give or take issues of who announces and ships what a couple months before or after a competitor, my early thinking is that the main differences between Greenplum and Teradata in this regard will be:
  • Virtual as opposed to just physical data marts, based on robust workload management software. (Advantage: Teradata)
  • Pricing, deployment options. (Advantage: Greenplum)
  • Features that don’t directly relate to enterprise/private cloud. (Advantage: Either, often Teradata.)
• Teradata is generally strengthening its data movement technology, e.g. for making various appliances work in sync. I’m not too clear yet on the details of that. I think this is what Teradata’s phrase “ecosystem management” refers to.
• Teradata is (pre-)announcing – at least as a statement of direction — an appliance based on solid-state drives (SSDs). I’ve thought for a while that Teradata was a leader in thinking through the issues around solid-state memory in data warehousing, so it makes sense that they’re among the leaders in actually coming to market as well. I plan to say more after meeting with, e.g., Carson Schmidt.
• Teradata has achieved a 300%ish speed-up in geospatial processing. I gather this is largely a byproduct of the parallel analytics work Teradata did around strengthening its SAS integration. However, there don’t seem to be a lot of Teradata geospatial users yet.
• Teradata Express, Teradata’s free Windows-based crippleware, is being ported to Amazon EC2 and VMware as well. Presumably to avoid cannibalizing Teradata product sales, there are quite a few limitations on Teradata Express, including system capacity, database size, and “no production use.”
• Teradata continues to extend its optimizations to handle queries issued by business intelligence tools. Previously, the focus of what Teradata discussed in this regard was query rewrite. But soon automatic recommendation and creation of Aggregate Join Indexes – i.e.., materialized views – will be included as well.

• A data model based on multidimensional arrays, not sets of tuples
• A storage model based on versions and not update in place
• Built-in support for provenance (lineage), workflows, and uncertainty
• Scalability to 100s of petabytes and 1,000s of nodes with high degrees of tolerance to failures
• Support for “external” data objects so that data sets can be queried and manipulated without ever having to be loaded into the database
• Open source in order to foster a community of contributors and to insure that data is never “locked up” — a critical requirement for scientists

However:

• I think that’s a dream/wish list. A lot of good could be done without meeting each of those six requirements in full.
• I think at least some of the XLDB/SciDB leaders realize this.
• In my opinion, a highly useful subset of the dream/wish list is achievable in the reasonably-intermediate future, in either of two ways:
  • Through a Hadoop-centric open source effort, especially since HadoopDB opens up the possibility of letting DBMS creators offload MPP scaling challenges to somebody else.
  • From commercial MPP software-only (as opposed to appliance) DBMS vendors. I think they can develop the needed technology. I also think it could be in their business interest to make licensing arrangements of the sort that the scientific and research communities would need.
• Talking about “scientific” big data is unhelpfully vague. Let’s just focus on multi-dimensional measurement- or model-centric data, from disciplines such as seismology (under the Earth’s surface), climatology (over the surface), and astronomy (outer space). That would also include disciplines whose three-spatial-dimensions-plus-time data comes from inside a laboratory or other man-made environment, such as high-energy physics, fluid dynamics, and so on.
• One place in all that where there should be a commercial-company market is in oil/gas extraction. And by the way, the energy industry is increasing its uptake of data warehousing technology faster these days than any other sector I can think of, except perhaps for …
• … web companies that do log file analysis. Facebook’s log data has arrays-within-arrays reminiscent of the scientists’. eBay has been a major backer of XLDB/SciDB. It’s far from fully known yet just how much overlap there is between log-file-analyzers’ data management needs and those of big-data scientists. But there clearly are at least some commonalities.
• I don’t get the impression that scientists focused on modeling — e.g. climate-predictors — have been big participants in XLDB. That’s a pity for at least two reasons. First, modeling is at the heart of some of the most important global issues scientists address (e.g., climate change). Second, it might be an area of particularly rich overlap with commercial data management needs.

Now let’s step back and consider approximately what is meant by the requirements listed above.

• The requirements for an array structure are evidently pretty deep. You can glean some of the reasons from the scientific database use cases posted on the SciDB website. In particular:
  • Coordinate data naturally fits into arrays.
  • Coordinate data also naturally fits into geospatial ranges and the like.
  • The “grid” for the array can be imprecise — or calculated via transformation — for a whole lot of different reasons.
  • Different measurements may be available for different points in the array. (I think this may be the essence of the array-valued-arrays requirement.)
• Some reasons scientists want versioning and support for data provenance are pretty obvious — you never want to lose the record of what the instrument readings said, or ever were believed to say. But it goes further. Data is “cooked” — i.e., transformed/reduced — and stored in huge volumes. So you’d like to later on be able to go back to the raw data and re-cook it.
• The workflow requirement seems to stem in many cases from data movement needs, that in turn in some cases stem from political issues. I haven’t yet understood why workflow would actually need to be baked into a scientific DBMS.
• By the time the database management systems we’re talking about could conceivably be ready, the need will be at least in the 10s of petabytes. 100s of petabytes is a reasonable design goal.
• Not that I’ve run any numbers on the matter, but it seems plausible that query fault-tolerance will be needed, at least in some cases.
• In many sciences (astronomy seems to be an exception), the default choice is to keep data in files rather than a DBMS. For example, CERN has a 10 terabyte or so Oracle database holding just the metadata for a vastly larger collection of data files. Even if the pendulum swings toward greater use of DBMS, the ongoing need for external file access is pretty obvious.
• I suspect that the insistence on open source is part legitimate, part knee-jerk excessive.
  • “Free” is the best possible price, of course.
  • Beyond cash cost, scientists want data access to be free of licensing encumbrance. There are two main reasons. First, people might want to manage subsets or copies of data remotely from its central repository, for a variety of reasons. Not all of those reasons are easy to overcome, so any closed-source licensing would have to be very comprehensive (e.g., global or at least continent-wide “site” licensing).
  • Second, they want assurance that data will always be accessible, even if licenses expire. That seems a little overwrought. Yes, moving data from one multi-petabyte repository to another could be a bit slow. But it’s not an eventuality to panic about.
  • As for actual community development — scientists sure have a variety of exotic data management needs. But I’m not sure how much talent or resource there is among scientists to do true DBMS development (as opposed to, say, refining some UDFs). Yes, one XLDB attendee was both an astronomer and a PostgreSQL Major Contributor, but he seemed like an exception. On the other hand, it’s not entirely implausible that, in the right framework, some people with database talent could be recruited to donate some time to the general advancement of science.
• I don’t know much about management of uncertain data, and will duck that subject for now.

Related links

• Martin Kersten’s response
• Jacek Becla’s response
• Scientific data sharing