Unchanged, however, is the more technical list of six things you can do with analytic technology, taken from a blog post late last year. Also unaltered are my definitions of investigative analytics and machine-generated data.

I write extensively on privacy. This technological overview of privacy threats doubles as a survey of advanced investigative analytics techniques now coming into practical use.

And finally, on a happier note — if you enjoyed the xkcd cartoon, here are two links to that one and a few more.

Statistics, data mining, machine learning, and/or predictive analytics. 

The more research-oriented aspects of business intelligence tools:

• Ad-hoc query.
• Drilldown.
• Most things done by BI-using “business analysts”
• Most things within BI called “data exploration.”

Analogous technologies as applied to non-tabular data types such as text or graph.

This will be be basis for my part of a webcast on March 10 at 11 am Pacific/2 pm Eastern time. The other main part of the webcast will be a demo by the webcast’s joint sponsors Aster Data and Tableau Software.

Some of Aster’s verbiage in describing and titling the webinar is so hyperbolic that I do not want to give the impression of endorsing it. But I am very hopeful that the webinar itself will be interesting and informative, and will point people at least somewhat in the direction of the benefits Aster is claiming.

Another part of the deal was that I’d give a talk this morning at 9:30 am. And when I give talks, I like to put up posts that cover whatever material I haven’t written up before, while also offering the talk’s listeners convenient links to materials I have already covered previously at length.

So anyway:

As I’ve been doing all year, I plan to start the talk with the subject of liberty and privacy. My most recent overview post on privacy and liberty has a bunch of links to what I and other people have said before.

This year’s Enzee Universe keynote speaker was Stephen Baker, author of Numerati. His talk, the only one all week I’ve attended in its entirety (I do intend for mine to be the second one, however ) reminded me that some of my ideas had been inspired by his book, specifically the part about sensors in our elderly relatives’ homes tracking every movement, for legitimate reasons of health care and physical safety.

One thing I’m reminded of when talking with users is that they tend to be a bit focused on their projects or areas, and almost never have the opportunity to consider the full range of possibilities open to them. So I’ve put in two slides to raise consciousness on the point.

• $ per user
  • $1000s or maybe $10,000s (perhaps a small team of analysts looking at Big Data)
  • $100s or maybe $1000s (perhaps conventional BI)
  • On the order of $.10 or $1 (stakeholder-facing analytics)
• Three benefits of better price/performance
  • Do the same thing, cheaper
  • Do the same thing, better
  • Do something different

Then I put together a list of “cool technologies in analytics” people might want to think about, including:

• Solid-state memory (there’s a whole section here about that; see the sidebar)
• Data mart spin-out
• Exploratory BI (e.g., QlikView)
• Advanced analytics (platforms)
  • DBMS-centric (more on that coming soon, but meanwhile February’s posts can be a placeholder)
  • MapReduce-centric (there’s a whole section here on that too)
• Advanced analytics (UIs and algorithms)
  • SQL tasting (I just coined that to talk about the useful idea of getting fast, partial results on long queries)
  • Stats/predictive (I’ve really got to build a blog category for that)
  • Graph
  • Matrix/optimization

And finally, I listed three “aggravating analytic challenges” in areas where I’m disappointed with the progress of and/or prospects for technology, including:

• KPI management
• Text/tabular integration
• Profile of Revealed Preferences aka “Social graph”

Posts on Netezza’s announcements around the time of Enzee Universe

• A long discussion of Netezza’s technology, focusing on the database parts
• A discussion of Netezza’s and IBM’s compression strategies
• Notes on how Netezza balances its silicon and uses its FPGAs
• A quickie on data warehouse loading latency
• How Netezza Migrator works
• Netezza’s strategy for RAM and Flash

After the first one, Netezza Marketing VP Tim Young took me aside for two pieces of constructive criticism. The surprising one* was that he felt I had been INSUFFICIENTLY critical of Netezza. Since then, every other conversation we’ve had about content creation has also featured ringing reassurances that Tim truly wants independent, non-pandering work.

*The unsurprising one was that I’d rushed. Well, duh. After months of telling me I had a 1 hour slot, Netezza cut me to ½ hour a few days beforehand. And my talk had been designed to be high-speed even in the longer time slot …

As a result, I accepted a subsequent gig from Netezza that I would barely consider from most other vendors. Namely, for this year’s Enzee Universe – June 21-23, aka Monday-Wednesday of this week, at the Westin Waterfront Hotel in Boston – I would do some contemporaneous blogging. The parameters we agreed on included: 

• I would just blog here on DBMS2, with Netezza allowed to reuse posts in their entirety on its site(s).
• I also would give a talk on the conference’s last day.
• I wouldn’t say much about conference sessions, because:
  • I’m not a session-attending kind of guy. (I wasn’t particularly good at sitting still in class in 8^th grade. I haven’t gotten much better since. And I have a huge aversion to other people’s uninterruptible PowerPoints.)
  • I think Netezza’s sessions are just as hype-filled as anybody else’s. (Much as I enjoyed traveling around the world with Netezza last year, it was painful hearing Jim Baum claim in city after city that Netezza boasts a 50X performance advantage vs. the competition.)
• Rather, I’d base things much more on individual conversations and meetings.
• Because I didn’t see how turnaround time could work otherwise, we’d have some of those meetings beforehand, and others early in the conference.

That last bit didn’t exactly wholly work out; for the second consecutive year Netezza pulled a surprise schedule switch a few days beforehand. But:

• I did have extensive, fascinating meetings at Netezza’s offices on Thursday, which were the fodder for multiple posts going up today.
• I have a nice meeting schedule set up for Tuesday.
• There should be plenty of opportunity for hallway and exhibit-floor conversation as the conference progresses.
• I even have my own private conference room, with a lovely name (the “Paine Room”).

So far as I know, the rest of the plan is still operative.

Posts already written as I draft this one include:

• A long discussion of Netezza’s technology, focusing on the database parts
• A discussion of Netezza’s and IBM’s compression strategies
• Notes on how Netezza balances its silicon and uses its FPGAs
• A quickie on data warehouse loading latency

I still need to write one focusing on Netezza’s advanced analytics strategy, and plan to edit in a link to it when it’s up.

Part of the time, I talked about things I’ve written about before. For example:

• Liberty and privacy. That’s a link to my most recent overview post on the liberty and privacy implications of modern analytic technology. The notes I spoke from were actually posted previously, after I spoke from them at the New England Database Summit at MIT in January. I’m gratified that, at both events, I got very positive feedback on liberty and privacy issues.
• Pick the right latency. That’s a link to a post (also based on a previous talk, in this case the one I traveled around the world giving for Netezza last September) in which I laid out the different levels of speed and latency an analytic application might require. I counted 9 orders of magnitude between the slowest and fastest, which is pretty much the difference between the speed of a turtle (at least a small, slow one) and the speed of light.
• On the more general point of operationalizing analytics, my best or at least most detailed writing to date may be in a 2004 whitepaper on analytic business processes which, sadly, is still fairly futuristic today.
• I offered a few ways to think about the different kinds of data that go into data warehouses.
  • Some of those were outlined in a post last January about three broad categories of data, distinguishing among human/tabular, human/non-tabular, and machine-generateddata.
  • That was a kind of sequel to a post last December about a three broad categories of data warehouse house growth drivers, namely more of the same vs. more detail vs. wholly new kinds of data.
  • I gave some examples of creating new data to analyze back in 2005 and 2006.
• Comments I made at various points were foreshadowed in a post on reinventing business intelligence.

I also raised a few points that I’m not finding good links for. I’ll try to cover those in future blog posts.

Related link

• Notes for my Boston Big Data Summit (no relation to Aster Data’s Big Data Summit series) talk in October, 2009

On a rough-draft basis, my talk concept is:

Implications of New Analytic Technology in four areas:

• Liberty & privacy
• Data acquisition & retention
• Data exploration
• Operationalized analytics

I haven’t done any work yet on the talk besides coming up with that snippet, and probably won’t until the week before I give it. Suggestions are welcome.

If anybody actually has a link to a clear discussion of legislative and regulatory data retention requirements, that would be cool. I know they’ve exploded, but I don’t  have the details.

• It is inevitable* that governments and other constituencies will obtain huge amounts of information, which can be used to drastically restrict everybody’s privacy and freedom.
• To protect against this grave threat, multiple layers of defense are needed, technical and legal/regulatory/social/political alike.
• One particular layer is getting insufficient attention, namely restrictions upon the use (as opposed to the acquisition or retention) of data.

*And indeed in many ways even desirable

I surprised people by leading with the liberty/privacy subject at my New England Database Summit keynote; considerable discussion ensued, largely supportive. I hope for a similar outcome when I keynote the Aster Big Data Summit in Washington, DC in May. And I expect to do even more to advance the liberty/privacy discussion as 2010 unfolds.

Fortunately, I’m not the only only thinking or talking about these liberty/privacy issues. A group of Very Big Companies (Google, eBay, Microsoft, AOL, AT&T, Intel, et al.) have backed something called Digital Due Process, to restore the Fourth Amendment’s teeth in the electronic age. (The Fourth Amendment to the US Constitution is the one that restricts and governs law enforcement activities in the areas of “search and seizure”.) That fits quite nicely with the privacy/liberty point I’m most emphasizing, to wit:

I think that regulating use actually suffices. Look again at the Bill of Rights. On a first reading, it seems that the Fourth and Fifth Amendments prevent the government from getting certain kinds of information. It can’t look inside our houses, and it can’t make us answer questions … wrong! Actually, the courts can compel you to testify on any subject they choose; if you refuse to answer, you can go to jail for contempt of court. But they can only compel you if you are given immunity, so that you can’t be convicted of the crimes the testimony reveals. I.e., the government can get the information it wants; it just can’t use that information to harm you. (Fourth Amendment rights are a little murkier; if the government finagles its way into your house for any reasons, there are still actively litigated questions as to what kinds of information it can use that it may find there.)

A week ago, Michael Arrington posted the social side of the same coin, saying It’s Time To Overlook Our Indiscretions, by which he seemed to mean: In a world where all kinds of embarrassing information will come out, we should not (always) penalize people for the existence of mildly embarrassing information about them. I am in vigorous agreement with that.

Progress is being made. But we’re just at the beginning of what will be a long, difficult, and hugely important process. The stakes, without exaggeration, are human freedom, on a national and indeed global scale.

Edit: Scott Yara of Greenplum says it simply (emphasis mine):

We need laws to keep sensitive stuff such as credit records and medical histories from getting into the wrong hands. But even with such laws, it’s inevitable that information will leak out. So we also need rules about how personal information that might come into the hands of employers, police, lenders, and others can be used.

Related links

• Declan McCullagh wrote up the Digital Due Process initiative
• India is fingerprinting and photographing its whole population
• Finland, notwithstanding that it is a large forest products producer, is demonstrating that an almost entirely paperless society is realistic.
• Microsoft Research is trying  to separate data-for-analysis from data-with-enough-detail-to-use-against-you

• The most important issue in database and analytic technology, in my opinion, isn’t technological at all – rather, it’s the legal and political steps needed to preserve liberty in the face of advancing, intrusive technology.
• Another important issue for society – and this one does involve a lot of technology – is scientific number crunching. In particular, database technology for scientific computing needs to be developed much further. I’ll have more to say on all this soon.
• More generally, technology needs to keep advancing for parallel analytics. Fortunately, it is. Watch this space over the next few weeks.
• Oracle has said, in effect, that its most important technological challenge of the decade is getting solid-state memory right. I agree.
• Data volumes will keep going up, up, up. Technology needs to keep evolving accordingly. Much of what I write is on that subject.
• Data needs to be processed and analyzed at very different latencies. And there’s much further to go in integrating disparate latencies.
• Analytic database management in the cloud hasn’t been solved yet, especially for Big Data. Among the reasons are the difficulty of moving data into the cloud (unless it originated there), the slowness of moving it from node to node in shared-nothing architectures (which reduces the elasticity benefit), and above all the long and unpredictable latencies of interprocessor communication while queries are running (a key subject of discussion at the Boston Big Data Summit).
• Better business intelligence user interfaces are increasingly available. I’m thinking particularly of approaches with buzzwords like visualization/interactive exploration or faceted. But they aren’t well-integrated into the overall analytic stack, as big BI vendors are trailing the smaller ones in this regards. (Part of the problem relates to my previous point.)
• Application development over text search isn’t in the same league as application development over relational DBMS. The choices are mainly XML (e.g., MarkLogic), SQL for text integrated into RDBMS (limited by the weakness of those integrations), and something like Attivio’s Java SDK. There’s a major conceptual barrier in building those apps, namely the unpredictability of query results. Still, it should be possible to do better.
• Similarly, text analytics and conventional analytics exist well side by side. They can even be in the same database and/or dashboard, although in practice that is limited by the strong SaaS focus of text mining vendors and users. But analytic integration of them is really hard. Linguistic imprecision is, in my opinion, only the #2 reason for this difficulty. The #1 reason is that trends detected by text analytics are much less precise than trends on tabular data – e.g., a 50% increase in a certain kind of complaint may be no more significant than a 5% change in a revenue variable.
• I’m increasingly persuaded that graph analytics can be handled without a graph-centric data model. But right now, it isn’t being handled well at all. Lots more needs to be done – although when it is, it will just exacerbate the privacy/liberty dangers that so concern me.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Interesting trends in database and analytic technology

One section covered recent/ongoing/near-future trends that I thought were particularly interesting, including:

Simpler database technology, by which I mean DBMS that are:

• Easier to administer than market-leading systems …
• … even if at the cost of being special-purpose
• E.g.,
  • MySQL and older mid-tier RDBMS such as Progress
  • Many analytic DBMS and appliances, most notably Netezza’s

For general purpose or OLTP uses, I’m not a big fan of MySQL (not enough progress in making it industrial-strength), PostgreSQL (no good company behind it – I’m a non-fan of EnterpriseDB), or Ingres (open source or not, it’s an antiquated system that hasn’t been invested in as much as Oracle, DB2 or SQL Server).

But I get the impression there are a lot of contenders among small startups, featuring very new architectures for OLTP or general-purpose database management. VoltDB comes to mind. NimbusDB is finally within range of getting funded. Dan Weinreb told me Friday he knows of a bunch of others as well. And that’s all before we even get into the NoSQL kind of alternative.

Flexible storage architectures. That’s starting out with an emphasis on hybrid columnar, as in the examples of Vertica and Greenplum. Oracle (to whom I’m under no NDA obligation) and other vendors (to whom I am) are going that way as well.

Multi-tier database architectures, by which I mean at least two things:

• The database tier/server tier split of Exadata
• Hybrid RAM/disk architectures, examples of which include
  • Vertica’s RAM-based write-optimized store
  • Sensage’s CEP-in-the-DBMS
  • This in-memory analytics stuff we keep hearing about from the BI vendors
  • Any true in-memory/disk hybrid, such as the regrettably sidelined solidDB
  • Smart thinking by numerous DBMS vendors about optimizing the use of RAM and/or Level 2 cache

Netezza is particularly interesting to watch in this regard because it:

• Had a pretty strict storage/other processing split in prior product generations and …
• … ditched that in its latest generation …
• … which however is focused on optimizing the use of RAM cache

Also noteworthy is Petascan, the stealth-mode –and therefore harder to watch right now – company I keep teasing about, which makes a strong case for carrying the database/storage tier split into the flash/solid-state memory technology generation. Calpont also has a server/storage tier split, but that’s of mainly theoretical interest unless and until Calpont actually ships an MPP version of InfiniDB.

Cheaper parts, which have of course been a huge trend for decades. Solid-state memory will soon conquer the world. Meanwhile, cheaper sensors drive that machine-generated data I keep talking about.

An ever-better understanding of scale-out technology, in several respects, including:

• Query, notably data movement for MPP DBMS
• Update, especially minimalistic DBMS approaches, be they sharded MySQL or more NoSQLish
• Number-crunching, especially via MapReduce and/or parallel analytic libraries integrated into DBMS

Cool trends I touched on more briefly include:

• More data being available for analysis. This was a core theme of my Enzee Universe keynote speeches; there are also some notes on it in my post based on my Boston Big Data Summit talk.
• More users being served by analytics. Ditto.
• Data exploration/visualization, ala QlikView, Spotfire, or Tableau, and also the faceted stuff.
• The democratization of data mining. But I’m not as sure of that one as of the others…

One area I flat-out forgot to mention is easy data mart spin-out.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Open issues in database and analytic technology