June 19, 2011

Investigative analytics and derived data: Enzee Universe 2011 talk

I’ll be speaking Monday, June 20 at IBM Netezza’s Enzee Universe conference. Thus, as is my custom:

The talk concept started out as “advanced analytics” (as opposed to fast query, a subject amply covered in the rest of any Netezza event), as a lunch break in what is otherwise a detailed “best practices” session. So I suggested we constrain the subject by focusing on a specific application area — customer acquisition and retention, something of importance to almost any enterprise, and which exploits most areas of analytic technology. Then I actually prepared the slides — and guess what? The mix of subjects will be skewed somewhat more toward generalities than I first intended, specifically in the areas of investigative analytics and derived data. And, as always when I speak, I’ll try to raise consciousness about the issues of liberty and privacy, our options as a society for addressing them, and the crucial role we play as an industry in helping policymakers deal with these technologically-intense subjects.

Slide 3 refers back to a post I made last December, saying there are six useful things you can do with analytic technology:

Slide 4 observes that investigative analytics:

Slide 5 gives my simplest overview of investigative analytics technology to date: 

Slide 6 points out that this is all supported by cheap data creation and acquisition, specifically in the area of machine-generated data, which gets the full benefit of Moore’s Law.

Slides 7-13 point out how the example problem domain involves lots of analytic tasks performed on lots of kinds of data. Specific examples cited include text analytics and graph/relationship analytics.

Slide 14 contains the punch line, so I’ll quote it in full:

Derived data

  • You can’t keep re-analyzing all that in raw form …
  • … so don’t.

If you have one takeaway from this session, let it be the utter importance of derived data.

Slide 16 lists kinds of derived data that are important in the single application of reducing telco churn:

And finally, Slide 17 is my first pass at best practices for dealing with derived data:

That last list looks like a starting point for a whole set of interesting future posts.


4 Responses to “Investigative analytics and derived data: Enzee Universe 2011 talk”

  1. AV on June 19th, 2011 10:42 pm

    Hi Curt:

    Should you also mention various query paradigms? For some use cases, the ideal solution would be to write a SQL query? When do I switch over to map reduce? Should I be using SAS-UDF type functionality (http://bit.ly/ceQ3Nk) embedded inside the database?

    Second, shed some light on the various components of the ‘derived data’ pipeline? For instance, why does EBay store raw data in HDFS (http://bit.ly/cJF4Jw) and then move derived data into a SQL data warehouse?

    Hope this helps.


  2. Curt Monash on June 19th, 2011 10:59 pm


    All good subjects. All beyond the scope of the slide deck, if not quite of the talk.

    Also, your first question is to some extent DBMS-specific.

  3. Enzee Universe 2011 « EMA Blog Community on June 20th, 2011 5:47 pm

    […] Investigative analytics and derived data: Enzee Universe 2011 talk (dbms2.com) […]

  4. Juggling analytic databases : DBMS 2 : DataBase Management System Services on March 16th, 2012 6:09 am

    […] Derived data based on other data already in the data warehouse. […]

Leave a Reply

Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:


Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.