September 10, 2009

Thinking about analytic speed

For a variety of reasons, I don’t plan to post my complete Enzee Universe keynote slide deck soon, if ever. But perhaps one or more of its subjects are worth spinning out in their own blog posts.

I’m going to start with analytic speed or, equivalently, analytic latency. There is, obviously, a huge industry emphasis on speed. Indeed, there’s so much emphasis that confusion often ensues. My goal in this post is not really to resolve the confusion; that would be ambitious to the max. But I’m at least trying to call attention to it, so that we can all be more careful in our discussions going forward, and perhaps contribute to a framework for those discussions as well.

Key points include:

1. There are two important senses of “latency” in analytics. One is just query response time. The other is the length of the interval between when data is captured and when it is available for analytic purposes. They’re often conflated — and indeed I shall do so for the remainder of this post.

2. There are many different kinds of analytic speed, which to a large extent can be viewed separately. Major areas include:

There certainly are relationships among those; e.g., a really great analytic DBMS could help speed up any and all of the last three categories. But when assessing your needs, you can go quite far viewing each of those areas separately.

3. It is indeed important to carefully assess your need-for-speed. Acceptable levels of analytic latency vary widely, ranging from sub-millisecond to multi-month. For example, I’ve put together a list:

That’s a range of at least 9 orders of magnitude, which is a lot like the difference between the speed of a turtle and the speed of light.

Comments

4 Responses to “Thinking about analytic speed”

  1. Three ways Fedex is a metaphor for data integration | DBMS 2 : DataBase Management System Services on March 6th, 2011 8:46 am

    […] latency SLA (Service-Level Agreement) they really do require. Similarly, enterprises can have a very broad range of latency requirements for data integration — and the simplest way to meet your latency SLAs might be by delivering […]

  2. What to think about BEFORE you make a technology decision | DBMS 2 : DataBase Management System Services on June 26th, 2011 1:51 pm

    […] query load /concurrent user count. Similarly important are requirements for various kinds of latency, the big two being query response time and how long it takes for data to first be available for […]

  3. Agile predictive analytics — the “easy” parts : DBMS 2 : DataBase Management System Services on November 28th, 2011 2:40 pm

    […] back to analysis time. And any decent analytic technology stack can give sub-hour latency; while that may not suffice from all standpoints, it’s plenty fast enough for analysis-time […]

  4. Analytic application themes | DBMS 2 : DataBase Management System Services on April 25th, 2013 4:42 am

    […] continued drop in high-frequency trading latency strengthens my 2009 contrast between the speed of a turtle and the speed of light; we’re now over a 3 * 10^10 difference between the speed of trading and the speed of generic […]

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.