July 25, 2010

False-positive alerts, non-collaborative BI, inaccurate metrics, and what to do about them

I’ve been hinting at some points for quite a long time, without really spelling them out in written form. So let’s fix that. I believe:

I shall explain. 

False positives on alarm systems are a huge problem. Alarms were ignored (indeed, turned off) on the Deepwater Horizon due to too many being false. When I had a fire in my house, the smoke alarm wasn’t on because of its penchant for earsplitting false positives. And false positives held back the software category of intrusion detection systems until it was eventually subsumed by related kinds of security appliance.

Similarly, false positives are an under-appreciated problem that restrains the advance of business intelligence software. It’s no big inconvenience when, out of your 3 biggest problems this week, your dashboard calls attention to 17 of them. But suppose you want to make alerts more granular, take them up in quantity a couple of orders of magnitude, and have your smartphone pinged each time the alert condition is met. Whoa. Now those bogus warnings start to be more of a consideration.

Why would you want some kind of “push” alerting? Use case circumstances include but are not limited to anything that might herald:

Why would these alerts ideally be granular and high-volume? Well, you might have lots of machines, lots of trucks, lots of customers, lots of potential investments, and so on. I could list many more examples, but these probably suffice to at least start the discussion.

I’m agnostic about what the UI would be. Sounds going off on your mobile phone? Email on your mobile device? An app more like a chat window? Something more like a mobile dashboard? Whatever it is, false positives would really screw up your work day.

Classical enterprise dashboards suffer from a similarly fundamental problem — dashboard technology is optimized for a screwed-up enterprise analytic methodology. That is:

Yes, you can do drilldown or even data exploration, QlikView-style or otherwise. You can communicate around the BI tools, whether via general portal technology or something built-in. Vendors keep trying to make it easier for you to build your own reports, charts, and dashboard elements. But you can’t very easily define and track your own metrics, and I don’t see a lot of effort toward making that better.

Perhaps vendors don’t try to provide strong metrics management because doing so might lead to – gasp! — more than “one version of the truth.” If so, such thinking is regrettably misguided. Examples of formulas – i.e., metrics — that should and can not be cast in stone* include:

*At least as a general rule – we surely can think of some simple cases or exceptions where extreme cookie-cutterness is just fine.

It’s not that there shouldn’t be preferred or default metrics for these quantities. Rather, my point is that individuals should be allowed to:

Why? Because the person closest to the situation may be the one best able to assess it, and also because recognizing that fact is a really good idea from the standpoints of employee morale, engagement, and even development. (Or you might say – shiny buzzword! – employee “empowerment”.) In particular, when a boss and her subordinate disagree about the ideal way to calculate a metric, they should be able to track both versions side-by-side. When there’s a significant difference between the two, that could be a fine time for an out-of-the-norm decision.* If that ability were available, portal-style BI collaboration would almost automatically come into its own.

*What kind of decision that is of course depends on the specifics of the case. If only one of two competing metrics says you should bother giving extra care to a specific customer, the answer is easy – you give it. In other cases, an actual meeting or other conversation may be in order to decide what to do.

A metric is just a function. To offer BI users the flexibility I want, it should be very easy to use these functions as inputs into other, custom functions. I.e., I’m talking about something with the flexibility of a stripped down Excel, although the UI would be more like that for a rules engine. And if you think about it, that’s exactly the same functionality needed to personalize your alerts and weed out false positives.

One last point – what I’m calling for could greatly increase the BI performance burden, perhaps even by three orders of magnitude. Well, that’s what we have all this great new super price-performance analytic database technology for. And anyhow the throughput hit won’t come overnight. It will be interesting to see where the boundaries end up between DBMS and inside-the-BI-tool analytic processing. But overall, the analytic DBMS industry – and the hardware vendors backing them up – should be able to handle anything the BI vendors throw at them.

Related links

Comments

7 Responses to “False-positive alerts, non-collaborative BI, inaccurate metrics, and what to do about them”

  1. Justin Hyde on July 26th, 2010 3:52 am

    Good article. I’d suggest that the metrics must be dimensional and might be thought of as services, i.e. Can be subscribed to to create derived metrics. This metric engine / designer would require that the 3 bi tools: etl, query, report, are merged into a single tool as the usage of the metric might dictate that it needed to be precalculated during the equivalent of the etl run.

  2. Curt Monash on July 26th, 2010 4:30 am

    Justin,

    I’d say the server side model is probably the BI tool knows how to do everything itself, but can also delegate to DBMS and/or ETL when that makes sense. This could mean a whole new level of cooperation between BI and DBMS technology.

    Naturally, they’d be more prone to do this with DBMS vendors that:

    A. They own
    B. Have high market ahare

    all else being equal – not that all else would be equal …

  3. Wayne Eckerson on July 26th, 2010 10:05 am

    The best way to control false positives is to allow end-users to set the alert thresholds themselves. That’s a better adaptive system than any artificial intelligence-based system can deliver.

    In terms of metrics, you need standard metrics so everyone knows what’s going on (e.g. customer profitability) and you don’t have a tower of Babel. But around the metrics (and even within them), power users should be able to explore the business activity behind the metrics to see what new information and insights they can glean.

    So metrics are for casual users and running the business; explorative analytics is for power users to seek insights or clarification around the metrics. These are wholly compatible.

    One thing you don’t want to do is create multiple versions of metrics for enterprise consumption…. chaos.

  4. Curt Monash on July 26th, 2010 10:17 am

    Wayne,

    But it’s not necessarily as simple as a threshold. A compound expression/condition/test may be what’s needed to avert false positives.

    As for preventing people from ever discussing alternate forms of metrics with each other — that’s way too Big Brotherish for my tastes …

  5. John M. Wildenthal on July 26th, 2010 4:38 pm

    The plurality of definitions makes me think of Chorus’ support for multiple marts. Is this sort of functionality part of what Greenplum is targeting? Each group getting to define its metrics differently in its own virtual mart and still access the official definition as well?

  6. Curt Monash on July 26th, 2010 9:15 pm

    John,

    I’m thinking of something much more granular than that. Chorus can be restricted to the model Wayne was suggesting — let your specialists do a better job of considering alternate possibilities. I want line managers to be allowed to think for themselves as well.

  7. Confluence: Analysis (Business Intelligence) on October 24th, 2011 8:44 pm

    Monitoring…

    Goals P1 Collect status and alerts from multiple source systems (tickers, data quality, automation,etc.) in one place and know whether a part of the system is broken sooner. Institute a better way to communicate such issues,……

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.