There are several reasons it’s hard to confirm great analytic user stories. First, there aren’t as many jaw-dropping use cases as one might think. For as I wrote about performance, new technology tends to make things better, but not radically so. After all, if its applications are …
… all that bloody important, then probably people have already been making do to get it done as best they can, even in an inferior way.
Further, some of the best stories are hard to confirm; even the famed beer/diapers story isn’t really true. Many application areas are hard to nail down due to confidentiality, especially but not only in such “adversarial” domains as anti-terrorism, anti-spam, or anti-fraud.
Even so, I have two questions in my inbox that boil down to “What are the coolest or most significant analytics stories out there?” So let’s round up some of what I know.
Financial trading is highly dependent on analytics, except perhaps for low-speed dealing in some traditional securities classes.* Specific algorithms are highly competitive, and can change within weeks. A broad range of analytic techniques have at least been tried. Some have failed so spectacularly as to precipitate a global financial crisis.
*Equities, corporate bonds, maybe municipal bonds — i.e., the stuff that’s still governed by “fundamental” — as opposed to generic quantitative — analysis.
It’s tough to be sure just what governments are doing with all that questionably-legal data they’ve assembled to fight terrorism, but we can be pretty sure that there’s a lot of relationship analytics involved. And whatever they’re doing seems to be working. As Glenn Greenwald frequently points out, there are substantially no terrorist plots in the United States, except for the ones that law enforcement itself invents. Anti-fraud applications have used relationship analytics for the past few years as well.
Also in the graph area, LinkedIn is doing a good job with People You May Know.
Also in crime-fighting, anti-spam techniques have gotten really good. Akismet blocks well over 99% of the spam comments to this blog, and I can’t recall the last false positive I saw. Google Mail approaches 99% in spam-catching (ironically, the ones that get through commonly come from mailing list sellers), and it’s been a long time since I noticed a false positive worth worrying about.
Shifting gears now, a huge fraction of analytic efforts are devoted to being more effective in one-on-one consumer marketing offers. Some of the most visible examples are in telecommunications, specifically in churn prevention. My favorite may be the case of multilingual text analytic integration in Switzerland:
I once confirmed that SPSS customer Cablecom‘s statistical models for churn and the like absolutely included text data; Cablecom even assigned different weights to the same apparent level of emotion depending on whether the text was in German, French, or Italian.
Other noteworthy adventures in marketing analytics include:
- Harrah’s successes in targeting gambling customers, even though I’m having trouble validating something I thought I’d read, namely that over 100% of Harrah’s profits came from analytics on loyalty card data.
- Various creepily effective efforts in de-anonymization.
- The recently famous story of Target targeting pregnant women.
So is it all about trading, marketing, and crime fighting? The answer is closer to being “Yes” than I would like. That said:
- Biomedical research is a hugely important area for analytic progress, and bioinformatics has gone from being a minor task to a full-time laboratory job. I hope we’ll have a bunch of great stories soon.
- Just as medicine needs to be personalized, so does education. Perhaps some low-hanging analytic fruit will be plucked soon.
- General optimization and operations research have been around for decades. They’re still going strong, with a lot more data points.
- Ditto process control. And the UIs are cooler now.
- Planning is a mess. Conceptual breakthroughs are sorely needed.