Departmental analytics — general observations
Department-level adoption of analytic technology isn’t the exception; it’s the norm. Reasons include:
- Many analytic challenges are inherently departmental.
- In many cases, central IT control of analytics isn’t needed.
- Departments move ahead without central approval or involvement because they can.
That said, arguments for centralizing analytic technology include:
- A lot of data is used by more than one department, for example:
- Financial transactions (one or more affected departments and also the central accounting group).
- Web logs (marketing and IT/web operations).
- Departments may not have the requisite technical expertise (and it may be redundant/cost-ineffective for them to acquire it).
What’s more, there are IT best practices to support department-level analytics. Some of the key ones boil down to:
- Be flexible in your analytic DBMS support.
- Be responsive to requests for ETL.
My conclusion is that central IT should encourage (and aid) departmental analytics. Let’s look at some details.
I think two huge categories of analytic problem are inherently departmental:
- Investigative analytics (pretty much all of it).
- Routine monitoring/dashboarding if the data is tracked just by one department.
Investigative analytics is a kind of research activity — you’re looking to discover previously unrecognized patterns. There are two approaches to this — you can do it in the department that has the relevant business knowledge, or you can outsource it to a special group of “discoverers” (commonly statisticians).* Either way, this is a small team/departmental kind of activity.
*Combining the two approaches is common — a department can have its own analytically adept discoverers, whether they’re call “quants” or just “business analysts”.
Reporting/monitoring BI at least has the potential to be enterprise-wide — but commonly it isn’t, as each department has its own operational data sources and metrics. Marketing departments may watch external data that the rest of the company doesn’t worry about. But it can be true across the board. Factory operations folks may track machine tool data the rest of us barely understand.
Even if a business need is strictly departmental, there can be at least two reasons to centralize technology implementation:
- The department doesn’t have the critical mass of IT expertise.
- Departmental IT has side effects on the rest of the company.
Whether those reasons hold up depends a lot on what kind of analytic scenario we’re talking about.
Let’s organize that part of this discussion in line with the taxonomy from my eight kinds of analytic database posts last July.
- Enterprise data warehouses fall under the purview of major IT organizations. That remains true even if we pivot to the more realistic concept of integrated data warehouse. However, less stuff needs to be protected in an EDW/IDW than some data authoritarians like to think.
- I wrote that the stresses on traditional data marts were “performance, concurrency, TCO.” This is a clue that the more demanding examples are right in IT’s wheelhouse. As for the less demanding cases — IT should be able to meet those needs without breaking a sweat.
- Agile investigative data marts are inherently departmental. If you have the talent to use one, you also have the talent to, for example, train into being a part time Netezza DBA. Who cares if you don’t have the expertise to do sophisticated tuning? Analytic DBMS are fast enough — and hardware is cheap enough — that you don’t that skill set anyway.
- Big investigative data marts can go either way. They’re technically challenging, so IT certainly has a claim on them. But in cases where the data, while big, is fairly homogeneous, it’s also not unrealistic for departments to handle the mart themselves.
- Bit buckets are often departmental today, with the department in question happening to be central IT. And central IT is where they’re likely to flourish, as the data they hold becomes ever more diverse.
- Archival data stores are a central IT matter. Nobody else is likely to care enough to do it right.
- Outsourced data marts, by definition, don’t live inside conventional enterprises. But they are often a way for business units to get access to data and analytics without relying on central IT.
- Operational analytics servers are likely to be sufficiently mission-critical that you want them handled by IT.
So in most cases I’d say: Departments can manage their own investigative data marts, and so of course can SaaS vendors and third-party data providers; other analytic databases should be run by central IT. (And of course, large departments with serious local IT can fuzz those distinctions up.) Beyond that, it would seem that whoever administers the database should administer the rest of the analytic stack as well.
That still leaves us with some practical questions, such as:
- Exactly what products should IT departments buy for which purposes? I hope a lot of posts in this blog are helpful in that consideration.
- How should development tasks be split between departments and central IT? It may take me a while to get a post together on that the subject, since in general the analytics-development picture is pretty complicated to lay out.
- How should departments and central IT work together to manage departmental investigative data marts? I hope to post on that subject soon.
Comments
Leave a Reply