Departmental analytics — best practices
I believe IT departments should support and encourage departmental analytics efforts, where “support” and “encourage” are not synonyms for “control”, “dominate”, “overwhelm”, or even “tame”. A big part of that is:
Let, and indeed help, departments have the data they want, when they want it, served with blazing performance.
Three things that absolutely should NOT be obstacles to these ends are:
- Corporate DBMS standards.
- Corporate data governance processes.
- The difficulties of ETL.
Reasons they shouldn’t or don’t need to be obstacles include:
- Analytic DBMS are often vastly more cost-effective than general-purpose ones.
- In particular, analytic DBMS are often much easier to install and manage than general-purpose ones.
- Heavy data governance bureaucracy is often unnecessary because:
- The department should know what the limitations on the data’s accuracy are.
- The department should know how much data accuracy is required.
- The side-effects on other departments of any data inaccuracy would be minimal.
- There are multiple good schemes for populating data marts, managed by cost-effective analytic DBMS, with data from integrated data warehouses.
- ELT (Extract/Load/Transform) almost always works, because data cleaning/data quality was handled at or before the IDW level, and because the analytic DBMS has the processing power to pull it off.
- ETL (Extract/Transform/Load) should be easy as well. (If isn’t, something may be lacking in your ETL set-up.)
- Analytic DBMS are increasingly adding capabilities for easy spin-out of real or virtual data marts. Other kinds of technology (e.g. virtualization) are having their database spin-out capabilities upgraded as well.
One point to remember in support of departmental autonomy is that departments’ views of what data to use may be more expansive than central IT’s. One reason is that important data may be external to the company, outside IT’s natural realm of concern. Examples of this include but are hardly limited to:
- Anything like “market data”.
- Anything like “sentiment analysis”.
- Data owned by supply chain partners.
Further, even the more innovative internal data sources are commonly departmental, for example various kinds of multi-structured data (text verbatims from customers, log file data, and so on).
Whatever is true of data management (and ETL) is true for metadata management, even if it’s done by some kind of business intelligence tool. What I mean by that is:
- Whoever manages data is also responsible for ingesting and emitting it …
- … and specifically for emitting it in understandable, well-organized, well-named formats, …
- … so that departments can take responsibility for what amounts to lightweight analytic application development.
As for the “application development” itself, I’m envisioning at least three things:
- Math.
- Sophisticated relational query.
- Data visualization.
I.e., I’m talking about what “analysts” and “quants” do. So to put the point even more simply:
- Analysts and quants should be able to consume data that’s organized in a friendly manner.
- Central IT should be friendly in how it serves data.
One corollary of this approach is that departments should try to adhere to corporate BI standards, at least for routine dashboard and reporting. Indeed, if a department brings in a business intelligence tool different from the corporate standard, there are three main possibilities:
- The tool is integrated with something else it makes sense to bring in, such as a third-party data supply or application.
- The tool has an important capability the corporate standard doesn’t have, such as more flexible visualization and drilldown.
- Central IT screwed up, making things much more difficult than they needed to be.
Comments
4 Responses to “Departmental analytics — best practices”
Leave a Reply
Hi Curt-
Thanks for the post and I definitely agree with you about that the ETL layer should be easy to implement, deploy and manage.
In your comment about “something may be lacking in your ETL set-up”, I’m interested to know if you observe anything in particular on a regular basis on the difficulties of ETL?
–Keith Kohl
Keith,
I suspect that most alleged difficulties with getting data out of a central data warehouse and into a departmental data mart are due to bureaucratic foot-dragging, not technology.
[…] Analytics — a "pro-local" approach ? Curt Monash has posted about Departmental Analytics – best practices. He is of the view that “corporate standards” should NOT be an obstacle to […]
[…] more, there are IT best practices to support department-level analytics. Some of the key ones boil down […]