>>It’s a truism that each generation of dashboard-like technology fails because it’s too inflexible. Users are shown the information that will provide them with the most insight. They appreciate it at first. But eventually it’s old hat, and when they want to do something new, the baked-in data model doesn’t support it.

Baked-in, static data models are precisely the fundamental problem we see when initially engaging with our customers. In many cases the problem is exacerbated by an underlying warehouse that cannot easily be changed to accommodate new business requirements. Warehouses built using traditional methods become too brittle over time and companies are unwilling to risk (or unable to justify the expense of) a major overhaul to their warehouse. Kalido customers have found a top-down model-driven approach with an automatically generated data model allows them to quickly respond to new requirements. Our partner QlikTech provides a great solution to the visualization problem because there is no pre-defined model, and combined with a flexible data warehouse as a source, its ability to dynamically deliver more comprehensive information to meet user demands is further enhanced.

Regarding Google wave there is a lot of buzz lately but as always Google delivers a very basic application everybody talks about it and then very few people use it.

The only success Google has is its search engine and in some way Gmail.

I think you hit on the problem, it’s the complexity of the underlying data stream and transformation to presentation. The basic issue can be best seen in the investment required to make every Analytics/DW/ETL environment work. My view of the root of this problem is that all of the definition and connectivity of elements and transformations is 100% custom. We all get sold the next greatest tool this or structure that, but the reality is that our industry has not invested in the methods, processes, and tools to codify information so that it can be bound at runtime.

More simply put, we do not identify & classify data when it is defined in a way that it can be used later. We businesses seem to lack the discipline to make data generators publish their data, or to have consistency of data definitions on which analytics depend. OLTP data is just one such example, unstructured textual data is another such example. As an industry, we do not build tools to exploit the movement of structure with data, in a manner which is efficient enough to actually use.

But more to the point – few if any tools can deal with the dynamic nature of data change in any of the major dimensions: structure, temporal, codification and value distribution. Dynamically cascading entity/attribute changes into the analytic systems is just one example. Exposing Models to data, training data, changing data demographics, the list goes on. Point is, in order to materially change the BI Process, the methods we use today will have to eliminate “custom” and ALL the tools in the chain will need to understand the data, structure, and rules which make that possible.

It’s either that, or the industry will need “Context Binding” methods, which select and Bind logic dependent on the context of the data element, to come into commercial use in order for the current BI methods to materially change. This seems to be where we’re heading, but I am reminded of the optimism surrounding the AI movement before we found the areas it could be useful – particularly since our development methods and processing systems are still deterministic…

BTW – Normal Forms do not solve this BI problem. 5NF provides an abstract structure which avoids the need for DDL changes. The logic to decode this form, to navigate and make meaning of the data content still requires an application or some embodiment of logic/rules to derive an analytic, and to make/take action on that analytic. Which is the exact same problem as a flat file.

Let’s just say I think you’re being hopelessly idealistic, especially in a world where we don’t create all the data we later query.

If my application looks at data initially created by customers, prospects, suppliers, webmasters, or whatever, it’s not very helpful to say “Well, I’m doing my own users a disservice unless I enforce strict naming conventions on the whole bloody world.”

Not to extend this beyond what’s appropriate to a blog thread, but if “blue” is of interest as an attribute yet also of interest as an embedded substring at the same time, then it would seem to me that their is a large semantic disconnect in play in the data model.

Mixing apples and oranges (attributes and substrings) is never a good path to follow. Naive users do that all the time since they just don’t know any better (”I just want anything that has ‘blue’ in it.”), but empowering them to shoot themselves in the foot is not in our long term interest.