The culture problem is related to data management systems forcing data to be transformed into a private and internal form, and all the process that fronts it. Dimensional Modeling is an example. Let’s stop physicalizing dimensional design because that’s what RDBMS products support.

On the cost front, generating data declines at roughly the inverse of moore’s law, not counting non-native per transaction data growth (I’m collecting more and more data about every event).

On the analytics side of this problem – there are many more scans of the full dataset to get a single metric, so this function cost grows non-linearly in relation to the data size.

So – Data costs are declining at the same rate of hardware. Data Analytics costs are RISING per unit of data. Put quite simply, at the upper end of the data size spectrum – data owners cannot afford to buy data management software.

SciDB is for analytics; Cassandra is for OLTP, hold the “T”, which I called HVSP in http://www.dbms2.com/2010/03/13/the-naming-of-the-foo/.

Hadoop is a closer competitor, as are RDBMS, MapReduce-enabled or otherwise.