I agree with you on avoiding a tit for tat argument, so I’ll just focus on asking for clarification.

Does the redistribution have to be carried out every time the underlying table is changed, or can it be done incrementally?

Since most DW schemas (especially star schemas) have very predictable access paths, does this mean most customers redistribute their fact and large dimension tables and leave the duplicates in place until the next load?

Since these redistributed tables are effectively materialized views, how do you keep them in RAM, as implied by your earlier emails? Surely they are too big?

Thanks for the comment on our value proposition. I agree that it’s much more attractive than any we’ve come across so far. I’m looking forward to seeing how it stacks up against yours

Stuart

Are these redistributions additional copies? fif not, what do you mean by “drop”?

Thanks,

CAM

However, I will answer this specific point.

There are a couple of bits of information that Stuart is missing. WX2 does indeed randomly distribute the data across all available nodes when the data is loaded. No decisions have to made in advance on how the data should be loaded to satisfy a particular query profile, which simplifies the whole process. It also true that if nothing else was done then WX2’s optimiser would automatically re-distribute the data, if the data was not suitably distributed, on a query by query basis. However this is not the normal mode of operation. WX2 has simple SQL extensions that allow the data to be re-distributed as a separate step. The optimiser will then use the new distribution for any subsequent queries without needing to perform a re-distribution. Re-distributions can be created and dropped as required.

To answer Stuart’s point about GbE and the performance of a re-distribution. A 20TB WX2 system is capable of re-distributing a 1TB table in around 20 seconds. This is assuming that every WX2 nodes needs to re-distribute all that tables rows and columns. Typically this is not the case because WX2 only re-distributes the rows and columns it needs for a particular query. WX2 nodes support multiple GbE links and we have lots of nodes. Incidentally we also work quite happily with Infiniband as well as GbE.

True, we have to move data around if a join is on a non-partitioned key, but that’s very unusual for all but fairly small tables in our architecture. Also, we can move data over Infiniband MUCH faster than GigE with almost zero impact on processor load. Kognitio’s comment about moving to memory being faster is meaningless, since GigE will be the (slower than disk) bottleneck.

Moving data around for every join seems pointless and would appear to be an unnecessary overhead that impacts both scaling and concurrency.

I don’t know much about text search, but the nature of that problem would seem to be very different to relational-based databases, where set logic allows you to process joins very efficiently. Why not take advantage of the inherent structure of the data where you can?

Given all this, I just don’t understand why the Kognitio designers chose to distribute data randomly.

Stuart
DATAllegro

I’d also note that this design is analogous to those used in text search engines, including (I presume) every major web search engine, and also FAST’s.

I’ve been meaning to post on exactly that point over on Text Technologies. Maybe I’ll do so now and point to this thread.

How is that different from the situation for DATallegro in cases where the join key is neither range partitioned nor hashed on?

Also, while I don’t think Kognitio moves their compressed bitmaps around from node to node, I’ll ping them to see if they want to jump into this discussion.

Thanks,

CAM