I am flattered that you confused me for David DeWitt and Stonebraker

They are the ones who are quoted as saying MapReduce wasn’t something new. MapReduce is a creation of Ghemawat and Dean.

All I’m saying is that recent claim by many that they’ve been “doing MapReduce all along” are not entirely true (and not entirely false either).

I’m not equating MR with the MPP redistribution framework, hence my comment that reads “… the simple answer is this: they have been doing something VERY MUCH LIKE MapReduce all along”.

Thanks,

-amrith

Our SQL-MR syntax goes a long way in ensuring that the relational model is preserved. For example, MR functions consume and produce relations; MR invocations are modeled as stored-procedure invocations. This means that a customer can migrate to non-Aster nCluster installations with an effort similar to migrating their user-defined functions from one platform to another.

The best part of our SQL-MR framework is that the implementation of the MR functions are in open languages chosen by the customer (e.g., Java, Perl, Python, C++, C#, Ruby, etc.). This means that the actual code is not proprietary to Aster nCluster. The code snippets/sub-functions can be re-used in other platforms as well. In addition, the Map Reduce programming model has widespread popularity allowing for portability since the structure of one’s code is first to design the Map-Reduce design, and secondarily express it in SQL-MR to the extent that one uses features unique to our platform.

The libraries of Aster SQL-MR functions that we provide are, of course, proprietary. They have innovations in data structure and processing that ensure high performance of the compute function. We’ve published the source code of some of these functions; for others, we’ve published the algorithms but not the source code; for the rest, we may not publish either the code/algorithm. In fact, we encourage our partners who write SQL-MR functions complete discretion on publishing their functions or providing only binaries to protect their IP.

The important point to note here is that we are committed to providing an open platform in which one function is not forced upon the end-user.

===

@Amrith: Whenever an innovative system becomes mainstream, there are always claims that the innovation is nothing new! We went through this in the 1990s when Java appeared on the scene as well.

We cannot equate Map-Reduce programming framework to the internal re-distribution mechanism of tuples in MPP databases. The Map-Reduce programming framework is innovative because it provides a way of attaining parallelism for arbitrary computations. The internal MPP DB tuple re-distribution mechanisms operated on one-tuple at a time with a static hash function that had the number of partitions statically pre-defined. The mechanism could not be re-used by users or database applications – in fact, it could not be re-used even by stored procedures that were part of the MPP DB framework.

If you are interested, please look at the Related Work section of our VLDB 2009 conference paper. http://www.asterdata.com/resources/downloads/whitepapers/sqlmr.pdf

This post was mentioned on Twitter by jameskobielus: Read @CurtMonash on MapReduce (http://bit.ly/2pdJ1W). None of this brand new. Nor is it true standard. Vendor implementations vary widely….

And as for the recent claim by many that they’ve been “doing MapReduce all along”, the simple answer is this: they have been doing something VERY MUCH LIKE MapReduce all along.

MPP databases horizontally partition the data and process partitions on distinct nodes. MapReduce does not perform the partitioning apriori, it does it at runtime. MPP implementations that I am familiar with always perform the partitioning of persistent data (tables) apriori with provisions to redistribute the data as part of the query processing mechanism.

Dean & Ghemawat write, “The Map invocations are distributed across multiple machines by automatically partitioning the input data into a set of M splits. The input splits can be processed
in parallel by different machines. Reduce invocations are distributed by partitioning the intermediate key space into R pieces using a partitioning function (e.g., hash(key) mod R). The number of partitions (R) and the partitioning function are specifed by the user.”

Each MPP implementation has a different name for the mechanism to perform this splitting. In effect therefore MapReduce is another mechanism for MPP’izing the solution to a problem and there is some merit to the claim being made by MPP database vendors that they’ve been doing MapReduce all along.

The Aster syntax is Aster-specific, just as if you used any other vendor’s proprietary SQL extensions.

CAM