Since I don’t see R going 64 bit on Windows anytime soon, I’m starting the process of specing out a system where R runs in a Linux 64 bit OS and has access to a lot more memory space to solve statistical problems. Currently, the idea is to make the Linux system a VM that is easily installed and has all quite a bit of the R libraries already installed.

All I need is time!

An example use-case where we’ve used R as a UDF: doing various forms of linear regression required the use of a matrix pseudo-inverse routine to solve the eigenvalue problem. Instead of writing our own pseudo-inverse routine, we instead used the one that comes with R to evaluate different approaches. The matrix solve part is actually pretty small, so we were able to do it in memory as the final stage of processing and the R routine was a good fit.

In the end, we ended up implementing our own pseudo-inverse routine, now available as the ‘pinv()’ from within Greenplum. It’s written in C internally and is blazingly fast.

So – the embedded R UDF capability within Greenplum is useful, but it’s often good to re-write the routine for performance optimization when moving to production. We provide many of these kinds of functions to our customers in the form of libraries. Note that we also provide a large array of built-in matrix manipulation routines as well.

It might be helpful if you listed a few ways R results might wind up on disk — if indeed there are a few different ways.

Thanks,

CAM

That’s one problem with R in general, it holds its results in RAM.

With Greenplum, we enable you to run R programs as stored procedures, which provides you the ability to reuse the math routines in R to some extent, specifically to help you calculate intermediate results as part of WINDOW functions or other OLAP use-cases.

We have also re-implemented some of the routines that R provides as native parallel functions within Greenplum, including multi-variable linear regression, a naive bayes classifier and some others.