@Curt Even a very distributed environment, like Hadoop, makes it very clear you should not expect to fit everything into memory.

The thing that makes the SAS map/reduce concept kind of reasonable to me, is that SAS is natively much more in a map/reduce file processing kind of mode then in a distributed RDBM mode. I could also imagine using something like Hadoop Streams to take existing sas and run it distributing, provided you were clever with your hashing (and could find a way around the ridiculous license cost).

Trying to get SAS to run natively inside an RDBMS on the other hand is an order of magnitude different problem.

SAS is very far from SQL.

So far SAS/Teradata seems to be mostly trying to rever engineer SAS procedures as C compiled objects inside Teredata, one at a time.

But many statisticians love SAS because so often they don’t have to worry at all about things like memory use, where as in R you always do.

My point was more that R on a sufficiently parallel system might be able to process more data …

Best,

CAM

SAS-as-a-service makes a lot of sense. With SAS, you load in data and get out reports, summaries,etc. It’s not like a database where you’re constantly reading the data you load in. Internet latency is a killer for DBMS access, but not for getting reports from SAS. So it makes perfect sense for SAS do SaaS. This will come with a completely different pricing structure.

Licensing SAS for the Amazon cloud – well that is just a SAS license. I don’t see why it would be cheaper just because it’s in a computer that runs in a “cloud”.

R (all versions to my knowledge) only deals with data in memory. If you want to handle more data than you have memory, you will have to chunk it through a piece at a time. There are libraries to help, but still it can be hard to get a handle on. And if your analytics library wants all of the data at once – well, you’re out of luck and it needs to be rewritten to handle chunks.

As of fairly recently, S-PLUS has a feature called Big Data that replaces the memory cache with a disk cache. So programs operate thinking they’re dealing with memory, but really it reads from the disk. There is work to implement this in R, but I don’t know of a widely used and stable version yet. So for now, the state is this: most R libraries don’t handle chunking data, so most libraries can only handle as much data as you have memory. R users have been known to upgrade to 64bit because of this. In the future, there are many possible solutions including allowing R working “memory” to be cached on disk, or rewriting libraries to handle data differently.

That depends the specific implementation of R, doesn’t it?

CAM

For example, you can hook SAS up to a terabyte of data and run some analysis. It might be slow and disk intensive, but it’ll work. Try a huge data set with R and it’ll fail; you’ll either need some custom programming or it’ll just be impossible, depending on the analysis.

For that and several other reasons, SAS seems safe for now. But I agree that doesn’t mean they should consider themselves safe in the long run.

http://support.sas.com/rnd/scalability/procs/index.html

However, it is true that this happens at the level of an individual processing step, not an entire SAS program.

http://www2.sas.com/proceedings/forum2007/036-2007.pdf