Announcing Stack Overflow Documentation



Sign up and start helping → Learn more about Documentation →

By "handle" I mean manipulate multi-columnar rows of data. How does R stack up against tools like Excel, SPSS, SAS, and others? Is R a viable tool for looking at "BIG DATA" (hundreds of millions to billions of rows)? If not, which statistical programming tools are best suited for analysis large data sets?

share|improve this question

closed as off-topic by Richard Scriven, Pascal, amdixon, Bhargav Rao, HaveNoDisplayName Dec 24 '15 at 7:10



As long as you don't store it to RAM, you can churn through virtually endless data using any language (Python). – Blender Apr 3 '11 at 6:01
Stick with Excel, it is web-scale. Oh, wait, am I two days late? – Dirk Eddelbuettel Apr 3 '11 at 16:45
Excel can't handle CSV files with more than one million rows. – Yay295 Oct 22 '14 at 18:05

If you look at the High-Performance Computing Task View on CRAN, you will get a good idea of what R can do in a sense of high performance.

share|improve this answer

You can in principal store as much data as you have RAM with the exception that, currently, vectors and matrices are restricted to 2^31 - 1 elements because R uses 32-bit indexes on vectors. General vectors (lists, and their derivative data frames) are restricted to 2^31 - 1 components, and each of those components has the same restrictions as vectors/matrices/lists/data.frames etc.

Of course these are theoretical limits, if you want to do anything with data in R it will inevitably require space to hold a couple of copies at least, as R will usually copy data passed in to functions etc.

There are efforts to allow on disk storage (rather than in RAM); but even those will be restricted to the 2^31-1 restrictions mentioned above in use in R at any one time. See the Large memory and out-of-memory data section of the High Performance Computing Task View linked to in @Roman's post.

share|improve this answer

Perhaps a good indication of its suitability for "big data" is the fact that R has emerged as the platform of choice for developers competing in Kaggle.com data modeling competitions. See the article on the Revolution Analytics website -- R beats out SAS and SPSS by a healthy margin. What R lacks in out of the box number crunching power it apparently makes up for in flexibility.

In addition to what's available on the web there are several new books for how to hot-rod R for tackling big data. The Art of R Programming (Matloff 2011; No Starch Press) provide introductions to writing optimized R code, parallel computing, and using R in conjunction with C. The entire book is well-written with great code samples and walk-throughs. Parallel R (McCallum & Weston 2011; O'Reilly) looks good too.

share|improve this answer

With relatively moderate resources such as:

  • Processor: Intel® Celeron(R) CPU N2830 @ 2.16GHz × 2
  • Memory: 3.7 GiB

it takes ~25 seconds to load the csv (200k rows * 90 columns) file,simple one variable plot takes few seconds.

share|improve this answer

I'll explain my short story with R and big data set.
I had a connector from R to RDBMS,

  • where I stored 80mln compounds.

I've build a queries which gathered some subset of this data.
Then manipulate on this subset.
R was simply choking with more than 200k rows in memory on my PC.

  • core duo
  • 4 GB ram

So working on some appropriate subset for machine is good approach.

share|improve this answer
Wow,.. guys which gave '-', could you please explain why? – bua Apr 4 '11 at 12:50
try adding a reproducible example, perhaps using SQLite instead of a RBDMS. or create a random vector with 200k rows and see if it chokes. Or tell us how many columns does your dataset have. etc etc – Eduardo Leoni Apr 4 '11 at 14:25
