

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer

Or more so "will it be"? Big Data makes statistics and relevant knowledge all the more important but seems to underplay Sampling Theory.

I've seen this hype around 'Big Data' and can't help wonder that "why" would I want to analyze everything? Wasn't there a reason for "Sampling Theory" to be designed/implemented/invented/discovered? I don't get the point of analyzing the entire 'population' of the dataset. Just because you can do it doesn't mean you should (Stupidity is a privilege but you shouldn't abuse it :)

So my question is this: Is it statistically relevant to analyze the entire data set? The best you could do would be to minimize error if you did sampling. But is the cost of minimizing that error really worth it? Is the "value of information" really worth the effort, time cost etc. that goes in analyzing big data over massively parallel computers?

Even if one analyzes the entire population, the outcome would still be at best a guess with a higher probability of being right. Probably a bit higher than sampling (or would it be a lot more?) Would the insight gained from analyzing the population vs analyzing the sample differ widely?

Or should we accept it as "times have changed"? Sampling as an activity could become less important given enough computational power :)

Note: I'm not trying to start a debate but looking for an answer to understand the why big data does what it does (i.e. analyze everything) and disregard the theory of sampling (or it doesn't?)

share|improve this question
See also: - how to draw valid conclusions from big data. – Anony-Mousse Sep 16 '12 at 9:56
(+1 long ago) I always enjoy reading your insightful questions. They are a real asset to this site. – cardinal Sep 17 '12 at 2:03
@cardinal - I sincerely appreciate your comment. Means a lot coming from you. – PhD Sep 17 '12 at 20:29
up vote 23 down vote accepted

In a word, yes. I believe there are still clear situations where sampling is appropriate, within and without the "big data" world, but the nature of big data will certainly change our approach to sampling, and we will use more datasets that are nearly complete representations of the underlying population.

On sampling: Depending on the circumstances it will almost always be clear if sampling is an appropriate thing to do. Sampling is not an inherently beneficial activity; it is just what we do because we need to make tradeoffs on the cost of implementing data collection. We are trying to characterize populations and need to select the appropriate method for gathering and analyzing data about the population. Sampling makes sense when the marginal cost of a method of data collection or data processing is high. Trying to reach 100% of the population is not a good use of resources in that case, because you are often better off addressing things like non-response bias than making tiny improvements in the random sampling error.

How is big data different? "Big data" addresses many of the same questions we've had for ages, but what's "new" is that the data collection happens off an existing, computer-mediated process, so the marginal cost of collecting data is essentially zero. This dramatically reduces our need for sampling.

When will we still use sampling? If your "big data" population is the right population for the problem, then you will only employ sampling in a few cases: the need to run separate experimental groups, or if the sheer volume of data is too large to capture and process (many of us can handle millions of rows of data with ease nowadays, so the boundary here is getting further and further out). If it seems like I'm dismissing your question, it's probably because I've rarely encountered situations where the volume of the data was a concern in either the collection or processing stages, although I know many have

The situation that seems hard to me is when your "big data" population doesn't perfectly represent your target population, so the tradeoffs are more apples to oranges. Say you are a regional transportation planner, and Google has offered to give you access to its Android GPS navigation logs to help you. While the dataset would no doubt be interesting to use, the population would probably be systematically biased against the low-income, the public-transportation users, and the elderly. In such a situation, traditional travel diaries sent to a random household sample, although costlier and smaller in number, could still be the superior method of data collection. But, this is not simply a question of "sampling vs. big data", it's a question of which population combined with the relevant data collection and analysis methods you can apply to that population will best meet your needs.

share|improve this answer

While there may be hell of a lot of Big Data being produced by the mobile devices and such, there is little usable data in it. If you want to predict the urban travel patterns using foursquare, you may be off by an order of magnitude in estimated flows. Worse, you won't know if you are overestimated or underestimating these flows. You can get an insanely accurate picture of the urban travel patterns of maniacal foursquare users, but unless everyone is required (1) to keep a working smartphone, (2) to run the foursquare app all the time, and (3) to register at any place they stay at for longer than 10 minutes (i.e., get an electronic Census; let libertarians complain about Google and Facebook knowing everything about you), your data will contain unknown biases, and your electronic Deweys will continue to defeat the real-word Trumans (clickable):

If anything, I would expect that this piece of history will be repeating itself, and some big "beer+diapers" forecasts produced from Big Data would be overturned by researchers using more rigorous sampling approaches. It is surprising that probability-based surveys remain accurate even despite falling response rates.

share|improve this answer
(+1) But, wouldn't a stereotypical foursquare user be the antithesis of paranoid. ;-) – cardinal Sep 10 '12 at 13:14
Yeah... probably a bad term. Let me change that to maniacal! – StasK Sep 10 '12 at 13:56
Big data is not the culprit. It is how it is used. When it has information and is applied appropriately it can be very helpful. Data mining is not all bad. – Michael Chernick Sep 11 '12 at 14:23
Great point on the use of big data for traffic information. Since companies like Google and Apple already do this, I think this is a great example of where (currently available) big data can fall short for some audiences, and I tried to incorporate it in my answer too. – Jonathan Sep 11 '12 at 16:15
@Michael, you are right, of course. The data is dirt cheap, but the ways of getting usable information out of it are no going down -- if anything, they are going up, as one now has to sift through more data to get the useful info out. – StasK Sep 11 '12 at 20:30

Whenever one applies techniques of statistical inference, it is important to be clear as to the population about which one aims to draw conclusions. Even if the data that has been collected is very big, it may still relate only to a small part of the population, and may not be very representative of the whole.

Suppose for example that a company operating in a certain industry has collected 'big data' on its customers in a certain country. If it wants to use that data to draw conclusions about its existing customers in that country, then sampling might not be very relevant. If however it wants to draw conclusions about a larger population - potential as well as existing customers, or customers in another country - then it becomes essential to consider to what extent the customers about whom data has been collected are representative - perhaps in income, age, gender, education, etc - of the larger population.

The time dimension also needs to be considered. If the aim is to use statistical inference to support predictions, then the population must be understood to extend into the future. If so, then again it becomes essential to consider whether the data set, however large, was obtained in circumstances representative of those that may obtain in the future.

share|improve this answer
Welcome to our site, Adam! (If you check in regularly, you will even find occasional opportunities to exercise your interests in geometry and number theory. :-) – whuber Sep 10 '12 at 19:49
Great point about the need to consider the population! That's one of the big ways people can get lazy about big data. – Jonathan Sep 11 '12 at 16:18

Yes, sampling is relevant and will remain relevant. Bottom line is that the accuracy of a statistical estimate is generally a function of the sample size, not the population to which we want to generalize. So a mean or an average proportion computed from a sample of 1,000 respondents will yield an estimate of a certain accuracy (with respect to the entire population from which we sampled), regardless of the size of the population (or “how big” the “big data” are are).

Having said that: There are specific issues and challenges that are relevant and should be mentioned:

  1. Taking a good probability sample is not always easy. Theoretically, every individual in the population to which we want to generalized (about which we want to make inferences) must have a known probability of being selected; ideally that probability should be the same (equal probability sample or EPSEM – Equal Probability of Selection). That is an important consideration and one should have a clear understanding of how the sampling process will assign selection probabilities to the members of the population to which one wants to generalize. For example, can one derive from Twitter feeds accurate estimates of overall sentiments in the population at large, including those individuals without twitter accounts?
  2. Big data may contain very complex details and information; put another way, the issue is not sampling, but (micro-) segmentation, pulling out the right details for a small subset of observations that are relevant. Here the challenge is not sampling, but to identify the specific stratification and segmentation of the big data that yields the most accurate actionable information that can be turned into valuable insights.
  3. Another general rule of opinion measurement is that non-sampling errors and biases are usually much bigger than the sampling error and biases. Just because you process 1 hundred gazillion records of respondents expressing an opinion doesn’t make the results more useful if you only have data of a 1000 person subsample, in particular if the questions for the respective survey were not written well and induced bias.
  4. Sometimes sampling is required: For example, if one were to build a predictive model from all data, how would one validate it? How would one compare the accuracy of different models? When there are “big data” (very large data repositories) then one can build multiple models and modeling scenarios for different samples, and validate them (try them out) in other independent samples. If one were to build one model for all data – how would one validate it?

You can check out our 'Big Data Revolution' here.

share|improve this answer
Welcome to our site, Kyra! – whuber Sep 10 '12 at 19:47
+1 for the link :) – PhD Sep 11 '12 at 16:25

From what I've seen of the big data/ML craze, thinking about sampling and the population from which your sample is drawn is just as important as ever--but thought about even less.

I'm "auditing" the Stanford ML class, and thus far we've covered regression and neural networks with nary a mention of population inference. Since this class has been taken by 6 figures-worth of people, there are now an awful lot of people out there who know how to fit data very will without any notion of the idea of a sample.

share|improve this answer
I completely agree. When observing the current craze about Machine Learning (practitioners and programmers mostly), Big Data and "data science", I find it ridiculously common for people to completely ignore sampling, inference, statistical reasoning understanding and meaning and to sacrifice it for blindly applying whatever algorithm is hip on it. You can even see that with questions and some answers here on cross validated. That's why I also believe it to be a hype that will soon be obsolete or borrow statistical epistemology and thus become a branch of stats (I view it as such anyway). – Momo Sep 11 '12 at 0:44
If that ML class is similar to one I audited some time ago, there was a technical assumption slipped in right before the Hoeffding inequality that the training data is a perfectly random sample from the population. Unfortunately, this is almost never the case, at least in my experience, and in the examples of applications of the techniques throughout the course. It is still not the case when you use "big data." – Douglas Zare Sep 16 '12 at 3:55

Cross validation is an specific example of sub-sampling which is quite important in ML/big data. More generally, big data is still usually a sample of a population, as other people here have mentioned.

But, I think OP might be specifically referring to sampling as it applies to a controlled experiments, versus observational data. Usually big data is thought of as the latter, but to me at least there are exceptions. I would think of randomized trials, A/B testing, and multiarmed bandits in e-commerce and social network settings as examples of "sampling in big data settings."

share|improve this answer

Many big data methods are actually designed around sampling.

The question should be more on the line of:

Shouldn't we use systematic sampling with big data, too?

A lot of the "big data" stuff is still pretty fresh, and sometimes naive. K-means for example can be trivially parallelized, and thus works for "big data" (I'm not going to talk about the results, they are not very meaningful; and probably not very different to those obtained on a sample!). As far as I know this is what the k-means implementation in Mahout does.

However, research is going beyond the naive parallelization (that may still require a large amount of iterations) and tries to do K-means in a fixed number of iterations. Example for this:

  • Fast clustering using MapReduce
    Ene, A. and Im, S. and Moseley, B.
    Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, 2011

And guess what, their approach is heavily based on sampling.

Next example: Decision forests. That is essentially: for several samples from the data set, build a decision tree each. Can again be trivially parallelized: put each sample on a separate machine. And again, it is a sampling based approach.

So sampling is one of the key ingredients to big data approaches!

And there is nothing wrong with this.

share|improve this answer

In the areas where Big Data is gaining popularity: Search, Advertising, Recommender Systems like Amazon, Netflix , there is a very Big incentive to explore the entire data set.

The objective of these systems is to tailor recommendations / suggestions to every single member of the population. Also, the number of attributes being studied is enormous. The average web analytics system may measure click-through rate, "thermal tracking" of the "hot areas" in a page, social interactions, etc and weigh these against a large set of predetermined objectives.

More importantly, most of the places where Big Data is now ubiquitous are "online" data streams i.e data is constantly being added / updated. Devising a sampling scheme which covers all these attributes without an inherent bias and still deliver promising results (read better margins) is a challenge.

Sampling still remains highly relevant for surveys, medical trials, A/B testing, quality assurance.

In a nutshell, sampling is very useful when the population to be studied is very large and you are interested in the macroscopic properties of the population. 100% checking (Big Data) is necessary for exploiting the microscopic properties of the system

Hope this helps :)

share|improve this answer
So is your point that they don't want to be able to generalize beyond the data they have to the data they don't yet? Or that they think their sample is so large they don't have to worry about those issues? Or that the underlying parameters would be changing over time, so is doesn't matter so long as they continue to update as new data flows in? – gung Oct 1 '12 at 14:12
@gung the issue is not the size of the sample, but the problem of generating an unbiased sample for a dataset which has a huge number of attributes. And generalization is usually done by Machine Learning algorithms, trained on a portion of the data set. The online data streams constantly coming in makes the issue of sampling secondary as batch updates can be used to change the parameters. – rrampage Oct 1 '12 at 20:17

protected by whuber Mar 4 '15 at 2:49


Would you like to answer one of these unanswered questions instead?
