Archive for the 'statistics' category

Least Square Linear Regression

Jan 30 2013 Published by under statistics







So what we want to do is find a linear equation, y = mx + b where the mean-square distance is minimal. All we need to do is find values for m (the slope of the line) and b (the point where the line crosses the y axis, also called the y intercept). And, in fact, b is relatively easy to compute once we know the slope of the line. So the real trick is to find the slope of the line.

The way that we do that is: first we compute the means of x and y, which we'll call \overline{x} and \overline{y}. Then using those, we compute the slope as:





}<br />






20 responses so far

Debunking Two Nate Silver Myths

Nov 09 2012 Published by under statistics

I followed our election pretty closely. My favorite source of information was Nate Silver. He's a smart guy, and I love the analysis that he does. He's using solid math in a good way to produce excellent results. But in the aftermath of the election, I've seen a lot of bad information going around about him, his methods, and his result.

First: I keep seeing proclamations that "Nate Silver proves that big data works".


There is nothing big data about Nate's methods. He's using straightforward Bayesian methods to combine data, and the number of data points is remarkably small.

Big data is one of the popular jargon keywords that people use to appear smart. But it does actually mean something. Big data is using massive quantities of information to find patterns: using a million data points isn't really big data. Big data means terabytes of information, and billions of datapoints.

When I was at Google, I did log analysis. We ran thousands of machines every day on billions of log records (I can't say the exact number, but it was in excess of 10 billion records per day) to extract information. It took a data center with 10,000 CPUs running full-blast for 12 hours a day to process a single days data. Using that data, we could extract some obvious things - like how many queries per day for each of the languages that Google supports. We could also extract some very non-obvious things that weren't explicitly in the data, but that were inferrable from the data - like probable network topologies of the global internet, based on communication latencies. That's big data.

For another example, look at this image produced by some of my coworkers. At foursquare, we about five million points of checkin data every day, and we've got a total of more than 2 1/2 billion data points. By looking at average checkin densities, and then comparing that to checkin densities after the hurricane, we can map out precisely where in the city there was electricity, and where there wasn't. We couldn't do that by watching one person, or a hundred people. But by looking at the patterns in millions and millions of records, we can. That is big data.

This doesn't take away from Nate's accomplishment in any way. He used data in an impressive and elegant way. The fact is, he didn't need big data to do this. Elections are determined by aggregate behavior, and you just don't need big data to predict them. The data that Nate used was small enough that a person could do the analysis of it with paper and pencil. It would be a huge amount of work to do by hand, but it's just nowhere close to the scale of what we call big data. And trying to do big data would have made it vastly more complicated without improving the result.

Second: there are a bunch of things like this.

The point that many people seem to be missing is that Silver was not simply predicting who would win in each state. He was publishing the odds that one or the other candidate would win in each statewide race. That's an important difference. It's precisely this data, which Silver presented so clearly and blogged about so eloquently, that makes it easy to check on how well he actually did. Unfortunately, these very numbers also suggest that his model most likely blew it by paradoxically underestimating the odds of President Obama's reelection while at the same time correctly predicting the outcomes of 82 of 83 contests (50 state presidential tallies and 32 of 33 Senate races).

Look at it this way, if a meteorologist says there a 90% chance of rain where you live and it doesn't rain, the forecast wasn't necessarily wrong, because 10% of the time it shouldn't rain - otherwise the odds would be something other than a 90% chance of rain. One way a meteorologist could be wrong, however, is by using a predictive model that consistently gives the incorrect probabilities of rain. Only by looking a the odds the meteorologist gave and comparing them to actual data could you tell in hindsight if there was something fishy with the prediction.

Bzzt. Sorry, wrong.


In a frequentist interpretation, saying that an outcome of an event has a probability X% of occuring, you're saying that if you were to run an infinite series of repetitions of the event, then on average,
the outcome would occur in X out of every 100 events.



The author that I quoted above is being a frequentist. In another section of his articple, he's more specific:

...The result is P= 0.199, which means there's a 19.9% chance that it rained every day that week. In other words, there's an 80.1% chance it didn't rain on at least one day of the week. If it did in fact rain everyday, you could say it was the result of a little bit of luck. After all, 19.9% isn't that small a chance of something happening.

That's frequentist intepretation of the probability - which makes sense, since as a physicist, the author is mainly working with repeated experiments - which is a great place for frequentist interpretation. But looking at the same data, a Bayesian would say: "I have an 19.9% certainty that it will rain today". Then they'd go look outside, see the clouds, and say "Ok, so it looks like rain - that means that I need to update my prediction. Now I'm 32% certain that it will rain". Note that nothing about the weather has changed: it's not true that before looking at the clouds, 80.1 percent of the time it wouldn't rain, and after looking, that changed. The actual fact of whether or not it will rain on that specific day didn't


One last metaphor, and I'll stop. Think about playing craps, where you're rolling two six sided dice.
For a particular die, a frequentist would say "A fair die has a 1 in 6 chance of coming up with a 1". A
Bayesian would say "If I don't know anything else, then my best guess is that I can be 16% certain that a 1
will result from a roll." The result is the same - but the reasoning is different. And because of the difference in reasoning, you can produce different predictions.


34 responses so far

Book Review: The Manga Guide to Statistics

Dec 13 2008 Published by under statistics



If you haven't heard of the "Manga Guides", they're an interesting idea. In Japan, comic books ("Manga") are much more common and socially accepte than they typically are in the US. It's not at all unusual to see Japanese adults sitting in the subway reading Manga. Manga has a very distinctive artistic style, with its own
set of common artistic conventions. The Manga Guides are textbooks written as
Manga-style comics. In this case, it's an introductory text on statistics.

The short version of the review: terrific book; engaging, thorough, and fun. Highly recommended. Details beneath the fold.

Continue Reading »

No responses yet

Margin of Error and Election Polls

Oct 28 2008 Published by under statistics

write you a blog article on that topic.


thing is, pretty much every one of those reports is
utter rubbish.

Continue Reading »

No responses yet

Probability Distributions

Apr 14 2008 Published by under statistics

I'd like to start with a quick apology. Sorry that both the abstract algebra and the new game theory posts have been moving so slowly. I've been a bit overwhelmed lately with things that need doing right away, and
by the time I'm done with all that, I'm too tired to write anything that
requires a lot of care. I've known the probability theory stuff for so long,
and the parts I'm writing about so far are so simple that it really doesn't take nearly as much effort.

With that out of the way, today, I'm going to write about probability distributions.

Continue Reading »

No responses yet

Random Variables

Apr 09 2008 Published by under statistics

The first key concept in probability is called a random variable.
Random variables are a key concept - but since they're a key concept of the
frequentist school, they are alas, one of the things that bring out more of
the Bayesian wars. But the idea of the random variable, and its key position
in understanding probability and statistics predates the divide between
frequentist and Bayesian though. So please, folks, be a little bit patient,
and don't bring the Bayesian flamewars into this post, OK? If you want to
rant about how stupid frequentist explanations are, please keep it in the comments here. I'm trying to
explain basic ideas, and you really can't talk about probability and
statistics without talking about random variables.

Continue Reading »

No responses yet

Schools of thought in Probability Theory

Apr 07 2008 Published by under statistics

To understand a lot of statistical ideas, you need to know about
probability. The two fields are inextricably entwined: sampled statistics
works because of probabilistic properties of populations.

I approach writing about probability with no small amount of trepidation.

For some reason that I've never quite understood, discussions of probability
theory bring out an intensity of emotion that is more extreme than anything else
I've seen in mathematics. It's an almost religious topic, like programming
languages in CS. This post is intended really as a flame attractor: that is, I'd request that if you want to argue about Bayesian probability versus frequentist probability, please do it here, and don't clutter up every comment thread that
discusses probability!

There are two main schools of thought in probability:
frequentism and Bayesianism, and the Bayesians have an intense contempt for the
frequentists. As I said, I really don't get it: the intensity seems to be mostly
one way - I can't count the number of times that I've read Bayesian screeds about
the intense stupidity of frequentists, but not the other direction. And while I
sit out the dispute - I'm undecided; sometimes I lean frequentist, and sometimes I
lean Bayesian - every time I write about probability, I get emails and comments
from tons of Bayesians tearing me to ribbons for not being sufficiently

It's hard to even define probability without getting into trouble, because the
two schools of thought end up defining it quite differently.

The frequentist approach to probability basically defines probability in terms
of experiment. If you repeated an experiment an infinite number of times, and
you'd find that out of every 1,000 trials, a given outcome occured 350 times, then
a frequentist would say that the probability of that outcome was 35%. Based on
that, a frequentist says that for a given event, there is a true
probability associated with it: the probability that you'd get from repeated
trials. The frequentist approach is thus based on studying the "real" probability
of things - trying to determine how close a given measurement from a set of
experiments is to the real probability. So a frequentist would define probability
as the mathematics of predicting the actual likelihood of certain events occuring
based on observed patterns.

The bayesian approach is based on incomplete knowledge. It says that you only
associate a probability with an event because there is uncertainty about it -
because you don't know all the facts. In reality, a given event either will happen
(probability=100%) or it won't happen (probability=0%). Anything else is an
approximation based on your incomplete knowledge. The Bayesian approach is
therefore based on the idea of refining predictions in the face of new knowledge.
A Bayesian would define probability as a mathematical system of measuring the
completeness of knowledge used to make predictions. So to a Bayesian, strictly speaking, it's incorrect to say "I predict that there's a 30% chance of P", but rather "Based on the current state of my knowledge, I am 30% certain that P will occur."

Like I said, I tend to sit in the middle. On the one hand, I think that the
Bayesian approach makes some things clearer. For example, a lot of people
frequently misunderstand how to apply statistics: they'll take a study showing
that, say, 10 out of 100 smokers will develop cancer, and assume that it means
that for a specific smoker, there's a 10% chance that they'll develop cancer.
That's not true. The study showing that 10 out of 100 people who smoke will develop cancer can be taken as a good starting point for making a prediction - but a Bayesian will be very clear on the fact that it's incomplete knowledge, and that it therefore isn't very meaningful unless you can add more information to increase the certainty.

On the other hand, Bayesian reasoning is often used by cranks.
A Bayesian
argues that you can do a probabilistic analysis of almost anything, by lining
up the set of factors that influence it, and combining your knowledge of those factors in the correct way. That's been used incredibly frequently by cranks for
arguing for the existence of God, for the "fact" that aliens have visited the
earth, for the "fact" that artists have been planting secret messages in
paintings, for the "fact" that there are magic codes embedded in various holy texts, etc. I've dealt with these sorts of arguments numerous times on this blog; the link above is a typical example.

Frequentism doesn't fall victim to that problem; a frequentist only
believes probabilities make sense in the setting of a repeatable experiment. You
can't properly formulate something like a probabilistic proof of God under the
frequentist approach, because the existence of a creator of the universe isn't a
problem amenable to repeated experimental trials. But frequentism suffers
from the idea that there is an absolute probability for things - which is often ridiculous.

I'd argue that they're both right, and both wrong, each in their own settings. There are definitely settings in which the idea of a fixed probability based on a model of repeatable, controlled experiment is, quite simply, silly. And there
are settings in which the idea of a probability only measuring a state of knowledge is equally silly.

No responses yet

Introduction to Linear Regression

Mar 27 2008 Published by under statistics

Suppose you've got a bunch of data. You believe that there's a linear
relationship between two of the values in that data, and you want to
find out whether that relationship really exists, and if so, what the properties
of that relationship are.

Continue Reading »

No responses yet

Basic Statistics: Mean and Standard Deviation

Mar 24 2008 Published by under statistics

Several people have asked me to write a few basic posts on statistics. I've
written a few basic posts on the subject - like, for example, this post on mean, median and mode. But I've never really started from the beginnings, for people
who really don't understand statistics at all.

To begin with: statistics is the mathematical analysis of aggregates. That is, it's a set of tool for looking at a large quantity of data about a population, and finding ways to measure, analyze, describe, and understand the information about the population.

There are two main kinds of statistics: sampled statistics, and
full-population statistics. Full-population statistics are
generated from information about all members of a population; sampled statistics
are generated by drawing a representative sample - a subset of the population that should have the same pattern of properties as the full population.

My first exposure to statistics was full-population statistics, and that's
what I'm going to talk about in the first couple of posts. After that, we'll move on to sampled statistics.

Continue Reading »

No responses yet

Sex Crimes and Illegal Immigrants: Misuse of Statistics for Politics

Jan 18 2007 Published by under statistics

Yet another reader sent me a great bad math link. (Keep 'em coming guys!) This one is an astonishingly nasty slight of hand, and a great example of how people misuse statistics to support a political agenda. It's by someone
named "Dr. Deborah Schurman-Kauflin", and it's an attempt to paint illegal
immigrants as a bunch of filthy criminal lowlifes. It's titled "The Dark Side of Illegal Immigration: Nearly One Million Sex Crimes Committed by Illegal Immigrants in the United States."

With a title like that, you'd think that she has actual data showing that nearly one million sex crimes were committed by illegal immigrants, wouldn't you? Well, you'd be wrong.

Continue Reading »

No responses yet

Older posts »
