What Data Mining Can and Can't Do

By Allan Alter  |  Posted 06-13-2007

What Data Mining Can and Can't Do

Peter Fader, professor of marketing at University of Pennsylvania's Wharton School, is the ultimate marketing quant—a world-class, award-winning expert on using behavioral data in sales forecasting and customer relationship management. He's perhaps best known for his July 2000 (PDF) expert witness testimony before the U.S. District Court in San Francisco that Napster actually boosted music sales. (Napster was then the subject of an injunction for copyright infringement and other allegations brought against it by several major music companies.)



FADER: Data mining tools are very good for classification purposes, for trying to understand why one group of people is different from another. What makes some people good credit risks or bad credit risks? What makes people Republicans or Democrats? To do that kind of task, I can't think of anything better than data mining techniques, and I think it justifies some of the money that's spent on it. Another question that's really important isn't which bucket people fall into, but when will things occur? How long will it be until this prospect becomes a customer? How long until this customer makes the next purchase? So many of the questions we ask have a longitudinal nature, and I think in that area data mining is quite weak. Data mining is good at saying, will it happen or not, but it's not particularly good at saying when things will happen.

Data mining can be good for certain time-sensitive things, like is this retailer the kind that would probably order a particular product during the Christmas season. But when you want to make specific forecasts about what particular customers are likely to do in the future, not just which brand they're likely to buy next, you need different sets of tools. There's a tremendous amount of intractable randomness to people's behavior that can't be captured simply by collecting 600 different explanatory variables about the customer, which is what data mining is all about.


The Risks of Data


Do you think people understand the limitations of data mining?

They don't. And this has nothing to do with data mining or marketing, but it has a lot to do with human nature. We're seeing the same issues arising in every area of science. As data collection technology and model-building capabilities get better, people keep thinking they can answer the previously unknowable questions. But whether it's the causes of diseases or mechanical failure, there's only so much we can pin down by capturing data.


I can't make generalizations that are too broad, but there are some people who are hammers looking for nails. They think they can answer any problem using one set of procedures, and that's a big mistake. When you go into other domains, you need to pull out different tools. One of the things that just makes me crazy is when people misuse the kinds of statistics that are associated with data mining. A lift curve will show us how well our predicted rank order of customer propensities corresponded to their actual behavior. That's a fine thing to do in a classification setting, but it's not particularly diagnostic in a longitudinal setting. We want 'when'-type diagnostics to answer 'when'-type questions. People just aren't looking in the right places to see whether their model's working.


The difference is that just because people have a tendency to do things doesn't mean that they will. You might be someone who buys from Amazon once a month on average. Does that mean over the next 10 years, over the next 120 months, you'll buy 120 items? No. You could go two years without buying, or you might buy five items in a given month. The amount of variability around your propensity is huge. That's where all this randomness comes in.

Have companies hurt themselves by misusing data mining tools?

Let me start with a positive example. I have tremendous admiration for what actuaries do, and therefore for the way insurance companies deal with their customers. Actuaries will not look at all your characteristics and say when you will die. They'll simply come up with a probabilistic statement about the likelihood that someone with your characteristics will die, or what percent of people who share characteristics will live to be 70. They understand that it's pretty much impossible to make statements about each and every policyholder.

Now, carry that over to the marketing world. Lots of firms talk about one-to-one marketing. I think that's a real disservice to most industries. One-to-one marketing only works when you have a very deep relationship with every customer. So one-to-one marketing works great in private wealth management, or in a business-to-business setting where you meet with the client at least once a month, and understand not just their business needs but what's going on in their life. But in areas approaching a mass market, where you can't truly distinguish each individual, you just have a bunch of people and a bunch of characteristics that describe them. Then the notion of one-to-one marketing is terrible. It will do more harm than good, because the customers will act more randomly than you expect, and the cost of trying to figure out what specific customers will do far outweighs the benefits you could get from that level of detail.

It's very hard to say who's going to buy this thing and when. To take that uncertainty and square it by looking across two products, or to raise it to the nth power by looking across a large portfolio of products, and say "these two go together," and make deterministic statements as opposed to talking about tendencies and probabilities, can be very, very harmful. It's much more important for companies to come up with appropriate groupings of similar people, and make statements about them as a group.

I don't want to pick on Amazon in particular; they really tout the capabilities of their recommendations systems. But maybe this customer was going to buy book B anyway, and therefore all the recommendations were irrelevant. Or maybe they were going to buy book C, which would have been a higher-margin item, so getting them to buy book B was a mistake. Or maybe they're becoming so upset by irrelevant recommendations that they're going away entirely. I don't want in any way to suggest that cross-selling shouldn't be done, but what I'm suggesting is that the net gains from it are less than people might think. It often can't justify the kinds of investments that firms are making in it.


Probability models are a class of models that people used back in the old days when data weren't abundantly available. These modeling procedures are based on a few premises: People do things in a random manner; the randomness can be characterized by simple probability distributions; and the propensities for people to do things vary-over time, across people, across circumstances. Probably the best known example is survival analysis, which stems largely from the actuary sciences. It's also used in manufacturing. You put a bunch of lightbulbs on a testing board and see how long they last. In many ways, that's what I suggest we do with customers. We're not going to make statements about any one lightbulb, just like we shouldn't make statements about any one customer. We'll make collective statements about how many of these bulbs will last for 1,000 hours. It turns out that the analogy of survival analysis in manufacturing and actuarial and life sciences carries over amazingly well to customers. A lot of managers would bristle at the idea, but I think that metaphor is far better than all this excessive customization and personalization that's been going on. Customers are different from each other just as lightbulbs are, but for reasons that we can't detect, and reasons that we'll have a very hard time taking advantage of.

Using Models to Solve


What kind of problems can probability models solve?




Can probability models capture longitudinal or predictive information?











