Thursday, October 29, 2009

Why You Need Anecdotal Evidence

Have you ever heard the phrase "That's just anecdotal"? Usually, it's used to discount somebody's opinion about the way something should be. In UX terms, if you've observed a user, saw them have a problem, and want to respond to what you saw, you may have someone tell you it's just anecdotal -- in other words, you don't have hard, objective, statistical evidence, so forget it.

But there's a contradiction here. There's a belief that, if you gather enough anecdotal evidence, it magically becomes hard evidence. You can even see this in the latest ad campaign for Windows. Millions of people sent in their ideas, or were surveyed or studied, and their opinions -- anecdotal evidence -- is now real evidence. How does this magic work? Well, it can't. Two people who said different but similar things, or even the same things under different conditions, can't be lumped together in some statistical box.

But that doesn't mean that anecdotal evidence is worthless. Quite the contrary. It's invaluable (and Microsoft should be commended for actually listening to users). Anecdotal evidence provides you with something that hard data can't -- feelings. In a typical usability test or, these days, a web site A/B test, success is measured by whether or not somebody succeeds in a task. How about whether they frowned or smiled, tapped their fingers on their desk impatiently, hummed to themselves, or cursed at their computer?

I recommend that you gather as much anecdotal evidence as you can and let it infect your world view. Try to feel what your users feel, think like they think, and use that to design your products. And, after that, gather hard evidence on whether you were right or wrong and move forward from there. But, if you don't start with a feeling about what's the right thing to do, no amount of hard evidence will help you.


Ron George said...

Coming from several years of UX Design at Microsoft I can also say that UX test results can be your own worst enemy. If you happen to be RITE testing and get a batch of 6 people who are all particularly clueless in a specific area that you are testing, it can throw your results way off. The worst part of that is having to try and justify not doing anything about it. I always say that UX can be used for good and evil, so use it wisely.

Roy Leban said...

@Ron: Absolutely -- I've discussed problems with usability testing before. If the test asks the question "Can users accomplish X?" (as most usability tests do), then the results are pretty worthless. Accomplishment doesn't tell you if your system is good, only if users can accomplish a task.

BTW, in your last sentence, you mean "usability testing" not "UX", right? UX is the process of crafting an experience that works for users. This is always a good thing, to the extent that you can get there. The problem is with specific methods that don't work or are applied incorrectly.