Coding Horror

programming and human factors

Not All Bugs Are Worth Fixing

triage-tag.png One thing that continually frustrates me when working with dedicated test teams is that, well, they find too many bugs.

Don't get me wrong. I want to be the first person to know about any bug that results in inconvenience for a user. But how do you distinguish between bugs that users are likely to encounter, and bugs that users will probably never see?

The first thing you do is take that list of bugs from the testers and have yourself a triage meeting:

The term "triage" was borrowed from medical triage where a doctor or nurse has to prioritize care for a large group of injured people. The main job of a software bug triage team is to decide which bugs need to be fixed (or conversely, which bugs we're willing to ship with).

Eric lists four questions that need to be answered during triage to decide whether a bug should be fixed or not:

  1. Severity: When this bug happens, how bad is the impact?
  2. Frequency: How often does this bug happen?
  3. Cost: How much effort would be required to fix this bug?
  4. Risk: What is the risk of fixing this bug?

Triage isn't exactly my idea of a good time. But you have to do it, because you'll always have far more bugs than you have development time. Nobody has the luxury of fixing all the bugs in their software.

Testers produce two kinds of bugs:

  1. A small subset of very serious bugs that everyone can immediately agree on. These are great. They're the kind of catches that make me thank my lucky stars that we have dedicated testers. You go, girl-slash-boy!
  2. Everything else. A vast, gray wasteland of pseudo-bugs that nobody can really agree on. Is it an inconvenience for the user? Would users really do things this way? Would a user ever run into this? Do we even care?

It's a clear win for the bugs everyone agrees on. That's usually about ten to twenty percent of the bug list in my experience. But for everything else, there's a serious problem: testers aren't real users. I'd give a bug from a customer ten times the weight of a bug reported by a tester.

The source of the bug is just one factor to consider. Bug triage isn't a science. It's highly subjective and totally dependent on the specifics of your application. In Bugs Are a Business Decision, Jan Goyvaerts describes how different triage can be for applications at each end of that spectrum:


The avionics software is the software that flies the plane. No, the pilots don't fly the plane, the flight computer does. How many bugs would you tolerate in the avionics software? How many do you think Boeing left unfixed? How many people have ever been killed by software bugs in modern airliners? Zero. A flawed flight computer would immediately ground all 747s worldwide. Boeing would not recover.

The in-flight entertainment system is a completely different story. It's not essential to the plane. It only serves to make the passengers forget how uncomfortable those economy seats really are. If the entertainment system barfs all over itself, the cost is minimal. Passengers are already out of their money, and most will choose their next flight based on price and schedule rather than which movies are on those tiny screens, if any. I was actually quite pleased with Chine Airlines' system, which offered economy passengers individual screens and a choice of a dozen or so on-demand movies (i.e. each passenger can start viewing any movie at any time, and even pause and rewind). That is, until the system started acting up. It locked up a few times causing everybody's movie to pause for several minutes. Once, the crew had to reboot the whole thing. That silly Linux penguin mocked me for several minutes while the boot messages crept by. X11 showed off its X-shaped cursor right in the middle of the screen even longer. Judging from the crew's attitude about it, the reboot seemed like something that's part of their training.

Bugs also cost money to fix. In My Life as a Code Economist, Eric Sink outlines all the decisions that go into whether or not a bug gets fixed at his company:








To me, triage is about one thing: making life better for your users. And the best way to do that is to base your triage decisions on data from actual usage -- via exception reporting, user feedback, and beta testing. Otherwise, triage is just a bunch of developers and testers in a room, trying to guess what users might do.

Written by Jeff Atwood
