Symptom Checkers: Triage of the Future?

Wide range of accuracy, but more help than harm

  • register today


    sign up

Action Points

  • Note that this study examining over 20 "symptom checkers" found widely varying diagnostic accuracy.
  • Though emergent situations were identified a majority of the time, less emergent cases were often triaged inappropriately.

Symptom checker software appropriately triaged emergent patient cases 80% of the time, but that accuracy fell off for non-emergent care (55%) and self care (34%) scenarios in a standardized test, researchers reported.

This level of accuracy varied widely among the 23 checkers evaluated with 45 standardized patient vignettes, but the top checker gave appropriate advice 84% of the time, Ateev Mehrotra, MD, MPH, of Beth Israel Deaconess Medical Center in Boston, and colleagues, reported in the BMJ.

Symptom checkers -- online tools to help individuals self-diagnose and triage -- are often hosted by medical schools, hospital systems, insurance companies, and, in the case of the U.K.'s National Health Service, government agencies.

The software works by assigning a diagnosis and giving potentially helpful directives or information based on the symptoms selected or entered with a computerized algorithm. The algorithms vary by checker; some use branching logic, Bayesian inference, or other methods.

Symptom checkers have become commonplace. According to the authors, one symptom checker, iTriage, reports 50 million users per year. Even the American Academy of Pediatrics and the Mayo Clinic have launched their own symptom checkers.

Mehrotra's team spent 5 months in 2014 looking for free, publicly available, English language symptom checkers. After exclusions, they identified 23 symptom checkers from either the U.S., the U.K., the Netherlands, and Poland.

Only 11 provided both diagnoses and triage advice; eight gave diagnoses only, and four gave triage only. The researchers tested each checker with 45 standardized patient vignettes from three different categories: emergent care required, non-emergent care appropriate, and self care reasonable.

The researchers used a mix of common and uncommon conditions based on prevalence of the diagnosis in U.S. ambulatory visits.

Some of the symptom checkers produced a high number of potential diagnoses for several of the vignettes, two gave as many as 99, so the researchers classified a tracker as providing a "diagnosis" only if the the accurate diagnosis was in the top 20 listed, creating a potential 770 patient evaluations.

Mehrotra's team then categorized all of the triage advice into three groups: emergent, including specifics about whether to call an ambulance or go to the emergency department, non-emergent, meaning to go to a primary care provider/specialist or urgent care clinic, or self care. This yielded a total of 532 patient evaluations.

On average, the symptom checkers listed an accurate diagnosis in the top 20 suggestions 58% of the time, and appropriate triage advice 57% of the time. The top-performing symptom checker was accurate 84% of the time.

In cases requiring emergent care, symptom checkers were accurate 80% of the time, but in cases of self care, they only provided accurate triage advice 34% of the time. In cases of non-emergent care, the checker was accurate 55% of the time.

Mehrotra's group calculated that overall performance on appropriate triage advice across all of the symptom checkers ranged from 33% to 78%, and noted that the symptom checkers were clearly inferior to seeing a physician, as they cited research suggesting physician diagnostic accuracy ranges from 85-90%.

However, they wrote, comparing in-person visits to symptom checkers isn't the point, as symptom checkers are likely being used for "quick and accessible guidance." And that appropriate triage as opposed to actual diagnostic accuracy is the key assessment of symptom checkers.

"Distinguishing between Rocky Mountain spotted fever and meningitis may be less important than ensuring patients seek emergent care," they wrote.

But, the authors maintained, symptom checkers were a better alternative to patients simply entering symptoms into search engines. The more accurate comparison, they wrote, would be a triage telephone line.

"[M]any nurse phone triage lines use the same underlying clinical logic as the symptom checkers evaluated in this study," the authors wrote. And, they noted, some health plan triage lines have nurses using symptom checker software on the phone.

The researchers ultimately concluded that symptom checkers were a better option than not seeking any medical advice at all or using basic search engines as diagnostic tools.

Study limitations include strict inclusion criteria leading to a small sample, and the use of vignettes that contained language that lay-persons and patients might not use when describing symptoms.

None of the authors reported any relevant financial relationships with industry. One of the tools evaluated was generated by Harvard Medical School, but none of the authors were involved in that project.


last updated

take posttest 2 comments