AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale
  Clinical Tool Learning
Qiao Jin Zhizheng Wang Yifan Yang Qingqing Zhu Donald Wright Thomas Huang W John Wilbur Zhe He Andrew Taylor Qingyu Chen Zhiyong Lu
Clinical calculators play a vital role in healthcare by offering accurate
evidence-based predictions for various purposes such as prognosis.
Nevertheless, their widespread utilization is frequently hindered by usability
challenges, poor dissemination, and restricted functionality. Augmenting large
language models with extensive collections of clinical calculators presents an
opportunity to overcome these obstacles and improve workflow efficiency, but
the scalability of the manual curation process poses a significant challenge.
In response, we introduce AgentMD, a novel language agent capable of curating
and applying clinical calculators across various clinical contexts. Using the
published literature, AgentMD has automatically curated a collection of 2,164
diverse clinical calculators with executable functions and structured
documentation, collectively named RiskCalcs. Manual evaluations show that
RiskCalcs tools achieve an accuracy of over 80% on three quality metrics. At
inference time, AgentMD can automatically select and apply the relevant
RiskCalcs tools given any patient description. On the newly established RiskQA
benchmark, AgentMD significantly outperforms chain-of-thought prompting with
GPT-4 (87.7% vs. 40.9% in accuracy). Additionally, we also applied AgentMD to
real-world clinical notes for analyzing both population-level and risk-level
patient characteristics. In summary, our study illustrates the utility of
language agents augmented with clinical calculators for healthcare analytics
and patient care.