Exploring LLM-based Agents for Root Cause Analysis
Devjeet Roy Xuchao Zhang Rashi Bhave Chetan Bansal Pedro Las-Casas Rodrigo Fonseca Saravan Rajmohan
The growing complexity of cloud based software systems has resulted in
incident management becoming an integral part of the software development
lifecycle. Root cause analysis (RCA), a critical part of the incident
management process, is a demanding task for on-call engineers, requiring deep
domain knowledge and extensive experience with a team's specific services.
Automation of RCA can result in significant savings of time, and ease the
burden of incident management on on-call engineers. Recently, researchers have
utilized Large Language Models (LLMs) to perform RCA, and have demonstrated
promising results. However, these approaches are not able to dynamically
collect additional diagnostic information such as incident related logs,
metrics or databases, severely restricting their ability to diagnose root
causes. In this work, we explore the use of LLM based agents for RCA to address
this limitation. We present a thorough empirical evaluation of a ReAct agent
equipped with retrieval tools, on an out-of-distribution dataset of production
incidents collected at Microsoft. Results show that ReAct performs
competitively with strong retrieval and reasoning baselines, but with highly
increased factual accuracy. We then extend this evaluation by incorporating
discussions associated with incident reports as additional inputs for the
models, which surprisingly does not yield significant performance improvements.
Lastly, we conduct a case study with a team at Microsoft to equip the ReAct
agent with tools that give it access to external diagnostic services that are
used by the team for manual RCA. Our results show how agents can overcome the
limitations of prior work, and practical considerations for implementing such a
system in practice.