Resolving Crash Bugs via Large Language Models: An Empirical Study
Xueying Du Mingwei Liu Juntao Li Hanlin Wang Xin Peng Yiling Lou
Crash bugs cause unexpected program behaviors or even termination, requiring
high-priority resolution. However, manually resolving crash bugs is challenging
and labor-intensive, and researchers have proposed various techniques for their
automated localization and repair. ChatGPT, a recent large language model
(LLM), has garnered significant attention due to its exceptional performance
across various domains. This work performs the first investigation into
ChatGPT's capability in resolve real-world crash bugs, focusing on its
effectiveness in both localizing and repairing code-related and
environment-related crash bugs. Specifically, we initially assess ChatGPT's
fundamental ability to resolve crash bugs with basic prompts in a single
iteration. We observe that ChatGPT performs better at resolving code-related
crash bugs compared to environment-related ones, and its primary challenge in
resolution lies in inaccurate localization. Additionally, we explore ChatGPT's
potential with various advanced prompts. Furthermore, by stimulating ChatGPT's
self-planning, it methodically investigates each potential crash-causing
environmental factor through proactive inquiry, ultimately identifying the root
cause of the crash. Based on our findings, we propose IntDiagSolver, an
interaction methodology designed to facilitate precise crash bug resolution
through continuous interaction with LLMs. Evaluating IntDiagSolver on multiple
LLMs reveals consistent enhancement in the accuracy of crash bug resolution,
including ChatGPT, Claude, and CodeLlama.