How ChatGPT is Solving Vulnerability Management Problem
Peiyu Liu Junming Liu Lirong Fu Kangjie Lu Yifan Xia Xuhong Zhang Wenzhi Chen Haiqin Weng Shouling Ji Wenhai Wang
Recently, ChatGPT has attracted great attention from the code analysis
domain. Prior works show that ChatGPT has the capabilities of processing
foundational code analysis tasks, such as abstract syntax tree generation,
which indicates the potential of using ChatGPT to comprehend code syntax and
static behaviors. However, it is unclear whether ChatGPT can complete more
complicated real-world vulnerability management tasks, such as the prediction
of security relevance and patch correctness, which require an all-encompassing
understanding of various aspects, including code syntax, program semantics, and
related manual comments.
  In this paper, we explore ChatGPT's capabilities on 6 tasks involving the
complete vulnerability management process with a large-scale dataset containing
78,445 samples. For each task, we compare ChatGPT against SOTA approaches,
investigate the impact of different prompts, and explore the difficulties. The
results suggest promising potential in leveraging ChatGPT to assist
vulnerability management. One notable example is ChatGPT's proficiency in tasks
like generating titles for software bug reports. Furthermore, our findings
reveal the difficulties encountered by ChatGPT and shed light on promising
future directions. For instance, directly providing random demonstration
examples in the prompt cannot consistently guarantee good performance in
vulnerability management. By contrast, leveraging ChatGPT in a self-heuristic
way -- extracting expertise from demonstration examples itself and integrating
the extracted expertise in the prompt is a promising research direction.
Besides, ChatGPT may misunderstand and misuse the information in the prompt.
Consequently, effectively guiding ChatGPT to focus on helpful information
rather than the irrelevant content is still an open problem.