Can Large Language Models Find And Fix Vulnerable Software?
David Noever
In this study, we evaluated the capability of Large Language Models (LLMs),
particularly OpenAI's GPT-4, in detecting software vulnerabilities, comparing
their performance against traditional static code analyzers like Snyk and
Fortify. Our analysis covered numerous repositories, including those from NASA
and the Department of Defense. GPT-4 identified approximately four times the
vulnerabilities than its counterparts. Furthermore, it provided viable fixes
for each vulnerability, demonstrating a low rate of false positives. Our tests
encompassed 129 code samples across eight programming languages, revealing the
highest vulnerabilities in PHP and JavaScript. GPT-4's code corrections led to
a 90% reduction in vulnerabilities, requiring only an 11% increase in code
lines. A critical insight was LLMs' ability to self-audit, suggesting fixes for
their identified vulnerabilities and underscoring their precision. Future
research should explore system-level vulnerabilities and integrate multiple
static code analyzers for a holistic perspective on LLMs' potential.