The Program Testing Ability of Large Language Models for Code
Weimin Xiong Yiwen Guo Hao Chen
Recent development of large language models (LLMs) for code like CodeX and
CodeT5+ demonstrates tremendous promise in achieving code intelligence. Their
ability of synthesizing code that completes a program for performing a
pre-defined task has been intensively tested and verified on benchmark datasets
including HumanEval and MBPP. Yet, evaluation of these LLMs from more
perspectives (than just program synthesis) is also anticipated, considering
their broad scope of applications in software engineering. In this paper, we
explore the ability of LLMs for testing programs/code. By performing thorough
analyses of recent LLMs for code in program testing, we show a series of
intriguing properties of these models and demonstrate how program testing
ability of LLMs can be improved. Following recent work which utilizes generated
test cases to enhance program synthesis, we further leverage our findings in
improving the quality of the synthesized programs and show +11.77% and +4.22%
higher code pass rates on HumanEval+ comparing with the GPT-3.5-turbo baseline
and the recent state-of-the-art, respectively.