EvoCodeBench: An Evolving Code Generation Benchmark Aligned with
  Real-World Code Repositories
Jia Li Ge Li Xuanming Zhang Yihong Dong Zhi Jin
How to evaluate Large Language Models (LLMs) in code generation is an open
question. Existing benchmarks demonstrate poor alignment with real-world code
repositories and are insufficient to evaluate the coding abilities of LLMs.
This paper proposes a new benchmark - EvoCodeBench to address the preceding
problems, which has three primary advances. (1) EvoCodeBench aligns with
real-world repositories in multiple dimensions, e.g., code distributions and
dependency distributions. (2) EvoCodeBench offers comprehensive annotations
(e.g., requirements, reference code, and reference dependencies), and robust
evaluation metrics (e.g., Pass@k and Recall@k). (3) EvoCodeBench is an evolving
benchmark to avoid data leakage. We build an automatic pipeline to update
EvoCodeBench from the latest repositories. We release the first version -
EvoCodeBench-2403, containing 275 samples from 25 real-world repositories.
Based on EvoCodeBench, we propose repository-level code generation and evaluate
10 popular LLMs (e.g., gpt-4, gpt-3.5, DeepSeek Coder, StarCoder 2, CodeLLaMa,
Gemma, and Qwen 1.5). Our experiments reveal the coding abilities of these LLMs
in real-world repositories. For example, the highest Pass@1 of gpt-4 only is
20.73% in our experiments. We also analyze failed cases and summarize the
shortcomings of existing LLMs in EvoCodeBench. We release EvoCodeBench, all
prompts, and LLMs' completions for further community analysis.