CodeS: Natural Language to Code Repository via Multi-Layer Sketch
Daoguang Zan Ailun Yu Wei Liu Dong Chen Bo Shen Wei Li Yafen Yao Yongshun Gong Xiaolin Chen Bei Guan Zhiguang Yang Yongji Wang Qianxiang Wang Lizhen Cui
The impressive performance of large language models (LLMs) on code-related
tasks has shown the potential of fully automated software development. In light
of this, we introduce a new software engineering task, namely Natural Language
to code Repository (NL2Repo). This task aims to generate an entire code
repository from its natural language requirements. To address this task, we
propose a simple yet effective framework CodeS, which decomposes NL2Repo into
multiple sub-tasks by a multi-layer sketch. Specifically, CodeS includes three
modules: RepoSketcher, FileSketcher, and SketchFiller. RepoSketcher first
generates a repository's directory structure for given requirements;
FileSketcher then generates a file sketch for each file in the generated
structure; SketchFiller finally fills in the details for each function in the
generated file sketch. To rigorously assess CodeS on the NL2Repo task, we carry
out evaluations through both automated benchmarking and manual feedback
analysis. For benchmark-based evaluation, we craft a repository-oriented
benchmark, SketchEval, and design an evaluation metric, SketchBLEU. For
feedback-based evaluation, we develop a VSCode plugin for CodeS and engage 30
participants in conducting empirical studies. Extensive experiments prove the
effectiveness and practicality of CodeS on the NL2Repo task.