Large Language Model-Aware In-Context Learning for Code Generation
Jia Li Ge Li Chongyang Tao Jia Li Huangzhao Zhang Fang Liu Zhi Jin
Large language models (LLMs) have shown impressive in-context learning (ICL)
ability in code generation. LLMs take a prompt consisting of requirement-code
examples and a new requirement as input, and output new programs. Existing
studies have found that ICL is highly dominated by the examples and thus arises
research on example selection. However, existing approaches randomly select
examples or only consider the textual similarity of requirements to retrieve,
leading to sub-optimal performance. In this paper, we propose a novel
learning-based selection approach named LAIL (LLM-Aware In-context Learning)
for code generation. Given a candidate example, we exploit LLMs themselves to
estimate it by considering the generation probabilities of ground-truth
programs given a requirement and the example. We then label candidate examples
as positive or negative through the probability feedback. Based on the labeled
data, we import a contrastive learning objective to train an effective
retriever that acquires the preference of LLMs in code generation. We apply
LAIL to three LLMs and evaluate it on three representative datasets (e.g.,
MBJP, MBPP, and MBCPP). LATA outperforms the state-of-the-art baselines by
11.58%, 6.89%, and 5.07% on CodeGen, and 4.38%, 2.85%, and 2.74% on GPT-3.5 in
terms of Pass@1, respectively.