RealGen: Retrieval Augmented Generation for Controllable Traffic
  Scenarios
Wenhao Ding Yulong Cao Ding Zhao Chaowei Xiao Marco Pavone
Simulation plays a crucial role in the development of autonomous vehicles
(AVs) due to the potential risks associated with real-world testing. Although
significant progress has been made in the visual aspects of simulators,
generating complex behavior among agents remains a formidable challenge. It is
not only imperative to ensure realism in the scenarios generated but also
essential to incorporate preferences and conditions to facilitate controllable
generation for AV training and evaluation. Traditional methods, mainly relying
on memorizing the distribution of training datasets, often fall short in
generating unseen scenarios. Inspired by the success of retrieval augmented
generation in large language models, we present RealGen, a novel
retrieval-based in-context learning framework for traffic scenario generation.
RealGen synthesizes new scenarios by combining behaviors from multiple
retrieved examples in a gradient-free way, which may originate from templates
or tagged scenarios. This in-context learning framework endows versatile
generative capabilities, including the ability to edit scenarios, compose
various behaviors, and produce critical scenarios. Evaluations show that
RealGen offers considerable flexibility and controllability, marking a new
direction in the field of controllable traffic scenario generation. Check our
project website for more information: https://realgen.github.io.