CtRL-Sim: Reactive and Controllable Driving Agents with Offline
  Reinforcement Learning
Luke Rowe Roger Girgis Anthony Gosselin Bruno Carrez Florian Golemo Felix Heide Liam Paull Christopher Pal
Evaluating autonomous vehicle stacks (AVs) in simulation typically involves
replaying driving logs from real-world recorded traffic. However, agents
replayed from offline data do not react to the actions of the AV, and their
behaviour cannot be easily controlled to simulate counterfactual scenarios.
Existing approaches have attempted to address these shortcomings by proposing
methods that rely on heuristics or learned generative models of real-world data
but these approaches either lack realism or necessitate costly iterative
sampling procedures to control the generated behaviours. In this work, we take
an alternative approach and propose CtRL-Sim, a method that leverages
return-conditioned offline reinforcement learning within a physics-enhanced
Nocturne simulator to efficiently generate reactive and controllable traffic
agents. Specifically, we process real-world driving data through the Nocturne
simulator to generate a diverse offline reinforcement learning dataset,
annotated with various reward terms. With this dataset, we train a
return-conditioned multi-agent behaviour model that allows for fine-grained
manipulation of agent behaviours by modifying the desired returns for the
various reward components. This capability enables the generation of a wide
range of driving behaviours beyond the scope of the initial dataset, including
those representing adversarial behaviours. We demonstrate that CtRL-Sim can
efficiently generate diverse and realistic safety-critical scenarios while
providing fine-grained control over agent behaviours. Further, we show that
fine-tuning our model on simulated safety-critical scenarios generated by our
model enhances this controllability.