Exploring Generative AI for Sim2Real in Driving Data Synthesis
Haonan Zhao Yiting Wang Thomas Bashford-Rogers Valentina Donzella Kurt Debattista
Datasets are essential for training and testing vehicle perception
algorithms. However, the collection and annotation of real-world images is
time-consuming and expensive. Driving simulators offer a solution by
automatically generating various driving scenarios with corresponding
annotations, but the simulation-to-reality (Sim2Real) domain gap remains a
challenge. While most of the Generative Artificial Intelligence (AI) follows
the de facto Generative Adversarial Nets (GANs)-based methods, the recent
emerging diffusion probabilistic models have not been fully explored in
mitigating Sim2Real challenges for driving data synthesis. To explore the
performance, this paper applied three different generative AI methods to
leverage semantic label maps from a driving simulator as a bridge for the
creation of realistic datasets. A comparative analysis of these methods is
presented from the perspective of image quality and perception. New synthetic
datasets, which include driving images and auto-generated high-quality
annotations, are produced with low costs and high scene variability. The
experimental results show that although GAN-based methods are adept at
generating high-quality images when provided with manually annotated labels,
ControlNet produces synthetic datasets with fewer artefacts and more structural
fidelity when using simulator-generated labels. This suggests that the
diffusion-based approach may provide improved stability and an alternative
method for addressing Sim2Real challenges.