Placing Objects in Context via Inpainting for Out-of-distribution
  Segmentation
Pau de Jorge Riccardo Volpi Puneet K. Dokania Philip H. S. Torr Gregory Rogez
When deploying a semantic segmentation model into the real world, it will
inevitably be confronted with semantic classes unseen during training. Thus, to
safely deploy such systems, it is crucial to accurately evaluate and improve
their anomaly segmentation capabilities. However, acquiring and labelling
semantic segmentation data is expensive and unanticipated conditions are
long-tail and potentially hazardous. Indeed, existing anomaly segmentation
datasets capture a limited number of anomalies, lack realism or have strong
domain shifts. In this paper, we propose the Placing Objects in Context (POC)
pipeline to realistically add any object into any image via diffusion models.
POC can be used to easily extend any dataset with an arbitrary number of
objects. In our experiments, we present different anomaly segmentation datasets
based on POC-generated data and show that POC can improve the performance of
recent state-of-the-art anomaly fine-tuning methods in several standardized
benchmarks. POC is also effective to learn new classes. For example, we use it
to edit Cityscapes samples by adding a subset of Pascal classes and show that
models trained on such data achieve comparable performance to the
Pascal-trained baseline. This corroborates the low sim-to-real gap of models
trained on POC-generated images.