Intra- & Extra-Source Exemplar-Based Style Synthesis for Improved Domain
  Generalization
Yumeng Li Dan Zhang Margret Keuper Anna Khoreva
The generalization with respect to domain shifts, as they frequently appear
in applications such as autonomous driving, is one of the remaining big
challenges for deep learning models. Therefore, we propose an exemplar-based
style synthesis pipeline to improve domain generalization in semantic
segmentation. Our method is based on a novel masked noise encoder for StyleGAN2
inversion. The model learns to faithfully reconstruct the image, preserving its
semantic layout through noise prediction. Using the proposed masked noise
encoder to randomize style and content combinations in the training set, i.e.,
intra-source style augmentation (ISSA) effectively increases the diversity of
training data and reduces spurious correlation. As a result, we achieve up to
$12.4\%$ mIoU improvements on driving-scene semantic segmentation under
different types of data shifts, i.e., changing geographic locations, adverse
weather conditions, and day to night. ISSA is model-agnostic and
straightforwardly applicable with CNNs and Transformers. It is also
complementary to other domain generalization techniques, e.g., it improves the
recent state-of-the-art solution RobustNet by $3\%$ mIoU in Cityscapes to Dark
Z\"urich. In addition, we demonstrate the strong plug-n-play ability of the
proposed style synthesis pipeline, which is readily usable for extra-source
exemplars e.g., web-crawled images, without any retraining or fine-tuning.
Moreover, we study a new use case to indicate neural network's generalization
capability by building a stylized proxy validation set. This application has
significant practical sense for selecting models to be deployed in the
open-world environment. Our code is available at
\url{https://github.com/boschresearch/ISSA}.