SETA: Semantic-Aware Token Augmentation for Domain Generalization
Jintao Guo Lei Qi Yinghuan Shi Yang Gao
Domain generalization (DG) aims to enhance the model robustness against
domain shifts without accessing target domains. A prevalent category of methods
for DG is data augmentation, which focuses on generating virtual samples to
simulate domain shifts. However, existing augmentation techniques in DG are
mainly tailored for convolutional neural networks (CNNs), with limited
exploration in token-based architectures, i.e., vision transformer (ViT) and
multi-layer perceptrons (MLP) models. In this paper, we study the impact of
prior CNN-based augmentation methods on token-based models, revealing their
performance is suboptimal due to the lack of incentivizing the model to learn
holistic shape information. To tackle the issue, we propose the SEmantic-aware
Token Augmentation (SETA) method. SETA transforms token features by perturbing
local edge cues while preserving global shape features, thereby enhancing the
model learning of shape information. To further enhance the generalization
ability of the model, we introduce two stylized variants of our method combined
with two state-of-the-art style augmentation methods in DG. We provide a
theoretical insight into our method, demonstrating its effectiveness in
reducing the generalization risk bound. Comprehensive experiments on five
benchmarks prove that our method achieves SOTA performances across various ViT
and MLP architectures. Our code is available at
https://github.com/lingeringlight/SETA.