ZoDi: Zero-Shot Domain Adaptation with Diffusion-Based Image Transfer
Hiroki Azuma Yusuke Matsui Atsuto Maki
Deep learning models achieve high accuracy in segmentation tasks among
others, yet domain shift often degrades the models' performance, which can be
critical in real-world scenarios where no target images are available. This
paper proposes a zero-shot domain adaptation method based on diffusion models,
called ZoDi, which is two-fold by the design: zero-shot image transfer and
model adaptation. First, we utilize an off-the-shelf diffusion model to
synthesize target-like images by transferring the domain of source images to
the target domain. In this we specifically try to maintain the layout and
content by utilising layout-to-image diffusion models with stochastic
inversion. Secondly, we train the model using both source images and
synthesized images with the original segmentation maps while maximizing the
feature similarity of images from the two domains to learn domain-robust
representations. Through experiments we show benefits of ZoDi in the task of
image segmentation over state-of-the-art methods. It is also more applicable
than existing CLIP-based methods because it assumes no specific backbone or
models, and it enables to estimate the model's performance without target
images by inspecting generated images. Our implementation will be publicly
available.