Improving Diversity in Zero-Shot GAN Adaptation with Semantic Variations
Seogkyu Jeon Bei Liu Pilhyeon Lee Kibeom Hong Jianlong Fu Hyeran Byun
Training deep generative models usually requires a large amount of data. To
alleviate the data collection cost, the task of zero-shot GAN adaptation aims
to reuse well-trained generators to synthesize images of an unseen target
domain without any further training samples. Due to the data absence, the
textual description of the target domain and the vision-language models, e.g.,
CLIP, are utilized to effectively guide the generator. However, with only a
single representative text feature instead of real images, the synthesized
images gradually lose diversity as the model is optimized, which is also known
as mode collapse. To tackle the problem, we propose a novel method to find
semantic variations of the target text in the CLIP space. Specifically, we
explore diverse semantic variations based on the informative text feature of
the target domain while regularizing the uncontrolled deviation of the semantic
information. With the obtained variations, we design a novel directional moment
loss that matches the first and second moments of image and text direction
distributions. Moreover, we introduce elastic weight consolidation and a
relation consistency loss to effectively preserve valuable content information
from the source domain, e.g., appearances. Through extensive experiments, we
demonstrate the efficacy of the proposed methods in ensuring sample diversity
in various scenarios of zero-shot GAN adaptation. We also conduct ablation
studies to validate the effect of each proposed component. Notably, our model
achieves a new state-of-the-art on zero-shot GAN adaptation in terms of both
diversity and quality.