Scenimefy: Learning to Craft Anime Scene via Semi-Supervised
  Image-to-Image Translation
Yuxin Jiang Liming Jiang Shuai Yang Chen Change Loy
Automatic high-quality rendering of anime scenes from complex real-world
images is of significant practical value. The challenges of this task lie in
the complexity of the scenes, the unique features of anime style, and the lack
of high-quality datasets to bridge the domain gap. Despite promising attempts,
previous efforts are still incompetent in achieving satisfactory results with
consistent semantic preservation, evident stylization, and fine details. In
this study, we propose Scenimefy, a novel semi-supervised image-to-image
translation framework that addresses these challenges. Our approach guides the
learning with structure-consistent pseudo paired data, simplifying the pure
unsupervised setting. The pseudo data are derived uniquely from a
semantic-constrained StyleGAN leveraging rich model priors like CLIP. We
further apply segmentation-guided data selection to obtain high-quality pseudo
supervision. A patch-wise contrastive style loss is introduced to improve
stylization and fine details. Besides, we contribute a high-resolution anime
scene dataset to facilitate future research. Our extensive experiments
demonstrate the superiority of our method over state-of-the-art baselines in
terms of both perceptual quality and quantitative performance.