Sem-CS: Semantic CLIPStyler for Text-Based Image Style Transfer
Chanda Grover Kamra Indra Deep Mastan Debayan Gupta
CLIPStyler demonstrated image style transfer with realistic textures using
only a style text description (instead of requiring a reference style image).
However, the ground semantics of objects in the style transfer output is lost
due to style spill-over on salient and background objects (content mismatch) or
over-stylization. To solve this, we propose Semantic CLIPStyler (Sem-CS), that
performs semantic style transfer. Sem-CS first segments the content image into
salient and non-salient objects and then transfers artistic style based on a
given style text description. The semantic style transfer is achieved using
global foreground loss (for salient objects) and global background loss (for
non-salient objects). Our empirical results, including DISTS, NIMA and user
study scores, show that our proposed framework yields superior qualitative and
quantitative performance. Our code is available at
github.com/chandagrover/sem-cs.