Style Generation: Image Synthesis based on Coarsely Matched Texts
Mengyao Cui Zhe Zhu Shao-Ping Lu Yulu Yang
Previous text-to-image synthesis algorithms typically use explicit textual
instructions to generate/manipulate images accurately, but they have difficulty
adapting to guidance in the form of coarsely matched texts. In this work, we
attempt to stylize an input image using such coarsely matched text as guidance.
To tackle this new problem, we introduce a novel task called text-based style
generation and propose a two-stage generative adversarial network: the first
stage generates the overall image style with a sentence feature, and the second
stage refines the generated style with a synthetic feature, which is produced
by a multi-modality style synthesis module. We re-filter one existing dataset
and collect a new dataset for the task. Extensive experiments and ablation
studies are conducted to validate our framework. The practical potential of our
work is demonstrated by various applications such as text-image alignment and
story visualization. Our datasets are published at
https://www.kaggle.com/datasets/mengyaocui/style-generation.