Conditional Text Image Generation with Diffusion Models
Yuanzhi Zhu Zhaohai Li Tianwei Wang Mengchao He Cong Yao
Current text recognition systems, including those for handwritten scripts and
scene text, have relied heavily on image synthesis and augmentation, since it
is difficult to realize real-world complexity and diversity through collecting
and annotating enough real text images. In this paper, we explore the problem
of text image generation, by taking advantage of the powerful abilities of
Diffusion Models in generating photo-realistic and diverse image samples with
given conditions, and propose a method called Conditional Text Image Generation
with Diffusion Models (CTIG-DM for short). To conform to the characteristics of
text images, we devise three conditions: image condition, text condition, and
style condition, which can be used to control the attributes, contents, and
styles of the samples in the image generation process. Specifically, four text
image generation modes, namely: (1) synthesis mode, (2) augmentation mode, (3)
recovery mode, and (4) imitation mode, can be derived by combining and
configuring these three conditions. Extensive experiments on both handwritten
and scene text demonstrate that the proposed CTIG-DM is able to produce image
samples that simulate real-world complexity and diversity, and thus can boost
the performance of existing text recognizers. Besides, CTIG-DM shows its
appealing potential in domain adaptation and generating images containing
Out-Of-Vocabulary (OOV) words.