Image Captions are Natural Prompts for Text-to-Image Models
Shiye Lei Hao Chen Sen Zhang Bo Zhao Dacheng Tao
With the rapid development of Artificial Intelligence Generated Content
(AIGC), it has become common practice in many learning tasks to train or
fine-tune large models on synthetic data due to the data-scarcity and privacy
leakage problems. Albeit promising with unlimited data generation, owing to
massive and diverse information conveyed in real images, it is challenging for
text-to-image generative models to synthesize informative training data with
hand-crafted prompts, which usually leads to inferior generalization
performance when training downstream models. In this paper, we theoretically
analyze the relationship between the training effect of synthetic data and the
synthetic data distribution induced by prompts. Then we correspondingly propose
a simple yet effective method that prompts text-to-image generative models to
synthesize more informative and diverse training data. Specifically, we caption
each real image with the advanced captioning model to obtain informative and
faithful prompts that extract class-relevant information and clarify the
polysemy of class names. The image captions and class names are concatenated to
prompt generative models for training image synthesis. Extensive experiments on
ImageNette, ImageNet-100, and ImageNet-1K verify that our method significantly
improves the performance of models trained on synthetic training data, i.e.,
10% classification accuracy improvements on average.