On the Cultural Gap in Text-to-Image Generation
Bingshuai Liu Longyue Wang Chenyang Lyu Yong Zhang Jinsong Su Shuming Shi Zhaopeng Tu
One challenge in text-to-image (T2I) generation is the inadvertent reflection
of culture gaps present in the training data, which signifies the disparity in
generated image quality when the cultural elements of the input text are rarely
collected in the training set. Although various T2I models have shown
impressive but arbitrary examples, there is no benchmark to systematically
evaluate a T2I model's ability to generate cross-cultural images. To bridge the
gap, we propose a Challenging Cross-Cultural (C3) benchmark with comprehensive
evaluation criteria, which can assess how well-suited a model is to a target
culture. By analyzing the flawed images generated by the Stable Diffusion model
on the C3 benchmark, we find that the model often fails to generate certain
cultural objects. Accordingly, we propose a novel multi-modal metric that
considers object-text alignment to filter the fine-tuning data in the target
culture, which is used to fine-tune a T2I model to improve cross-cultural
generation. Experimental results show that our multi-modal metric provides
stronger data selection performance on the C3 benchmark than existing metrics,
in which the object-text alignment is crucial. We release the benchmark, data,
code, and generated images to facilitate future research on culturally diverse
T2I generation (https://github.com/longyuewangdcu/C3-Bench).