SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation
Zhixuan Liu Peter Schaldenbrand Beverley-Claire Okogwu Wenxuan Peng Youngsik Yun Andrew Hundt Jihie Kim Jean Oh
Accurate representation in media is known to improve the well-being of the
people who consume it. Generative image models trained on large web-crawled
datasets such as LAION are known to produce images with harmful stereotypes and
misrepresentations of cultures. We improve inclusive representation in
generated images by (1) engaging with communities to collect a culturally
representative dataset that we call the Cross-Cultural Understanding Benchmark
(CCUB) and (2) proposing a novel Self-Contrastive Fine-Tuning (SCoFT) method
that leverages the model's known biases to self-improve. SCoFT is designed to
prevent overfitting on small datasets, encode only high-level information from
the data, and shift the generated distribution away from misrepresentations
encoded in a pretrained model. Our user study conducted on 51 participants from
5 different countries based on their self-selected national cultural
affiliation shows that fine-tuning on CCUB consistently generates images with
higher cultural relevance and fewer stereotypes when compared to the Stable
Diffusion baseline, which is further improved with our SCoFT technique.