ToViLaG: Your Visual-Language Generative Model is Also An Evildoer
Xinpeng Wang Xiaoyuan Yi Han Jiang Shanlin Zhou Zhihua Wei Xing Xie
Warning: this paper includes model outputs showing offensive content. Recent
large-scale Visual-Language Generative Models (VLGMs) have achieved
unprecedented improvement in multimodal image/text generation. However, these
models might also generate toxic content, e.g., offensive text and pornography
images, raising significant ethical risks. Despite exhaustive studies on toxic
degeneration of language models, this problem remains largely unexplored within
the context of visual-language generation. This work delves into the propensity
for toxicity generation and susceptibility to toxic data across various VLGMs.
For this purpose, we built ToViLaG, a dataset comprising 32K
co-toxic/mono-toxic text-image pairs and 1K innocuous but evocative text that
tends to stimulate toxicity. Furthermore, we propose WInToRe, a novel toxicity
metric tailored to visual-language generation, which theoretically reflects
different aspects of toxicity considering both input and output. On such a
basis, we benchmarked the toxicity of a diverse spectrum of VLGMs and
discovered that some models do more evil than expected while some are more
vulnerable to infection, underscoring the necessity of VLGMs detoxification.
Therefore, we develop an innovative bottleneck-based detoxification method. Our
method could reduce toxicity while maintaining comparable generation quality,
providing a promising initial solution to this line of research.