A Unified Framework and Dataset for Assessing Gender Bias in
  Vision-Language Models
Ashutosh Sathe Prachi Jain Sunayana Sitaram
Large vision-language models (VLMs) are widely getting adopted in industry
and academia. In this work we build a unified framework to systematically
evaluate gender-profession bias in VLMs. Our evaluation encompasses all
supported inference modes of the recent VLMs, including image-to-text,
text-to-text, text-to-image, and image-to-image. We construct a synthetic,
high-quality dataset of text and images that blurs gender distinctions across
professional actions to benchmark gender bias. In our benchmarking of recent
vision-language models (VLMs), we observe that different input-output
modalities result in distinct bias magnitudes and directions. We hope our work
will help guide future progress in improving VLMs to learn socially unbiased
representations. We will release our data and code.