Probing Intersectional Biases in Vision-Language Models with
  Counterfactual Examples
Phillip Howard Avinash Madasu Tiep Le Gustavo Lujan Moreno Vasudev Lal
While vision-language models (VLMs) have achieved remarkable performance
improvements recently, there is growing evidence that these models also posses
harmful biases with respect to social attributes such as gender and race. Prior
studies have primarily focused on probing such bias attributes individually
while ignoring biases associated with intersections between social attributes.
This could be due to the difficulty of collecting an exhaustive set of
image-text pairs for various combinations of social attributes from existing
datasets. To address this challenge, we employ text-to-image diffusion models
to produce counterfactual examples for probing intserctional social biases at
scale. Our approach utilizes Stable Diffusion with cross attention control to
produce sets of counterfactual image-text pairs that are highly similar in
their depiction of a subject (e.g., a given occupation) while differing only in
their depiction of intersectional social attributes (e.g., race & gender). We
conduct extensive experiments using our generated dataset which reveal the
intersectional social biases present in state-of-the-art VLMs.