Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias
Sierra Wyllie Ilia Shumailov Nicolas Papernot
Model-induced distribution shifts (MIDS) occur as previous model outputs
pollute new model training sets over generations of models. This is known as
model collapse in the case of generative models, and performative prediction or
unfairness feedback loops for supervised models. When a model induces a
distribution shift, it also encodes its mistakes, biases, and unfairnesses into
the ground truth of its data ecosystem. We introduce a framework that allows us
to track multiple MIDS over many generations, finding that they can lead to
loss in performance, fairness, and minoritized group representation, even in
initially unbiased datasets. Despite these negative consequences, we identify
how models might be used for positive, intentional, interventions in their data
ecosystems, providing redress for historical discrimination through a framework
called algorithmic reparation (AR). We simulate AR interventions by curating
representative training batches for stochastic gradient descent to demonstrate
how AR can improve upon the unfairnesses of models and data ecosystems subject
to other MIDS. Our work takes an important step towards identifying,
mitigating, and taking accountability for the unfair feedback loops enabled by
the idea that ML systems are inherently neutral and objective.