A Classification of Feedback Loops and Their Relation to Biases in
  Automated Decision-Making Systems
Nicolò Pagan Joachim Baumann Ezzat Elokda Giulia De Pasquale Saverio Bolognani Anikó Hannák
Prediction-based decision-making systems are becoming increasingly prevalent
in various domains. Previous studies have demonstrated that such systems are
vulnerable to runaway feedback loops, e.g., when police are repeatedly sent
back to the same neighborhoods regardless of the actual rate of criminal
activity, which exacerbate existing biases. In practice, the automated
decisions have dynamic feedback effects on the system itself that can
perpetuate over time, making it difficult for short-sighted design choices to
control the system's evolution. While researchers started proposing longer-term
solutions to prevent adverse outcomes (such as bias towards certain groups),
these interventions largely depend on ad hoc modeling assumptions and a
rigorous theoretical understanding of the feedback dynamics in ML-based
decision-making systems is currently missing. In this paper, we use the
language of dynamical systems theory, a branch of applied mathematics that
deals with the analysis of the interconnection of systems with dynamic
behaviors, to rigorously classify the different types of feedback loops in the
ML-based decision-making pipeline. By reviewing existing scholarly work, we
show that this classification covers many examples discussed in the algorithmic
fairness community, thereby providing a unifying and principled framework to
study feedback loops. By qualitative analysis, and through a simulation example
of recommender systems, we show which specific types of ML biases are affected
by each type of feedback loop. We find that the existence of feedback loops in
the ML-based decision-making pipeline can perpetuate, reinforce, or even reduce
ML biases.