Differential Privacy, Linguistic Fairness, and Training Data Influence:
  Impossibility and Possibility Theorems for Multilingual Language Models
Phillip Rust Anders Søgaard
Language models such as mBERT, XLM-R, and BLOOM aim to achieve multilingual
generalization or compression to facilitate transfer to a large number of
(potentially unseen) languages. However, these models should ideally also be
private, linguistically fair, and transparent, by relating their predictions to
training data. Can these requirements be simultaneously satisfied? We show that
multilingual compression and linguistic fairness are compatible with
differential privacy, but that differential privacy is at odds with training
data influence sparsity, an objective for transparency. We further present a
series of experiments on two common NLP tasks and evaluate multilingual
compression and training data influence sparsity under different privacy
guarantees, exploring these trade-offs in more detail. Our results suggest that
we need to develop ways to jointly optimize for these objectives in order to
find practical trade-offs.