Uncertainty, Calibration, and Membership Inference Attacks: An
  Information-Theoretic Perspective
Meiyi Zhu Caili Guo Chunyan Feng Osvaldo Simeone
In a membership inference attack (MIA), an attacker exploits the
overconfidence exhibited by typical machine learning models to determine
whether a specific data point was used to train a target model. In this paper,
we analyze the performance of the state-of-the-art likelihood ratio attack
(LiRA) within an information-theoretical framework that allows the
investigation of the impact of the aleatoric uncertainty in the true data
generation process, of the epistemic uncertainty caused by a limited training
data set, and of the calibration level of the target model. We compare three
different settings, in which the attacker receives decreasingly informative
feedback from the target model: confidence vector (CV) disclosure, in which the
output probability vector is released; true label confidence (TLC) disclosure,
in which only the probability assigned to the true label is made available by
the model; and decision set (DS) disclosure, in which an adaptive prediction
set is produced as in conformal prediction. We derive bounds on the advantage
of an MIA adversary with the aim of offering insights into the impact of
uncertainty and calibration on the effectiveness of MIAs. Simulation results
demonstrate that the derived analytical bounds predict well the effectiveness
of MIAs.