Privacy Side Channels in Machine Learning Systems
Edoardo Debenedetti Giorgio Severi Nicholas Carlini Christopher A. Choquette-Choo Matthew Jagielski Milad Nasr Eric Wallace Florian Tramèr
Most current approaches for protecting privacy in machine learning (ML)
assume that models exist in a vacuum, when in reality, ML models are part of
larger systems that include components for training data filtering, output
monitoring, and more. In this work, we introduce privacy side channels: attacks
that exploit these system-level components to extract private information at
far higher rates than is otherwise possible for standalone models. We propose
four categories of side channels that span the entire ML lifecycle (training
data filtering, input preprocessing, output post-processing, and query
filtering) and allow for either enhanced membership inference attacks or even
novel threats such as extracting users' test queries. For example, we show that
deduplicating training data before applying differentially-private training
creates a side-channel that completely invalidates any provable privacy
guarantees. Moreover, we show that systems which block language models from
regenerating training data can be exploited to allow exact reconstruction of
private keys contained in the training set -- even if the model did not
memorize these keys. Taken together, our results demonstrate the need for a
holistic, end-to-end privacy analysis of machine learning.