Hot PATE: Private Aggregation of Distributions for Diverse Task
Edith Cohen Xin Lyu Jelani Nelson Tamas Sarlos Uri Stemmer
The Private Aggregation of Teacher Ensembles (PATE)
framework~\cite{PapernotAEGT:ICLR2017} is a versatile approach to
privacy-preserving machine learning. In PATE, teacher models are trained on
distinct portions of sensitive data, and their predictions are privately
aggregated to label new training examples for a student model.
  Until now, PATE has primarily been explored with classification-like tasks,
where each example possesses a ground-truth label, and knowledge is transferred
to the student by labeling public examples. Generative AI models, however,
excel in open ended \emph{diverse} tasks with multiple valid responses and
scenarios that may not align with traditional labeled examples. Furthermore,
the knowledge of models is often encapsulated in the response distribution
itself and may be transferred from teachers to student in a more fluid way. We
propose \emph{hot PATE}, tailored for the diverse setting. In hot PATE, each
teacher model produces a response distribution and the aggregation method must
preserve both privacy and diversity of responses. We demonstrate, analytically
and empirically, that hot PATE achieves privacy-utility tradeoffs that are
comparable to, and in diverse settings, significantly surpass, the baseline
``cold'' PATE.