Double-Dip: Thwarting Label-Only Membership Inference Attacks with
  Transfer Learning and Randomization
Arezoo Rajabi Reeya Pimple Aiswarya Janardhanan Surudhi Asokraj Bhaskar Ramasubramanian Radha Poovendran
Transfer learning (TL) has been demonstrated to improve DNN model performance
when faced with a scarcity of training samples. However, the suitability of TL
as a solution to reduce vulnerability of overfitted DNNs to privacy attacks is
unexplored. A class of privacy attacks called membership inference attacks
(MIAs) aim to determine whether a given sample belongs to the training dataset
(member) or not (nonmember). We introduce Double-Dip, a systematic empirical
study investigating the use of TL (Stage-1) combined with randomization
(Stage-2) to thwart MIAs on overfitted DNNs without degrading classification
accuracy. Our study examines the roles of shared feature space and parameter
values between source and target models, number of frozen layers, and
complexity of pretrained models. We evaluate Double-Dip on three (Target,
Source) dataset paris: (i) (CIFAR-10, ImageNet), (ii) (GTSRB, ImageNet), (iii)
(CelebA, VGGFace2). We consider four publicly available pretrained DNNs: (a)
VGG-19, (b) ResNet-18, (c) Swin-T, and (d) FaceNet. Our experiments demonstrate
that Stage-1 reduces adversary success while also significantly increasing
classification accuracy of nonmembers against an adversary with either
white-box or black-box DNN model access, attempting to carry out SOTA
label-only MIAs. After Stage-2, success of an adversary carrying out a
label-only MIA is further reduced to near 50%, bringing it closer to a random
guess and showing the effectiveness of Double-Dip. Stage-2 of Double-Dip also
achieves lower ASR and higher classification accuracy than regularization and
differential privacy-based methods.