Learning-Based Difficulty Calibration for Enhanced Membership Inference
  Attacks
Haonan Shi Tu Ouyang An Wang
Machine learning models, in particular deep neural networks, are currently an
integral part of various applications, from healthcare to finance. However,
using sensitive data to train these models raises concerns about privacy and
security. One method that has emerged to verify if the trained models are
privacy-preserving is Membership Inference Attacks (MIA), which allows
adversaries to determine whether a specific data point was part of a model's
training dataset. While a series of MIAs have been proposed in the literature,
only a few can achieve high True Positive Rates (TPR) in the low False Positive
Rate (FPR) region (0.01%~1%). This is a crucial factor to consider for an MIA
to be practically useful in real-world settings. In this paper, we present a
novel approach to MIA that is aimed at significantly improving TPR at low FPRs.
Our method, named learning-based difficulty calibration for MIA(LDC-MIA),
characterizes data records by their hardness levels using a neural network
classifier to determine membership. The experiment results show that LDC-MIA
can improve TPR at low FPR by up to 4x compared to the other difficulty
calibration based MIAs. It also has the highest Area Under ROC curve (AUC)
across all datasets. Our method's cost is comparable with most of the existing
MIAs, but is orders of magnitude more efficient than one of the
state-of-the-art methods, LiRA, while achieving similar performance.