Has Approximate Machine Unlearning been evaluated properly? From
  Auditing to Side Effects
Cheng-Long Wang Qi Li Zihang Xiang Di Wang
The growing concerns surrounding data privacy and security have underscored
the critical necessity for machine unlearning--aimed at fully removing data
lineage from machine learning models. MLaaS providers expect this to be their
ultimate safeguard for regulatory compliance. Despite its critical importance,
the pace at which privacy communities have been developing and implementing
strong methods to verify the effectiveness of machine unlearning has been
disappointingly slow, with this vital area often receiving insufficient focus.
This paper seeks to address this shortfall by introducing well-defined and
effective metrics for black-box unlearning auditing tasks. We transform the
auditing challenge into a question of non-membership inference and develop
efficient metrics for auditing. By relying exclusively on the original and
unlearned models--eliminating the need to train additional shadow models--our
approach simplifies the evaluation of unlearning at the individual data point
level. Utilizing these metrics, we conduct an in-depth analysis of current
approximate machine unlearning algorithms, identifying three key directions
where these approaches fall short: utility, resilience, and equity. Our aim is
that this work will greatly improve our understanding of approximate machine
unlearning methods, taking a significant stride towards converting the
theoretical right to data erasure into a auditable reality.