Corrective Machine Unlearning
Shashwat Goel Ameya Prabhu Philip Torr Ponnurangam Kumaraguru Amartya Sanyal
Machine Learning models increasingly face data integrity challenges due to
the use of large-scale training datasets drawn from the internet. We study what
model developers can do if they detect that some data was manipulated or
incorrect. Such manipulated data can cause adverse effects like vulnerability
to backdoored samples, systematic biases, and in general, reduced accuracy on
certain input domains. Often, all manipulated training samples are not known,
and only a small, representative subset of the affected data is flagged.
  We formalize "Corrective Machine Unlearning" as the problem of mitigating the
impact of data affected by unknown manipulations on a trained model, possibly
knowing only a subset of impacted samples. We demonstrate that the problem of
corrective unlearning has significantly different requirements from traditional
privacy-oriented unlearning. We find most existing unlearning methods,
including the gold-standard retraining-from-scratch, require most of the
manipulated data to be identified for effective corrective unlearning. However,
one approach, SSD, achieves limited success in unlearning adverse effects with
just a small portion of the manipulated samples, showing the tractability of
this setting. We hope our work spurs research towards developing better methods
for corrective unlearning and offers practitioners a new strategy to handle
data integrity challenges arising from web-scale training.