Unlearning via Sparse Representations
Vedant Shah Frederik Träuble Ashish Malik Hugo Larochelle Michael Mozer Sanjeev Arora Yoshua Bengio Anirudh Goyal
Machine \emph{unlearning}, which involves erasing knowledge about a
\emph{forget set} from a trained model, can prove to be costly and infeasible
by existing techniques. We propose a nearly compute-free zero-shot unlearning
technique based on a discrete representational bottleneck. We show that the
proposed technique efficiently unlearns the forget set and incurs negligible
damage to the model's performance on the rest of the data set. We evaluate the
proposed technique on the problem of \textit{class unlearning} using three
datasets: CIFAR-10, CIFAR-100, and LACUNA-100. We compare the proposed
technique to SCRUB, a state-of-the-art approach which uses knowledge
distillation for unlearning. Across all three datasets, the proposed technique
performs as well as, if not better than SCRUB while incurring almost no
computational cost.