Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models
Guihong Li Hsiang Hsu Chun-Fu Chen Radu Marculescu
The rapid growth of machine learning has spurred legislative initiatives such
as ``the Right to be Forgotten,'' allowing users to request data removal. In
response, ``machine unlearning'' proposes the selective removal of unwanted
data without the need for retraining from scratch. While the
Neural-Tangent-Kernel-based (NTK-based) unlearning method excels in
performance, it suffers from significant computational complexity, especially
for large-scale models and datasets. Our work introduces ``Fast-NTK,'' a novel
NTK-based unlearning algorithm that significantly reduces the computational
complexity by incorporating parameter-efficient fine-tuning methods, such as
fine-tuning batch normalization layers in a CNN or visual prompts in a vision
transformer. Our experimental results demonstrate scalability to much larger
neural networks and datasets (e.g., 88M parameters; 5k images), surpassing the
limitations of previous full-model NTK-based approaches designed for smaller
cases (e.g., 8M parameters; 500 images). Notably, our approach maintains a
performance comparable to the traditional method of retraining on the retain
set alone. Fast-NTK can thus enable for practical and scalable NTK-based
unlearning in deep neural networks.