Distill to Delete: Unlearning in Graph Networks with Knowledge
  Distillation
Yash Sinha Murari Mandal Mohan Kankanhalli
Graph unlearning has emerged as a pivotal method to delete information from a
pre-trained graph neural network (GNN). One may delete nodes, a class of nodes,
edges, or a class of edges. An unlearning method enables the GNN model to
comply with data protection regulations (i.e., the right to be forgotten),
adapt to evolving data distributions, and reduce the GPU-hours carbon footprint
by avoiding repetitive retraining. Existing partitioning and aggregation-based
methods have limitations due to their poor handling of local graph dependencies
and additional overhead costs. More recently, GNNDelete offered a
model-agnostic approach that alleviates some of these issues. Our work takes a
novel approach to address these challenges in graph unlearning through
knowledge distillation, as it distills to delete in GNN (D2DGN). It is a
model-agnostic distillation framework where the complete graph knowledge is
divided and marked for retention and deletion. It performs distillation with
response-based soft targets and feature-based node embedding while minimizing
KL divergence. The unlearned model effectively removes the influence of deleted
graph elements while preserving knowledge about the retained graph elements.
D2DGN surpasses the performance of existing methods when evaluated on various
real-world graph datasets by up to $43.1\%$ (AUC) in edge and node unlearning
tasks. Other notable advantages include better efficiency, better performance
in removing target elements, preservation of performance for the retained
elements, and zero overhead costs. Notably, our D2DGN surpasses the
state-of-the-art GNNDelete in AUC by $2.4\%$, improves membership inference
ratio by $+1.3$, requires $10.2\times10^6$ fewer FLOPs per forward pass and up
to $\mathbf{3.2}\times$ faster.