Discovering Knowledge-Critical Subnetworks in Pretrained Language Models
Deniz Bayazit Negar Foroutan Zeming Chen Gail Weiss Antoine Bosselut
Pretrained language models (LMs) encode implicit representations of knowledge
in their parameters. However, localizing these representations and
disentangling them from each other remains an open problem. In this work, we
investigate whether pretrained language models contain various
knowledge-critical subnetworks: particular sparse computational subgraphs
responsible for encoding specific knowledge the model has memorized. We propose
a multi-objective differentiable weight masking scheme to discover these
subnetworks and show that we can use them to precisely remove specific
knowledge from models while minimizing adverse effects on the behavior of the
original language model. We demonstrate our method on multiple GPT2 variants,
uncovering highly sparse subnetworks (98%+) that are solely responsible for
specific collections of relational knowledge. When these subnetworks are
removed, the remaining network maintains most of its initial capacity (modeling
language and other memorized relational knowledge) but struggles to express the
removed knowledge, and suffers performance drops on examples needing this
removed knowledge on downstream tasks after finetuning.