Identification of Knowledge Neurons in Protein Language Models
Divya Nori Shivali Singireddy Marina Ten Have
Neural language models have become powerful tools for learning complex
representations of entities in natural language processing tasks. However,
their interpretability remains a significant challenge, particularly in domains
like computational biology where trust in model predictions is crucial. In this
work, we aim to enhance the interpretability of protein language models,
specifically the state-of-the-art ESM model, by identifying and characterizing
knowledge neurons - components that express understanding of key information.
After fine-tuning the ESM model for the task of enzyme sequence classification,
we compare two knowledge neuron selection methods that preserve a subset of
neurons from the original model. The two methods, activation-based and
integrated gradient-based selection, consistently outperform a random baseline.
In particular, these methods show that there is a high density of knowledge
neurons in the key vector prediction networks of self-attention modules. Given
that key vectors specialize in understanding different features of input
sequences, these knowledge neurons could capture knowledge of different enzyme
sequence motifs. In the future, the types of knowledge captured by each neuron
could be characterized.