Empirical Study on Updating Key-Value Memories in Transformer
  Feed-forward Layers
Zihan Qiu Zeyu Huang Youcheng Huang Jie Fu
The feed-forward networks (FFNs) in transformers are recognized as a group of
key-value neural memories to restore abstract high-level knowledge. In this
work, we conduct an empirical ablation study on updating keys (the 1st layer in
the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those
two methods in various knowledge editing and fine-tuning tasks of large
language models to draw insights to understand FFNs further. Code is available
at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$.