A Unified Framework for Model Editing
Akshat Gupta Dev Sajnani Gopala Anumanchipalli
Model editing is a growing area focused on updating the knowledge embedded
within models. Among the various methodologies, ROME and MEMIT stand out as
leading "locate-and-edit" model editing techniques. While MEMIT enables batched
editing of memories, ROME is limited to changing one fact at a time. This paper
introduces a unifying framework that brings ROME and MEMIT under a single
conceptual umbrella, optimizing for the same goal, which we call the
"preservation-memorization" objective. This objective aims to preserve the
representations of certain selected vectors while memorizing the
representations of new factual information. Specifically, ROME optimizes this
objective using an equality constraint, whereas MEMIT employs a more flexible
least-square constraint. In addition to making batched edits, MEMIT also edits
the model at multiple layers. We disentangle the distribution of edits to
multiple layers from the optimization objective of MEMIT and show that these
edit-distribution algorithms should be considered separate entities worthy of
their own line of research.
  Finally, we present EMMET - an Equality-constrained Mass Model Editing
algorithm for Transformers, a new batched memory-editing algorithm. With EMMET,
we present a closed form solution for the equality-constrained version of the
preservation-memorization objective. We show that EMMET is able to perform
batched-edits on par with MEMIT up to a batch-size of 256 and discuss the
challenges in stabilizing EMMET. By articulating the "locate-and-edit" model
editing algorithms under a simple conceptual framework of
"preservation-memorization", we aim to bridge the gap between intuition and
mathematics and hope to simplify the journey for future researchers in model
editing.