Updating Language Models with Unstructured Facts: Towards Practical
  Knowledge Editing
Xiaobao Wu Liangming Pan William Yang Wang Anh Tuan Luu
Knowledge editing aims to inject knowledge updates into language models to
keep them correct and up-to-date. However, its current evaluation strategies
are notably impractical: they solely update with well-curated structured facts
(triplets with subjects, relations, and objects), whereas real-world knowledge
updates commonly emerge in unstructured texts like news articles. In this
paper, we propose a new benchmark, Unstructured Knowledge Editing (UKE). It
evaluates editing performance directly using unstructured texts as knowledge
updates, termed unstructured facts. Hence UKE avoids the laborious construction
of structured facts and enables efficient and responsive knowledge editing,
becoming a more practical benchmark. We conduct extensive experiments on newly
built datasets and demonstrate that UKE poses a significant challenge to
state-of-the-art knowledge editing methods, resulting in their critical
performance declines. We further show that this challenge persists even if we
extract triplets as structured facts. Our analysis discloses key insights to
motivate future research in UKE for more practical knowledge editing.