Metaphor Understanding Challenge Dataset for LLMs
Xiaoyu Tong Rochelle Choenni Martha Lewis Ekaterina Shutova
Metaphors in natural language are a reflection of fundamental cognitive
processes such as analogical reasoning and categorisation, and are deeply
rooted in everyday communication. Metaphor understanding is therefore an
essential task for large language models (LLMs). We release the Metaphor
Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor
understanding capabilities of LLMs. The dataset provides over 10k paraphrases
for sentences containing metaphor use, as well as 1.5k instances containing
inapt paraphrases. The inapt paraphrases were carefully selected to serve as
control to determine whether the model indeed performs full metaphor
interpretation or rather resorts to lexical similarity. All apt and inapt
paraphrases were manually annotated. The metaphorical sentences cover natural
metaphor uses across 4 genres (academic, news, fiction, and conversation), and
they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5
demonstrate that MUNCH presents a challenging task for LLMs. The dataset is
freely accessible at
https://github.com/xiaoyuisrain/metaphor-understanding-challenge.