Exploring the Limitations of Large Language Models in Compositional
  Relation Reasoning
Jinman Zhao Xueyan Zhang
We present a comprehensive evaluation of large language models(LLMs)' ability
to reason about composition relations through a benchmark encompassing 1,500
test cases in English, designed to cover six distinct types of composition
relations: Positional, Comparative, Personal, Mathematical, Identity, and
Other. Acknowledging the significance of multilingual capabilities, we expanded
our assessment to include translations of these cases into Chinese, Japanese,
French, and Korean. Our Multilingual Composition Relation (MCR) benchmark aims
at investigating the robustness and adaptability of LLMs in handling
composition relation reasoning across diverse linguistic contexts.