The Two Word Test: A Semantic Benchmark for Large Language Models
Nicholas Riccardi Rutvik H. Desai
Large Language Models (LLMs) have shown remarkable abilities recently,
including passing advanced professional exams and demanding benchmark tests.
This performance has led many to suggest that they are close to achieving
humanlike or 'true' understanding of language, and even Artificial General
Intelligence (AGI). Here, we provide a new open-source benchmark that can
assess semantic abilities of LLMs using two-word phrases using a task that can
be performed relatively easily by humans without advanced training. Combining
multiple words into a single concept is a fundamental aspect of human language
and intelligence. The test requires meaningfulness judgments of 1768 noun-noun
combinations that have been rated as meaningful (e.g., baby boy) or not
meaningful (e.g., goat sky). by 150 human raters. We provide versions of the
task that probe meaningfulness ratings on a 0-4 scale as well as binary
judgments. We conducted a series of experiments using the TWT on GPT-4,
GPT-3.5, and Bard, with both versions. Results demonstrated that, compared to
humans, all models perform poorly at rating meaningfulness of these phrases.
GPT-3.5 and Bard are also unable to make binary discriminations between
sensible and nonsense phrases as making sense. GPT-4 makes a substantial
improvement in binary discrimination of combinatorial phrases but is still
significantly worse than human performance. The TWT can be used to understand
the limitations and weaknesses of current LLMs, and potentially improve them.
The test also reminds us that caution is warranted in attributing 'true
understanding' or AGI to LLMs. TWT is available at:
https://github.com/NickRiccardi/two-word-test