GPT-4 Understands Discourse at Least as Well as Humans Do
Thomas Shultz Jamie Wise Ardavan Salehi Nobandegani
We test whether a leading AI system GPT-4 understands discourse as well as
humans do, using a standardized test of discourse comprehension. Participants
are presented with brief stories and then answer eight yes/no questions probing
their comprehension of the story. The questions are formatted to assess the
separate impacts of directness (stated vs. implied) and salience (main idea vs.
details). GPT-4 performs slightly, but not statistically significantly, better
than humans given the very high level of human performance. Both GPT-4 and
humans exhibit a strong ability to make inferences about information that is
not explicitly stated in a story, a critical test of understanding.