Improving fit to human reading times via temperature-scaled surprisal
Tong Liu Iza Škrjanec Vera Demberg
Past studies have provided broad support for that words with lower
predictability (i.e., higher surprisal) require more time for comprehension by
using large language models (LLMs) to simulate humans' cognitive load. In
general, these studies have implicitly assumed that the probability scores from
LLMs are accurate, ignoring the discrepancies between human cognition and LLMs
from this standpoint. Inspired by the concept of probability calibration, we
are the first work to focus on the probability distribution for human reading
simulation. We propose to use temperature-scaled surprisal, a surprisal
calculated by shaped probability, to be the predictor of human reading times.
Our results across three corpora consistently revealed that such a surprisal
can drastically improve the prediction of reading times. Setting the
temperature to be approximately 2.5 across all models and datasets can yield up
to an 89% of increase in delta log-likelihood in our setting. We also propose a
calibration metric to quantify the possible human-likeness bias. Further
analysis was done and provided insights into this phenomenon.