Language Aligned Visual Representations Predict Human Behavior in
  Naturalistic Learning Tasks
Can Demircan Tankred Saanum Leonardo Pettini Marcel Binz Blazej M Baczkowski Paula Kaanders Christian F Doeller Mona M Garvert Eric Schulz
Humans possess the ability to identify and generalize relevant features of
natural objects, which aids them in various situations. To investigate this
phenomenon and determine the most effective representations for predicting
human behavior, we conducted two experiments involving category learning and
reward learning. Our experiments used realistic images as stimuli, and
participants were tasked with making accurate decisions based on novel stimuli
for all trials, thereby necessitating generalization. In both tasks, the
underlying rules were generated as simple linear functions using stimulus
dimensions extracted from human similarity judgments. Notably, participants
successfully identified the relevant stimulus features within a few trials,
demonstrating effective generalization. We performed an extensive model
comparison, evaluating the trial-by-trial predictive accuracy of diverse deep
learning models' representations of human choices. Intriguingly,
representations from models trained on both text and image data consistently
outperformed models trained solely on images, even surpassing models using the
features that generated the task itself. These findings suggest that
language-aligned visual representations possess sufficient richness to describe
human generalization in naturalistic settings and emphasize the role of
language in shaping human cognition.