Towards Modeling Learner Performance with Large Language Models
Seyed Parsa Neshaei Richard Lee Davis Adam Hazimeh Bojan Lazarevski Pierre Dillenbourg Tanja Käser
Recent work exploring the capabilities of pre-trained large language models
(LLMs) has demonstrated their ability to act as general pattern machines by
completing complex token sequences representing a wide array of tasks,
including time-series prediction and robot control. This paper investigates
whether the pattern recognition and sequence modeling capabilities of LLMs can
be extended to the domain of knowledge tracing, a critical component in the
development of intelligent tutoring systems (ITSs) that tailor educational
experiences by predicting learner performance over time. In an empirical
evaluation across multiple real-world datasets, we compare two approaches to
using LLMs for this task, zero-shot prompting and model fine-tuning, with
existing, non-LLM approaches to knowledge tracing. While LLM-based approaches
do not achieve state-of-the-art performance, fine-tuned LLMs surpass the
performance of naive baseline models and perform on par with standard Bayesian
Knowledge Tracing approaches across multiple metrics. These findings suggest
that the pattern recognition capabilities of LLMs can be used to model complex
learning trajectories, opening a novel avenue for applying LLMs to educational
contexts. The paper concludes with a discussion of the implications of these
findings for future research, suggesting that further refinements and a deeper
understanding of LLMs' predictive mechanisms could lead to enhanced performance
in knowledge tracing tasks.