Predicting Learning Performance with Large Language Models: A Study in
  Adult Literacy
Liang Zhang Jionghao Lin Conrad Borchers John Sabatini John Hollander Meng Cao Xiangen Hu
Intelligent Tutoring Systems (ITSs) have significantly enhanced adult
literacy training, a key factor for societal participation, employment
opportunities, and lifelong learning. Our study investigates the application of
advanced AI models, including Large Language Models (LLMs) like GPT-4, for
predicting learning performance in adult literacy programs in ITSs. This
research is motivated by the potential of LLMs to predict learning performance
based on its inherent reasoning and computational capabilities. By using
reading comprehension datasets from the ITS, AutoTutor, we evaluate the
predictive capabilities of GPT-4 versus traditional machine learning methods in
predicting learning performance through five-fold cross-validation techniques.
Our findings show that the GPT-4 presents the competitive predictive abilities
with traditional machine learning methods such as Bayesian Knowledge Tracing,
Performance Factor Analysis, Sparse Factor Analysis Lite (SPARFA-Lite), tensor
factorization and eXtreme Gradient Boosting (XGBoost). While XGBoost (trained
on local machine) outperforms GPT-4 in predictive accuracy, GPT-4-selected
XGBoost and its subsequent tuning on the GPT-4 platform demonstrates superior
performance compared to local machine execution. Moreover, our investigation
into hyper-parameter tuning by GPT-4 versus grid-search suggests comparable
performance, albeit with less stability in the automated approach, using
XGBoost as the case study. Our study contributes to the field by highlighting
the potential of integrating LLMs with traditional machine learning models to
enhance predictive accuracy and personalize adult literacy education, setting a
foundation for future research in applying LLMs within ITSs.