Predicting challenge moments from students' discourse: A comparison of
  GPT-4 to two traditional natural language processing approaches
Wannapon Suraworachet Jennifer Seon Mutlu Cukurova
Effective collaboration requires groups to strategically regulate themselves
to overcome challenges. Research has shown that groups may fail to regulate due
to differences in members' perceptions of challenges which may benefit from
external support. In this study, we investigated the potential of leveraging
three distinct natural language processing models: an expert knowledge
rule-based model, a supervised machine learning (ML) model and a Large Language
model (LLM), in challenge detection and challenge dimension identification
(cognitive, metacognitive, emotional and technical/other challenges) from
student discourse, was investigated. The results show that the supervised ML
and the LLM approaches performed considerably well in both tasks, in contrast
to the rule-based approach, whose efficacy heavily relies on the engineered
features by experts. The paper provides an extensive discussion of the three
approaches' performance for automated detection and support of students'
challenge moments in collaborative learning activities. It argues that,
although LLMs provide many advantages, they are unlikely to be the panacea to
issues of the detection and feedback provision of socially shared regulation of
learning due to their lack of reliability, as well as issues of validity
evaluation, privacy and confabulation. We conclude the paper with a discussion
on additional considerations, including model transparency to explore feasible
and meaningful analytical feedback for students and educators using LLMs.