The potential of large language models for improving probability
  learning: A study on ChatGPT3.5 and first-year computer engineering students
Angel Udias Antonio Alonso-Ayuso Ignacio Sanchez Sonia Hernandez Maria Eugenia Castellanos Raquel Montes Diez Emilio Lopez Cano
In this paper, we assess the efficacy of ChatGPT (version Feb 2023), a
large-scale language model, in solving probability problems typically presented
in introductory computer engineering exams. Our study comprised a set of 23
probability exercises administered to students at Rey Juan Carlos University
(URJC) in Madrid. The responses produced by ChatGPT were evaluated by a group
of five statistics professors, who assessed them qualitatively and assigned
grades based on the same criteria used for students. Our results indicate that
ChatGPT surpasses the average student in terms of phrasing, organization, and
logical reasoning. The model's performance remained consistent for both the
Spanish and English versions of the exercises. However, ChatGPT encountered
difficulties in executing basic numerical operations. Our experiments
demonstrate that requesting ChatGPT to provide the solution in the form of an R
script proved to be an effective approach for overcoming these limitations. In
summary, our results indicate that ChatGPT surpasses the average student in
solving probability problems commonly presented in introductory computer
engineering exams. Nonetheless, the model exhibits limitations in reasoning
around certain probability concepts. The model's ability to deliver
high-quality explanations and illustrate solutions in any programming language,
coupled with its performance in solving probability exercises, suggests that
large language models have the potential to serve as learning assistants.