Comparative Analysis of GPT-4 and Human Graders in Evaluating Praise
  Given to Students in Synthetic Dialogues
Dollaya Hirunyasiri Danielle R. Thomas Jionghao Lin Kenneth R. Koedinger Vincent Aleven
Research suggests that providing specific and timely feedback to human tutors
enhances their performance. However, it presents challenges due to the
time-consuming nature of assessing tutor performance by human evaluators. Large
language models, such as the AI-chatbot ChatGPT, hold potential for offering
constructive feedback to tutors in practical settings. Nevertheless, the
accuracy of AI-generated feedback remains uncertain, with scant research
investigating the ability of models like ChatGPT to deliver effective feedback.
In this work-in-progress, we evaluate 30 dialogues generated by GPT-4 in a
tutor-student setting. We use two different prompting approaches, the zero-shot
chain of thought and the few-shot chain of thought, to identify specific
components of effective praise based on five criteria. These approaches are
then compared to the results of human graders for accuracy. Our goal is to
assess the extent to which GPT-4 can accurately identify each praise criterion.
We found that both zero-shot and few-shot chain of thought approaches yield
comparable results. GPT-4 performs moderately well in identifying instances
when the tutor offers specific and immediate praise. However, GPT-4
underperforms in identifying the tutor's ability to deliver sincere praise,
particularly in the zero-shot prompting scenario where examples of sincere
tutor praise statements were not provided. Future work will focus on enhancing
prompt engineering, developing a more general tutoring rubric, and evaluating
our method using real-life tutoring dialogues.