Exploring the Potential of Large Language Models in Generating
  Code-Tracing Questions for Introductory Programming Courses
Aysa Xuemo Fan Ranran Haoran Zhang Luc Paquette Rui Zhang
In this paper, we explore the application of large language models (LLMs) for
generating code-tracing questions in introductory programming courses. We
designed targeted prompts for GPT4, guiding it to generate code-tracing
questions based on code snippets and descriptions. We established a set of
human evaluation metrics to assess the quality of questions produced by the
model compared to those created by human experts. Our analysis provides
insights into the capabilities and potential of LLMs in generating diverse
code-tracing questions. Additionally, we present a unique dataset of human and
LLM-generated tracing questions, serving as a valuable resource for both the
education and NLP research communities. This work contributes to the ongoing
dialogue on the potential uses of LLMs in educational settings.