More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve
  Visually Diverse Images of Parsons Problems
Irene Hou Owen Man Sophie Mettille Sebastian Gutierrez Kenneth Angelikas Stephen MacNeil
The advent of large language models is reshaping computing education. Recent
research has demonstrated that these models can produce better explanations
than students, answer multiple-choice questions at or above the class average,
and generate code that can pass automated tests in introductory courses. These
capabilities have prompted instructors to rapidly adapt their courses and
assessment methods to accommodate changes in learning objectives and the
potential for academic integrity violations. While some scholars have advocated
for the integration of visual problems as a safeguard against the capabilities
of language models, new multimodal language models now have vision and language
capabilities that may allow them to analyze and solve visual problems. In this
paper, we evaluate the performance of two large multimodal models on visual
assignments, with a specific focus on Parsons problems presented across diverse
visual representations. Our results show that GPT-4V solved 96.7\% of these
visual problems, struggling minimally with a single Parsons problem.
Conversely, Bard performed poorly by only solving 69.2\% of problems,
struggling with common issues like hallucinations and refusals. These findings
suggest that merely transitioning to visual programming problems might not be a
panacea to issues of academic integrity in the generative AI era.