Student Mastery or AI Deception? Analyzing ChatGPT's Assessment
  Proficiency and Evaluating Detection Strategies
Kevin Wang Seth Akins Abdallah Mohammed Ramon Lawrence
Generative AI systems such as ChatGPT have a disruptive effect on learning
and assessment. Computer science requires practice to develop skills in problem
solving and programming that are traditionally developed using assignments.
Generative AI has the capability of completing these assignments for students
with high accuracy, which dramatically increases the potential for academic
integrity issues and students not achieving desired learning outcomes. This
work investigates the performance of ChatGPT by evaluating it across three
courses (CS1,CS2,databases). ChatGPT completes almost all introductory
assessments perfectly. Existing detection methods, such as MOSS and JPlag
(based on similarity metrics) and GPTzero (AI detection), have mixed success in
identifying AI solutions. Evaluating instructors and teaching assistants using
heuristics to distinguish between student and AI code shows that their
detection is not sufficiently accurate. These observations emphasize the need
for adapting assessments and improved detection methods.