FABRIC: Automated Scoring and Feedback Generation for Essays
Jieun Han Haneul Yoo Junho Myung Minsun Kim Hyunseung Lim Yoonsu Kim Tak Yeon Lee Hwajung Hong Juho Kim So-Yeon Ahn Alice Oh
Automated essay scoring (AES) provides a useful tool for students and
instructors in writing classes by generating essay scores in real-time.
However, previous AES models do not provide more specific rubric-based scores
nor feedback on how to improve the essays, which can be even more important
than the overall scores for learning. We present FABRIC, a pipeline to help
students and instructors in English writing classes by automatically generating
1) the overall scores, 2) specific rubric-based scores, and 3) detailed
feedback on how to improve the essays. Under the guidance of English education
experts, we chose the rubrics for the specific scores as content, organization,
and language. The first component of the FABRIC pipeline is DREsS, a real-world
Dataset for Rubric-based Essay Scoring (DREsS). The second component is CASE, a
Corruption-based Augmentation Strategy for Essays, with which we can improve
the accuracy of the baseline model by 45.44%. The third component is EssayCoT,
the Essay Chain-of-Thought prompting strategy which uses scores predicted from
the AES model to generate better feedback. We evaluate the effectiveness of the
new dataset DREsS and the augmentation strategy CASE quantitatively and show
significant improvements over the models trained with existing datasets. We
evaluate the feedback generated by EssayCoT with English education experts to
show significant improvements in the helpfulness of the feedback across all
rubrics. Lastly, we evaluate the FABRIC pipeline with students in a college
English writing class who rated the generated scores and feedback with an
average of 6 on the Likert scale from 1 to 7.