Transformer-based Joint Modelling for Automatic Essay Scoring and
  Off-Topic Detection
Sourya Dipta Das Yash Vadi Kuldeep Yadav
Automated Essay Scoring (AES) systems are widely popular in the market as
they constitute a cost-effective and time-effective option for grading systems.
Nevertheless, many studies have demonstrated that the AES system fails to
assign lower grades to irrelevant responses. Thus, detecting the off-topic
response in automated essay scoring is crucial in practical tasks where
candidates write unrelated text responses to the given task in the question. In
this paper, we are proposing an unsupervised technique that jointly scores
essays and detects off-topic essays. The proposed Automated Open Essay Scoring
(AOES) model uses a novel topic regularization module (TRM), which can be
attached on top of a transformer model, and is trained using a proposed hybrid
loss function. After training, the AOES model is further used to calculate the
Mahalanobis distance score for off-topic essay detection. Our proposed method
outperforms the baseline we created and earlier conventional methods on two
essay-scoring datasets in off-topic detection as well as on-topic scoring.
Experimental evaluation results on different adversarial strategies also show
how the suggested method is robust for detecting possible human-level
perturbations.