Bidirectional Transformer Reranker for Grammatical Error Correction
Ying Zhang Hidetaka Kamigaito Manabu Okumura
Pre-trained seq2seq models have achieved state-of-the-art results in the
grammatical error correction task. However, these models still suffer from a
prediction bias due to their unidirectional decoding. Thus, we propose a
bidirectional Transformer reranker (BTR), that re-estimates the probability of
each candidate sentence generated by the pre-trained seq2seq model. The BTR
preserves the seq2seq-style Transformer architecture but utilizes a BERT-style
self-attention mechanism in the decoder to compute the probability of each
target token by using masked language modeling to capture bidirectional
representations from the target context. For guiding the reranking, the BTR
adopts negative sampling in the objective function to minimize the
unlikelihood. During inference, the BTR gives final results after comparing the
reranked top-1 results with the original ones by an acceptance threshold.
Experimental results show that, in reranking candidates from a pre-trained
seq2seq model, T5-base, the BTR on top of T5-base could yield 65.47 and 71.27
F0.5 scores on the CoNLL-14 and BEA test sets, respectively, and yield 59.52
GLEU score on the JFLEG corpus, with improvements of 0.36, 0.76 and 0.48 points
compared with the original T5-base. Furthermore, when reranking candidates from
T5-large, the BTR on top of T5-base improved the original T5-large by 0.26
points on the BEA test set.