Byte-Level Grammatical Error Correction Using Synthetic and Curated
  Corpora
Svanhvít Lilja Ingólfsdóttir Pétur Orri Ragnarsson Haukur Páll Jónsson Haukur Barri Símonarson Vilhjálmur Þorsteinsson Vésteinn Snæbjarnarson
Grammatical error correction (GEC) is the task of correcting typos, spelling,
punctuation and grammatical issues in text. Approaching the problem as a
sequence-to-sequence task, we compare the use of a common subword unit
vocabulary and byte-level encoding. Initial synthetic training data is created
using an error-generating pipeline, and used for finetuning two subword-level
models and one byte-level model. Models are then finetuned further on
hand-corrected error corpora, including texts written by children, university
students, dyslexic and second-language writers, and evaluated over different
error types and origins. We show that a byte-level model enables higher
correction quality than a subword approach, not only for simple spelling
errors, but also for more complex semantic, stylistic and grammatical issues.
In particular, initial training on synthetic corpora followed by finetuning on
a relatively small parallel corpus of real-world errors helps the byte-level
model correct a wide range of commonly occurring errors. Our experiments are
run for the Icelandic language but should hold for other similar languages,
particularly morphologically rich ones.