Offline Detection of Misspelled Handwritten Words by Convolving
  Recognition Model Features with Text Labels
Andrey Totev Tomas Ward
Offline handwriting recognition (HWR) has improved significantly with the
advent of deep learning architectures in recent years. Nevertheless, it remains
a challenging problem and practical applications often rely on post-processing
techniques for restricting the predicted words via lexicons or language models.
Despite their enhanced performance, such systems are less usable in contexts
where out-of-vocabulary words are anticipated, e.g. for detecting misspelled
words in school assessments. To that end, we introduce the task of comparing a
handwriting image to text. To solve the problem, we propose an unrestricted
binary classifier, consisting of a HWR feature extractor and a multimodal
classification head which convolves the feature extractor output with the
vector representation of the input text. Our model's classification head is
trained entirely on synthetic data created using a state-of-the-art generative
adversarial network. We demonstrate that, while maintaining high recall, the
classifier can be calibrated to achieve an average precision increase of 19.5%
compared to addressing the task by directly using state-of-the-art HWR models.
Such massive performance gains can lead to significant productivity increases
in applications utilizing human-in-the-loop automation.