Online Gesture Recognition using Transformer and Natural Language
  Processing
G. C. M. Silvestre F. Balado O. Akinremi M. Ramo
The Transformer architecture is shown to provide a powerful machine
transduction framework for online handwritten gestures corresponding to glyph
strokes of natural language sentences. The attention mechanism is successfully
used to create latent representations of an end-to-end encoder-decoder model,
solving multi-level segmentation while also learning some language features and
syntax rules. The additional use of a large decoding space with some learned
Byte-Pair-Encoding (BPE) is shown to provide robustness to ablated inputs and
syntax rules. The encoder stack was directly fed with spatio-temporal data
tokens potentially forming an infinitely large input vocabulary, an approach
that finds applications beyond that of this work. Encoder transfer learning
capabilities is also demonstrated on several languages resulting in faster
optimisation and shared parameters. A new supervised dataset of online
handwriting gestures suitable for generic handwriting recognition tasks was
used to successfully train a small transformer model to an average normalised
Levenshtein accuracy of 96% on English or German sentences and 94% in French.