Finite-context Indexing of Restricted Output Space for NLP Models Facing
  Noisy Input
Minh Nguyen Nancy F. Chen
NLP models excel on tasks with clean inputs, but are less accurate with noisy
inputs. In particular, character-level noise such as human-written typos and
adversarially-engineered realistic-looking misspellings often appears in text
and can easily trip up NLP models. Prior solutions to address character-level
noise often alter the content of the inputs (low fidelity), thus inadvertently
lowering model accuracy on clean inputs. We proposed FiRo, an approach to boost
NLP model performance on noisy inputs without sacrificing performance on clean
inputs. FiRo sanitizes the input text while preserving its fidelity by
inferring the noise-free form for each token in the input. FiRo uses
finite-context aggregation to obtain contextual embeddings which is then used
to find the noise-free form within a restricted output space. The output space
is restricted to a small cluster of probable candidates in order to predict the
noise-free tokens more accurately. Although the clusters are small, FiRo's
effective vocabulary (union of all clusters) can be scaled up to better
preserve the input content. Experimental results show NLP models that use FiRo
outperforming baselines on six classification tasks and one sequence labeling
task at various degrees of noise.