Make BERT-based Chinese Spelling Check Model Enhanced by Layerwise
  Attention and Gaussian Mixture Model
Yongchang Cao Liang He Zhen Wu Xinyu Dai
BERT-based models have shown a remarkable ability in the Chinese Spelling
Check (CSC) task recently. However, traditional BERT-based methods still suffer
from two limitations. First, although previous works have identified that
explicit prior knowledge like Part-Of-Speech (POS) tagging can benefit in the
CSC task, they neglected the fact that spelling errors inherent in CSC data can
lead to incorrect tags and therefore mislead models. Additionally, they ignored
the correlation between the implicit hierarchical information encoded by BERT's
intermediate layers and different linguistic phenomena. This results in
sub-optimal accuracy. To alleviate the above two issues, we design a
heterogeneous knowledge-infused framework to strengthen BERT-based CSC models.
To incorporate explicit POS knowledge, we utilize an auxiliary task strategy
driven by Gaussian mixture model. Meanwhile, to incorporate implicit
hierarchical linguistic knowledge within the encoder, we propose a novel form
of n-gram-based layerwise self-attention to generate a multilayer
representation. Experimental results show that our proposed framework yields a
stable performance boost over four strong baseline models and outperforms the
previous state-of-the-art methods on two datasets.