A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check
Xunjian Yin Xiaojun Wan
With the development of pre-trained models and the incorporation of phonetic
and graphic information, neural models have achieved high scores in Chinese
Spelling Check (CSC). However, it does not provide a comprehensive reflection
of the models' capability due to the limited test sets. In this study, we
abstract the representative model paradigm, implement it with nine structures
and experiment them on comprehensive test sets we constructed with different
purposes. We perform a detailed analysis of the results and find that: 1)
Fusing phonetic and graphic information reasonably is effective for CSC. 2)
Models are sensitive to the error distribution of the test set, which reflects
the shortcomings of models and reveals the direction we should work on. 3)
Whether or not the errors and contexts have been seen has a significant impact
on models. 4) The commonly used benchmark, SIGHAN, can not reliably evaluate
models' performance.