Perception and Semantic Aware Regularization for Sequential Confidence
  Calibration
Zhenghua Peng Yu Luo Tianshui Chen Keke Xu Shuangping Huang
Deep sequence recognition (DSR) models receive increasing attention due to
their superior application to various applications. Most DSR models use merely
the target sequences as supervision without considering other related
sequences, leading to over-confidence in their predictions. The DSR models
trained with label smoothing regularize labels by equally and independently
smoothing each token, reallocating a small value to other tokens for mitigating
overconfidence. However, they do not consider tokens/sequences correlations
that may provide more effective information to regularize training and thus
lead to sub-optimal performance. In this work, we find tokens/sequences with
high perception and semantic correlations with the target ones contain more
correlated and effective information and thus facilitate more effective
regularization. To this end, we propose a Perception and Semantic aware
Sequence Regularization framework, which explore perceptively and semantically
correlated tokens/sequences as regularization. Specifically, we introduce a
semantic context-free recognition and a language model to acquire similar
sequences with high perceptive similarities and semantic correlation,
respectively. Moreover, over-confidence degree varies across samples according
to their difficulties. Thus, we further design an adaptive calibration
intensity module to compute a difficulty score for each samples to obtain
finer-grained regularization. Extensive experiments on canonical sequence
recognition tasks, including scene text and speech recognition, demonstrate
that our method sets novel state-of-the-art results. Code is available at
https://github.com/husterpzh/PSSR.