CMFN: Cross-Modal Fusion Network for Irregular Scene Text Recognition
Jinzhi Zheng Ruyi Ji Libo Zhang Yanjun Wu Chen Zhao
Scene text recognition, as a cross-modal task involving vision and text, is
an important research topic in computer vision. Most existing methods use
language models to extract semantic information for optimizing visual
recognition. However, the guidance of visual cues is ignored in the process of
semantic mining, which limits the performance of the algorithm in recognizing
irregular scene text. To tackle this issue, we propose a novel cross-modal
fusion network (CMFN) for irregular scene text recognition, which incorporates
visual cues into the semantic mining process. Specifically, CMFN consists of a
position self-enhanced encoder, a visual recognition branch and an iterative
semantic recognition branch. The position self-enhanced encoder provides
character sequence position encoding for both the visual recognition branch and
the iterative semantic recognition branch. The visual recognition branch
carries out visual recognition based on the visual features extracted by CNN
and the position encoding information provided by the position self-enhanced
encoder. The iterative semantic recognition branch, which consists of a
language recognition module and a cross-modal fusion gate, simulates the way
that human recognizes scene text and integrates cross-modal visual cues for
text recognition. The experiments demonstrate that the proposed CMFN algorithm
achieves comparable performance to state-of-the-art algorithms, indicating its
effectiveness.