Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video
  Recognition
Xiao Wang Yao Rong Shiao Wang Yuan Chen Zhe Wu Bo Jiang Yonghong Tian Jin Tang
Pattern recognition based on RGB-Event data is a newly arising research topic
and previous works usually learn their features using CNN or Transformer. As we
know, CNN captures the local features well and the cascaded self-attention
mechanisms are good at extracting the long-range global relations. It is
intuitive to combine them for high-performance RGB-Event based video
recognition, however, existing works fail to achieve a good balance between the
accuracy and model parameters, as shown in Fig.~\ref{firstimage}. In this work,
we propose a novel RGB-Event based recognition framework termed TSCFormer,
which is a relatively lightweight CNN-Transformer model. Specifically, we
mainly adopt the CNN as the backbone network to first encode both RGB and Event
data. Meanwhile, we initialize global tokens as the input and fuse them with
RGB and Event features using the BridgeFormer module. It captures the global
long-range relations well between both modalities and maintains the simplicity
of the whole model architecture at the same time. The enhanced features will be
projected and fused into the RGB and Event CNN blocks, respectively, in an
interactive manner using F2E and F2V modules. Similar operations are conducted
for other CNN blocks to achieve adaptive fusion and local-global feature
enhancement under different resolutions. Finally, we concatenate these three
features and feed them into the classification head for pattern recognition.
Extensive experiments on two large-scale RGB-Event benchmark datasets
(PokerEvent and HARDVS) fully validated the effectiveness of our proposed
TSCFormer. The source code and pre-trained models will be released at
https://github.com/Event-AHU/TSCFormer.