Multi-modality action recognition based on dual feature shift in vehicle
  cabin monitoring
Dan Lin Philip Hann Yung Lee Yiming Li Ruoyu Wang Kim-Hui Yap Bingbing Li You Shing Ngim
Driver Action Recognition (DAR) is crucial in vehicle cabin monitoring
systems. In real-world applications, it is common for vehicle cabins to be
equipped with cameras featuring different modalities. However, multi-modality
fusion strategies for the DAR task within car cabins have rarely been studied.
In this paper, we propose a novel yet efficient multi-modality driver action
recognition method based on dual feature shift, named DFS. DFS first integrates
complementary features across modalities by performing modality feature
interaction. Meanwhile, DFS achieves the neighbour feature propagation within
single modalities, by feature shifting among temporal frames. To learn common
patterns and improve model efficiency, DFS shares feature extracting stages
among multiple modalities. Extensive experiments have been carried out to
verify the effectiveness of the proposed DFS model on the Drive\&Act dataset.
The results demonstrate that DFS achieves good performance and improves the
efficiency of multi-modality driver action recognition.