SOAR: Scene-debiasing Open-set Action Recognition
Yuanhao Zhai Ziyi Liu Zhenyu Wu Yi Wu Chunluan Zhou David Doermann Junsong Yuan Gang Hua
Deep learning models have a risk of utilizing spurious clues to make
predictions, such as recognizing actions based on the background scene. This
issue can severely degrade the open-set action recognition performance when the
testing samples have different scene distributions from the training samples.
To mitigate this problem, we propose a novel method, called Scene-debiasing
Open-set Action Recognition (SOAR), which features an adversarial scene
reconstruction module and an adaptive adversarial scene classification module.
The former prevents the decoder from reconstructing the video background given
video features, and thus helps reduce the background information in feature
learning. The latter aims to confuse scene type classification given video
features, with a specific emphasis on the action foreground, and helps to learn
scene-invariant information. In addition, we design an experiment to quantify
the scene bias. The results indicate that the current open-set action
recognizers are biased toward the scene, and our proposed SOAR method better
mitigates such bias. Furthermore, our extensive experiments demonstrate that
our method outperforms state-of-the-art methods, and the ablation studies
confirm the effectiveness of our proposed modules.