MV2MAE: Multi-View Video Masked Autoencoders
Ketul Shah Robert Crandall Jie Xu Peng Zhou Marian George Mayank Bansal Rama Chellappa
Videos captured from multiple viewpoints can help in perceiving the 3D
structure of the world and benefit computer vision tasks such as action
recognition, tracking, etc. In this paper, we present a method for
self-supervised learning from synchronized multi-view videos. We use a
cross-view reconstruction task to inject geometry information in the model. Our
approach is based on the masked autoencoder (MAE) framework. In addition to the
same-view decoder, we introduce a separate cross-view decoder which leverages
cross-attention mechanism to reconstruct a target viewpoint video using a video
from source viewpoint, to help representations robust to viewpoint changes. For
videos, static regions can be reconstructed trivially which hinders learning
meaningful representations. To tackle this, we introduce a motion-weighted
reconstruction loss which improves temporal modeling. We report
state-of-the-art results on the NTU-60, NTU-120 and ETRI datasets, as well as
in the transfer learning setting on NUCLA, PKU-MMD-II and ROCOG-v2 datasets,
demonstrating the robustness of our approach. Code will be made available.