Learning Priors for Non Rigid SfM from Casual Videos
Yoni Kasten Wuyue Lu Haggai Maron
We tackle the long-standing challenge of reconstructing 3D structures and
camera positions from videos. The problem is particularly hard when objects are
transformed in a non-rigid way. Current approaches to this problem make
unrealistic assumptions or require a long optimization time.
  We present TracksTo4D, a novel deep learning-based approach that enables
inferring 3D structure and camera positions from dynamic content originating
from in-the-wild videos using a single feed-forward pass on a sparse point
track matrix. To achieve this, we leverage recent advances in 2D point tracking
and design an equivariant neural architecture tailored for directly processing
2D point tracks by leveraging their symmetries. TracksTo4D is trained on a
dataset of in-the-wild videos utilizing only the 2D point tracks extracted from
the videos, without any 3D supervision. Our experiments demonstrate that
TracksTo4D generalizes well to unseen videos of unseen semantic categories at
inference time, producing equivalent results to state-of-the-art methods while
significantly reducing the runtime compared to other baselines.