S$^3$Track: Self-supervised Tracking with Soft Assignment Flow
Fatemeh Azimi Fahim Mannan Felix Heide
In this work, we study self-supervised multiple object tracking without using
any video-level association labels. We propose to cast the problem of multiple
object tracking as learning the frame-wise associations between detections in
consecutive frames. To this end, we propose differentiable soft object
assignment for object association, making it possible to learn features
tailored to object association with differentiable end-to-end training. With
this training approach in hand, we develop an appearance-based model for
learning instance-aware object features used to construct a cost matrix based
on the pairwise distances between the object features. We train our model using
temporal and multi-view data, where we obtain association pseudo-labels using
optical flow and disparity information. Unlike most self-supervised tracking
methods that rely on pretext tasks for learning the feature correspondences,
our method is directly optimized for cross-object association in complex
scenarios. As such, the proposed method offers a reidentification-based MOT
approach that is robust to training hyperparameters and does not suffer from
local minima, which are a challenge in self-supervised methods. We evaluate our
proposed model on the KITTI, Waymo, nuScenes, and Argoverse datasets,
consistently improving over other unsupervised methods ($7.8\%$ improvement in
association accuracy on nuScenes).