Physics-Based Rigid Body Object Tracking and Friction Filtering From
  RGB-D Videos
Rama Krishna Kandukuri Michael Strecke Joerg Stueckler
Physics-based understanding of object interactions from sensory observations
is an essential capability in augmented reality and robotics. It enables
capturing the properties of a scene for simulation and control. In this paper,
we propose a novel approach for real-to-sim which tracks rigid objects in 3D
from RGB-D images and infers physical properties of the objects. We use a
differentiable physics simulation as state-transition model in an Extended
Kalman Filter which can model contact and friction for arbitrary mesh-based
shapes and in this way estimate physically plausible trajectories. We
demonstrate that our approach can filter position, orientation, velocities, and
concurrently can estimate the coefficient of friction of the objects. We
analyse our approach on various sliding scenarios in synthetic image sequences
of single objects and colliding objects. We also demonstrate and evaluate our
approach on a real-world dataset. We will make our novel benchmark datasets
publicly available to foster future research in this novel problem setting and
comparison with our method.