Integrated Object Deformation and Contact Patch Estimation from
  Visuo-Tactile Feedback
Mark Van der Merwe Youngsun Wi Dmitry Berenson Nima Fazeli
Reasoning over the interplay between object deformation and force
transmission through contact is central to the manipulation of compliant
objects. In this paper, we propose Neural Deforming Contact Field (NDCF), a
representation that jointly models object deformations and contact patches from
visuo-tactile feedback using implicit representations. Representing the object
geometry and contact with the environment implicitly allows a single model to
predict contact patches of varying complexity. Additionally, learning geometry
and contact simultaneously allows us to enforce physical priors, such as
ensuring contacts lie on the surface of the object. We propose a neural network
architecture to learn a NDCF, and train it using simulated data. We then
demonstrate that the learned NDCF transfers directly to the real-world without
the need for fine-tuning. We benchmark our proposed approach against a baseline
representing geometry and contact patches with point clouds. We find that NDCF
performs better on simulated data and in transfer to the real-world.