NIFTY: Neural Object Interaction Fields for Guided Human Motion
  Synthesis
Nilesh Kulkarni Davis Rempe Kyle Genova Abhijit Kundu Justin Johnson David Fouhey Leonidas Guibas
We address the problem of generating realistic 3D motions of humans
interacting with objects in a scene. Our key idea is to create a neural
interaction field attached to a specific object, which outputs the distance to
the valid interaction manifold given a human pose as input. This interaction
field guides the sampling of an object-conditioned human motion diffusion
model, so as to encourage plausible contacts and affordance semantics. To
support interactions with scarcely available data, we propose an automated
synthetic data pipeline. For this, we seed a pre-trained motion model, which
has priors for the basics of human movement, with interaction-specific anchor
poses extracted from limited motion capture data. Using our guided diffusion
model trained on generated synthetic data, we synthesize realistic motions for
sitting and lifting with several objects, outperforming alternative approaches
in terms of motion quality and successful action completion. We call our
framework NIFTY: Neural Interaction Fields for Trajectory sYnthesis.