Multi-Object Manipulation via Object-Centric Neural Scattering Functions
Stephen Tian Yancheng Cai Hong-Xing Yu Sergey Zakharov Katherine Liu Adrien Gaidon Yunzhu Li Jiajun Wu
Learned visual dynamics models have proven effective for robotic manipulation
tasks. Yet, it remains unclear how best to represent scenes involving
multi-object interactions. Current methods decompose a scene into discrete
objects, but they struggle with precise modeling and manipulation amid
challenging lighting conditions as they only encode appearance tied with
specific illuminations. In this work, we propose using object-centric neural
scattering functions (OSFs) as object representations in a model-predictive
control framework. OSFs model per-object light transport, enabling
compositional scene re-rendering under object rearrangement and varying
lighting conditions. By combining this approach with inverse parameter
estimation and graph-based neural dynamics models, we demonstrate improved
model-predictive control performance and generalization in compositional
multi-object environments, even in previously unseen scenarios and harsh
lighting conditions.