DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven
  Differentiable Physics
Zhiao Huang Feng Chen Yewen Pu Chunru Lin Hao Su Chuang Gan
Combining gradient-based trajectory optimization with differentiable physics
simulation is an efficient technique for solving soft-body manipulation
problems. Using a well-crafted optimization objective, the solver can quickly
converge onto a valid trajectory. However, writing the appropriate objective
functions requires expert knowledge, making it difficult to collect a large set
of naturalistic problems from non-expert users. We introduce DiffVL, a method
that enables non-expert users to communicate soft-body manipulation tasks -- a
combination of vision and natural language, given in multiple stages -- that
can be readily leveraged by a differential physics solver. We have developed
GUI tools that enable non-expert users to specify 100 tasks inspired by
real-life soft-body manipulations from online videos, which we'll make public.
We leverage large language models to translate task descriptions into
machine-interpretable optimization objectives. The optimization objectives can
help differentiable physics solvers to solve these long-horizon multistage
tasks that are challenging for previous baselines.