Grounding Language Plans in Demonstrations Through Counterfactual
  Perturbations
Yanwei Wang Tsun-Hsuan Wang Jiayuan Mao Michael Hagenow Julie Shah
Grounding the common-sense reasoning of Large Language Models in physical
domains remains a pivotal yet unsolved problem for embodied AI. Whereas prior
works have focused on leveraging LLMs directly for planning in symbolic spaces,
this work uses LLMs to guide the search of task structures and constraints
implicit in multi-step demonstrations. Specifically, we borrow from
manipulation planning literature the concept of mode families, which group
robot configurations by specific motion constraints, to serve as an abstraction
layer between the high-level language representations of an LLM and the
low-level physical trajectories of a robot. By replaying a few human
demonstrations with synthetic perturbations, we generate coverage over the
demonstrations' state space with additional successful executions as well as
counterfactuals that fail the task. Our explanation-based learning framework
trains an end-to-end differentiable neural network to predict successful
trajectories from failures and as a by-product learns classifiers that ground
low-level states and images in mode families without dense labeling. The
learned grounding classifiers can further be used to translate language plans
into reactive policies in the physical domain in an interpretable manner. We
show our approach improves the interpretability and reactivity of imitation
learning through 2D navigation and simulated and real robot manipulation tasks.
Website: https://sites.google.com/view/grounding-plans