Sketch-Plan-Generalize: Continual Few-Shot Learning of Inductively
  Generalizable Spatial Concepts for Language-Guided Robot Manipulation
Namasivayam Kalithasan Sachit Sachdeva Himanshu Gaurav Singh Divyanshu Aggarwal Gurarmaan Singh Panjeta Vishal Bindal Arnav Tuli Rohan Paul Parag Singla
Our goal is to build embodied agents that can learn inductively generalizable
spatial concepts in a continual manner, e.g, constructing a tower of a given
height. Existing work suffers from certain limitations (a) (Liang et al., 2023)
and their multi-modal extensions, rely heavily on prior knowledge and are not
grounded in the demonstrations (b) (Liu et al., 2023) lack the ability to
generalize due to their purely neural approach. A key challenge is to achieve a
fine balance between symbolic representations which have the capability to
generalize, and neural representations that are physically grounded. In
response, we propose a neuro-symbolic approach by expressing inductive concepts
as symbolic compositions over grounded neural concepts. Our key insight is to
decompose the concept learning problem into the following steps 1) Sketch:
Getting a programmatic representation for the given instruction 2) Plan:
Perform Model-Based RL over the sequence of grounded neural action concepts to
learn a grounded plan 3) Generalize: Abstract out a generic (lifted) Python
program to facilitate generalizability. Continual learning is achieved by
interspersing learning of grounded neural concepts with higher level symbolic
constructs. Our experiments demonstrate that our approach significantly
outperforms existing baselines in terms of its ability to learn novel concepts
and generalize inductively.