Conditionally Combining Robot Skills using Large Language Models
K. R. Zentner Ryan Julian Brian Ichter Gaurav S. Sukhatme
This paper combines two contributions. First, we introduce an extension of
the Meta-World benchmark, which we call "Language-World," which allows a large
language model to operate in a simulated robotic environment using
semi-structured natural language queries and scripted skills described using
natural language. By using the same set of tasks as Meta-World, Language-World
results can be easily compared to Meta-World results, allowing for a point of
comparison between recent methods using Large Language Models (LLMs) and those
using Deep Reinforcement Learning. Second, we introduce a method we call Plan
Conditioned Behavioral Cloning (PCBC), that allows finetuning the behavior of
high-level plans using end-to-end demonstrations. Using Language-World, we show
that PCBC is able to achieve strong performance in a variety of few-shot
regimes, often achieving task generalization with as little as a single
demonstration. We have made Language-World available as open-source software at
https://github.com/krzentner/language-world/.