InCoRo: In-Context Learning for Robotics Control with Feedback Loops
Jiaqiang Ye Zhu Carla Gomez Cano David Vazquez Bermudez Michal Drozdzal
One of the challenges in robotics is to enable robotic units with the
reasoning capability that would be robust enough to execute complex tasks in
dynamic environments. Recent advances in LLMs have positioned them as go-to
tools for simple reasoning tasks, motivating the pioneering work of Liang et
al. [35] that uses an LLM to translate natural language commands into low-level
static execution plans for robotic units. Using LLMs inside robotics systems
brings their generalization to a new level, enabling zero-shot generalization
to new tasks. This paper extends this prior work to dynamic environments. We
propose InCoRo, a system that uses a classical robotic feedback loop composed
of an LLM controller, a scene understanding unit, and a robot. Our system
continuously analyzes the state of the environment and provides adapted
execution commands, enabling the robot to adjust to changing environmental
conditions and correcting for controller errors. Our system does not require
any iterative optimization to learn to accomplish a task as it leverages
in-context learning with an off-the-shelf LLM model. Through an extensive
validation process involving two standardized industrial robotic units -- SCARA
and DELTA types -- we contribute knowledge about these robots, not popular in
the community, thereby enriching it. We highlight the generalization
capabilities of our system and show that (1) in-context learning in combination
with the current state-of-the-art LLMs is an effective way to implement a
robotic controller; (2) in static environments, InCoRo surpasses the prior art
in terms of the success rate; (3) in dynamic environments, we establish new
state-of-the-art for the SCARA and DELTA units, respectively. This research
paves the way towards building reliable, efficient, intelligent autonomous
systems that adapt to dynamic environments.