VoicePilot: Harnessing LLMs as Speech Interfaces for Physically
  Assistive Robots
Akhil Padmanabha Jessie Yuan Janavi Gupta Zulekha Karachiwalla Carmel Majidi Henny Admoni Zackory Erickson
Physically assistive robots present an opportunity to significantly increase
the well-being and independence of individuals with motor impairments or other
forms of disability who are unable to complete activities of daily living.
Speech interfaces, especially ones that utilize Large Language Models (LLMs),
can enable individuals to effectively and naturally communicate high-level
commands and nuanced preferences to robots. Frameworks for integrating LLMs as
interfaces to robots for high level task planning and code generation have been
proposed, but fail to incorporate human-centric considerations which are
essential while developing assistive interfaces. In this work, we present a
framework for incorporating LLMs as speech interfaces for physically assistive
robots, constructed iteratively with 3 stages of testing involving a feeding
robot, culminating in an evaluation with 11 older adults at an independent
living facility. We use both quantitative and qualitative data from the final
study to validate our framework and additionally provide design guidelines for
using LLMs as speech interfaces for assistive robots. Videos and supporting
files are located on our project website:
https://sites.google.com/andrew.cmu.edu/voicepilot/