Constrained Robotic Navigation on Preferred Terrains Using LLMs and
  Speech Instruction: Exploiting the Power of Adverbs
Faraz Lotfi Farnoosh Faraji Nikhil Kakodkar Travis Manderson David Meger Gregory Dudek
This paper explores leveraging large language models for map-free off-road
navigation using generative AI, reducing the need for traditional data
collection and annotation. We propose a method where a robot receives verbal
instructions, converted to text through Whisper, and a large language model
(LLM) model extracts landmarks, preferred terrains, and crucial adverbs
translated into speed settings for constrained navigation. A language-driven
semantic segmentation model generates text-based masks for identifying
landmarks and terrain types in images. By translating 2D image points to the
vehicle's motion plane using camera parameters, an MPC controller can guides
the vehicle towards the desired terrain. This approach enhances adaptation to
diverse environments and facilitates the use of high-level instructions for
navigating complex and challenging terrains.