NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot
  Learning in Natural Human-Robot Interaction
Snehesh Shrestha Yantian Zha Saketh Banagiri Ge Gao Yiannis Aloimonos Cornelia Fermuller
Recent advancements in multimodal Human-Robot Interaction (HRI) datasets have
highlighted the fusion of speech and gesture, expanding robots' capabilities to
absorb explicit and implicit HRI insights. However, existing speech-gesture HRI
datasets often focus on elementary tasks, like object pointing and pushing,
revealing limitations in scaling to intricate domains and prioritizing human
command data over robot behavior records. To bridge these gaps, we introduce
NatSGD, a multimodal HRI dataset encompassing human commands through speech and
gestures that are natural, synchronized with robot behavior demonstrations.
NatSGD serves as a foundational resource at the intersection of machine
learning and HRI research, and we demonstrate its effectiveness in training
robots to understand tasks through multimodal human commands, emphasizing the
significance of jointly considering speech and gestures. We have released our
dataset, simulator, and code to facilitate future research in human-robot
interaction system learning; access these resources at
https://www.snehesh.com/natsgd/