Rewriting the Script: Adapting Text Instructions for Voice Interaction
Alyssa Hwang Natasha Oza Chris Callison-Burch Andrew Head
Voice assistants have sharply risen in popularity in recent years, but their
use has been limited mostly to simple applications like music, hands-free
search, or control of internet-of-things devices. What would it take for voice
assistants to guide people through more complex tasks? In our work, we study
the limitations of the dominant approach voice assistants take to complex task
guidance: reading aloud written instructions. Using recipes as an example, we
observe twelve participants cook at home with a state-of-the-art voice
assistant. We learn that the current approach leads to nine challenges,
including obscuring the bigger picture, overwhelming users with too much
information, and failing to communicate affordances. Instructions delivered by
a voice assistant are especially difficult because they cannot be skimmed as
easily as written instructions. Alexa in particular did not surface crucial
details to the user or answer questions well. We draw on our observations to
propose eight ways in which voice assistants can ``rewrite the script'' --
summarizing, signposting, splitting, elaborating, volunteering, reordering,
redistributing, and visualizing -- to transform written sources into forms that
are readily communicated through spoken conversation. We conclude with a vision
of how modern advancements in natural language processing can be leveraged for
intelligent agents to guide users effectively through complex tasks.