Hands-Free VR
Jorge Askur Vazquez Fernandez Jae Joong Lee Santiago Andrés Serrano Vacca Alejandra Magana Bedrich Benes Voicu Popescu
The paper introduces Hands-Free VR, a voice-based natural-language interface
for VR. The user gives a command using their voice, the speech audio data is
converted to text using a speech-to-text deep learning model that is fine-tuned
for robustness to word phonetic similarity and to spoken English accents, and
the text is mapped to an executable VR command using a large language model
that is robust to natural language diversity. Hands-Free VR was evaluated in a
controlled within-subjects study (N = 22) that asked participants to find
specific objects and to place them in various configurations. In the control
condition participants used a conventional VR user interface to grab, carry,
and position the objects using the handheld controllers. In the experimental
condition participants used Hands-Free VR. The results confirm that: (1)
Hands-Free VR is robust to spoken English accents, as for 20 of our
participants English was not their first language, and to word phonetic
similarity, correctly transcribing the voice command 96.71% of the time; (2)
Hands-Free VR is robust to natural language diversity, correctly mapping the
transcribed command to an executable command in 97.83% of the time; (3)
Hands-Free VR had a significant efficiency advantage over the conventional VR
interface in terms of task completion time, total viewpoint translation, total
view direction rotation, and total left and right hand translations; (4)
Hands-Free VR received high user preference ratings in terms of ease of use,
intuitiveness, ergonomics, reliability, and desirability.