Learning When to Speak: Latency and Quality Trade-offs for Simultaneous
  Speech-to-Speech Translation with Offline Models
Liam Dugan Anshul Wadhawan Kyle Spence Chris Callison-Burch Morgan McGuire Victor Zordan
Recent work in speech-to-speech translation (S2ST) has focused primarily on
offline settings, where the full input utterance is available before any output
is given. This, however, is not reasonable in many real-world scenarios. In
latency-sensitive applications, rather than waiting for the full utterance,
translations should be spoken as soon as the information in the input is
present. In this work, we introduce a system for simultaneous S2ST targeting
real-world use cases. Our system supports translation from 57 languages to
English with tunable parameters for dynamically adjusting the latency of the
output -- including four policies for determining when to speak an output
sequence. We show that these policies achieve offline-level accuracy with
minimal increases in latency over a Greedy (wait-$k$) baseline. We open-source
our evaluation code and interactive test script to aid future SimulS2ST
research and application development.