Improving Isochronous Machine Translation with Target Factors and
  Auxiliary Counters
Proyag Pal Brian Thompson Yogesh Virkar Prashant Mathur Alexandra Chronopoulou Marcello Federico
To translate speech for automatic dubbing, machine translation needs to be
isochronous, i.e. translated speech needs to be aligned with the source in
terms of speech durations. We introduce target factors in a transformer model
to predict durations jointly with target language phoneme sequences. We also
introduce auxiliary counters to help the decoder to keep track of the timing
information while generating target phonemes. We show that our model improves
translation quality and isochrony compared to previous work where the
translation model is instead trained to predict interleaved sequences of
phonemes and durations.