Inter-connection: Effective Connection between Pre-trained Encoder and
  Decoder for Speech Translation
Yuta Nishikawa Satoshi Nakamura
In end-to-end speech translation, speech and text pre-trained models improve
translation quality. Recently proposed models simply connect the pre-trained
models of speech and text as encoder and decoder. Therefore, only the
information from the final layer of encoders is input to the decoder. Since it
is clear that the speech pre-trained model outputs different information from
each layer, the simple connection method cannot fully utilize the information
that the speech pre-trained model has. In this study, we propose an
inter-connection mechanism that aggregates the information from each layer of
the speech pre-trained model by weighted sums and inputs into the decoder. This
mechanism increased BLEU by approximately 2 points in en-de, en-ja, and en-zh
by increasing parameters by 2K when the speech pre-trained model was frozen.
Furthermore, we investigated the contribution of each layer for each language
by visualizing layer weights and found that the contributions were different.