SpeechCLIP+: Self-supervised multi-task representation learning for
  speech via CLIP and speech-image data
Hsuan-Fu Wang Yi-Jen Shih Heng-Jui Chang Layne Berry Puyuan Peng Hung-yi Lee Hsin-Min Wang David Harwath
The recently proposed visually grounded speech model SpeechCLIP is an
innovative framework that bridges speech and text through images via CLIP
without relying on text transcription. On this basis, this paper introduces two
extensions to SpeechCLIP. First, we apply the Continuous Integrate-and-Fire
(CIF) module to replace a fixed number of CLS tokens in the cascaded
architecture. Second, we propose a new hybrid architecture that merges the
cascaded and parallel architectures of SpeechCLIP into a multi-task learning
framework. Our experimental evaluation is performed on the Flickr8k and
SpokenCOCO datasets. The results show that in the speech keyword extraction
task, the CIF-based cascaded SpeechCLIP model outperforms the previous cascaded
SpeechCLIP model using a fixed number of CLS tokens. Furthermore, through our
hybrid architecture, cascaded task learning boosts the performance of the
parallel branch in image-speech retrieval tasks.