Incorporating Pre-trained Model Prompting in Multimodal Stock Volume
  Movement Prediction
Ruibo Chen Zhiyuan Zhang Yi Liu Ruihan Bao Keiko Harimoto Xu Sun
Multimodal stock trading volume movement prediction with stock-related news
is one of the fundamental problems in the financial area. Existing multimodal
works that train models from scratch face the problem of lacking universal
knowledge when modeling financial news. In addition, the models ability may be
limited by the lack of domain-related knowledge due to insufficient data in the
datasets. To handle this issue, we propose the Prompt-based MUltimodal Stock
volumE prediction model (ProMUSE) to process text and time series modalities.
We use pre-trained language models for better comprehension of financial news
and adopt prompt learning methods to leverage their capability in universal
knowledge to model textual information. Besides, simply fusing two modalities
can cause harm to the unimodal representations. Thus, we propose a novel
cross-modality contrastive alignment while reserving the unimodal heads beside
the fusion head to mitigate this problem. Extensive experiments demonstrate
that our proposed ProMUSE outperforms existing baselines. Comprehensive
analyses further validate the effectiveness of our architecture compared to
potential variants and learning mechanisms.