Artificial-Spiking Hierarchical Networks for Vision-Language
  Representation Learning
Yeming Chen Siyu Zhang Yaoru Sun Weijian Liang Haoran Wang
With the success of self-supervised learning, multimodal foundation models
have rapidly adapted a wide range of downstream tasks driven by vision and
language (VL) pretraining. State-of-the-art methods achieve impressive
performance by pre-training on large-scale datasets. However, bridging the
semantic gap between the two modalities remains a nonnegligible challenge for
VL tasks. In this work, we propose an efficient computation framework for
multimodal alignment by introducing a novel visual semantic module to further
improve the performance of the VL tasks. Specifically, we propose a flexible
model, namely Artificial-Spiking Hierarchical Networks (ASH-Nets), which
combines the complementary advantages of Artificial neural networks (ANNs) and
Spiking neural networks (SNNs) to enrich visual semantic representations. In
particular, a visual concrete encoder and a semantic abstract encoder are
constructed to learn continuous and discrete latent variables to enhance the
flexibility of semantic encoding. Considering the spatio-temporal properties of
SNNs modeling, we introduce a contrastive learning method to optimize the
inputs of similar samples. This can improve the computational efficiency of the
hierarchical network, while the augmentation of hard samples is beneficial to
the learning of visual representations. Furthermore, the Spiking to Text
Uni-Alignment Learning (STUA) pre-training method is proposed, which only
relies on text features to enhance the encoding ability of abstract semantics.
We validate the performance on multiple well-established downstream VL tasks.
Experiments show that the proposed ASH-Nets achieve competitive results.