Large AI Model Empowered Multimodal Semantic Communications
Feibo Jiang Yubo Peng Li Dong Kezhi Wang Kun Yang Cunhua Pan Xiaohu You
Multimodal signals, including text, audio, image and video, can be integrated
into Semantic Communication (SC) for providing an immersive experience with low
latency and high quality at the semantic level. However, the multimodal SC has
several challenges, including data heterogeneity, semantic ambiguity, and
signal fading. Recent advancements in large AI models, particularly in
Multimodal Language Model (MLM) and Large Language Model (LLM), offer potential
solutions for these issues. To this end, we propose a Large AI Model-based
Multimodal SC (LAM-MSC) framework, in which we first present the MLM-based
Multimodal Alignment (MMA) that utilizes the MLM to enable the transformation
between multimodal and unimodal data while preserving semantic consistency.
Then, a personalized LLM-based Knowledge Base (LKB) is proposed, which allows
users to perform personalized semantic extraction or recovery through the LLM.
This effectively addresses the semantic ambiguity. Finally, we apply the
Conditional Generative adversarial networks-based channel Estimation (CGE) to
obtain Channel State Information (CSI). This approach effectively mitigates the
impact of fading channels in SC. Finally, we conduct simulations that
demonstrate the superior performance of the LAM-MSC framework.