SHARCS: Shared Concept Space for Explainable Multimodal Learning
Gabriele Dominici Pietro Barbiero Lucie Charlotte Magister Pietro Liò Nikola Simidjievski
Multimodal learning is an essential paradigm for addressing complex
real-world problems, where individual data modalities are typically
insufficient to accurately solve a given modelling task. While various deep
learning approaches have successfully addressed these challenges, their
reasoning process is often opaque; limiting the capabilities for a principled
explainable cross-modal analysis and any domain-expert intervention. In this
paper, we introduce SHARCS (SHARed Concept Space) -- a novel concept-based
approach for explainable multimodal learning. SHARCS learns and maps
interpretable concepts from different heterogeneous modalities into a single
unified concept-manifold, which leads to an intuitive projection of
semantically similar cross-modal concepts. We demonstrate that such an approach
can lead to inherently explainable task predictions while also improving
downstream predictive performance. Moreover, we show that SHARCS can operate
and significantly outperform other approaches in practically significant
scenarios, such as retrieval of missing modalities and cross-modal
explanations. Our approach is model-agnostic and easily applicable to different
types (and number) of modalities, thus advancing the development of effective,
interpretable, and trustworthy multimodal approaches.