Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision
  Transformers for High-Level Image Classification
Delfina Sol Martinez Pandiani Nicolas Lazzari Valentina Presutti
The increasing demand for automatic high-level image understanding,
particularly in detecting abstract concepts (AC) within images, underscores the
necessity for innovative and more interpretable approaches. These approaches
need to harmonize traditional deep vision methods with the nuanced,
context-dependent knowledge humans employ to interpret images at intricate
semantic levels. In this work, we leverage situated perceptual knowledge of
cultural images to enhance performance and interpretability in AC image
classification. We automatically extract perceptual semantic units from images,
which we then model and integrate into the ARTstract Knowledge Graph (AKG).
This resource captures situated perceptual semantics gleaned from over 14,000
cultural images labeled with ACs. Additionally, we enhance the AKG with
high-level linguistic frames. We compute KG embeddings and experiment with
relative representations and hybrid approaches that fuse these embeddings with
visual transformer embeddings. Finally, for interpretability, we conduct
posthoc qualitative analyses by examining model similarities with training
instances. Our results show that our hybrid KGE-ViT methods outperform existing
techniques in AC image classification. The posthoc interpretability analyses
reveal the visual transformer's proficiency in capturing pixel-level visual
attributes, contrasting with our method's efficacy in representing more
abstract and semantic scene elements. We demonstrate the synergy and
complementarity between KGE embeddings' situated perceptual knowledge and deep
visual model's sensory-perceptual understanding for AC image classification.
This work suggests a strong potential of neuro-symbolic methods for knowledge
integration and robust image representation for use in downstream intricate
visual comprehension tasks. All the materials and code are available online.