Multi-Modal Cognitive Maps based on Neural Networks trained on Successor
  Representations
Paul Stoewer Achim Schilling Andreas Maier Patrick Krauss
Cognitive maps are a proposed concept on how the brain efficiently organizes
memories and retrieves context out of them. The entorhinal-hippocampal complex
is heavily involved in episodic and relational memory processing, as well as
spatial navigation and is thought to built cognitive maps via place and grid
cells. To make use of the promising properties of cognitive maps, we set up a
multi-modal neural network using successor representations which is able to
model place cell dynamics and cognitive map representations. Here, we use
multi-modal inputs consisting of images and word embeddings. The network learns
the similarities between novel inputs and the training database and therefore
the representation of the cognitive map successfully. Subsequently, the
prediction of the network can be used to infer from one modality to another
with over $90\%$ accuracy. The proposed method could therefore be a building
block to improve current AI systems for better understanding of the environment
and the different modalities in which objects appear. The association of
specific modalities with certain encounters can therefore lead to context
awareness in novel situations when similar encounters with less information
occur and additional information can be inferred from the learned cognitive
map. Cognitive maps, as represented by the entorhinal-hippocampal complex in
the brain, organize and retrieve context from memories, suggesting that large
language models (LLMs) like ChatGPT could harness similar architectures to
function as a high-level processing center, akin to how the hippocampus
operates within the cortex hierarchy. Finally, by utilizing multi-modal inputs,
LLMs can potentially bridge the gap between different forms of data (like
images and words), paving the way for context-awareness and grounding of
abstract concepts through learned associations, addressing the grounding
problem in AI.