Unveiling LLMs: The Evolution of Latent Representations in a Temporal
  Knowledge Graph
Marco Bronzini Carlo Nicolini Bruno Lepri Jacopo Staiano Andrea Passerini
Large Language Models (LLMs) demonstrate an impressive capacity to recall a
vast range of common factual knowledge information. However, unravelling the
underlying reasoning of LLMs and explaining their internal mechanisms of
exploiting this factual knowledge remain active areas of investigation. Our
work analyzes the factual knowledge encoded in the latent representation of
LLMs when prompted to assess the truthfulness of factual claims. We propose an
end-to-end framework that jointly decodes the factual knowledge embedded in the
latent space of LLMs from a vector space to a set of ground predicates and
represents its evolution across the layers using a temporal knowledge graph.
Our framework relies on the technique of activation patching which intervenes
in the inference computation of a model by dynamically altering its latent
representations. Consequently, we neither rely on external models nor training
processes. We showcase our framework with local and global interpretability
analyses using two claim verification datasets: FEVER and CLIMATE-FEVER. The
local interpretability analysis exposes different latent errors from
representation to multi-hop reasoning errors. On the other hand, the global
analysis uncovered patterns in the underlying evolution of the model's factual
knowledge (e.g., store-and-seek factual information). By enabling graph-based
analyses of the latent representations, this work represents a step towards the
mechanistic interpretability of LLMs.