Uncertainty Estimation of Transformers' Predictions via Topological
  Analysis of the Attention Matrices
Elizaveta Kostenok Daniil Cherniavskii Alexey Zaytsev
Determining the degree of confidence of deep learning model in its prediction
is an open problem in the field of natural language processing. Most of the
classical methods for uncertainty estimation are quite weak for text
classification models. We set the task of obtaining an uncertainty estimate for
neural networks based on the Transformer architecture. A key feature of such
mo-dels is the attention mechanism, which supports the information flow between
the hidden representations of tokens in the neural network. We explore the
formed relationships between internal representations using Topological Data
Analysis methods and utilize them to predict model's confidence. In this paper,
we propose a method for uncertainty estimation based on the topological
properties of the attention mechanism and compare it with classical methods. As
a result, the proposed algorithm surpasses the existing methods in quality and
opens up a new area of application of the attention mechanism, but requires the
selection of topological features.