Intriguing Equivalence Structures of the Embedding Space of Vision
  Transformers
Shaeke Salman Md Montasir Bin Shams Xiuwen Liu
Pre-trained large foundation models play a central role in the recent surge
of artificial intelligence, resulting in fine-tuned models with remarkable
abilities when measured on benchmark datasets, standard exams, and
applications. Due to their inherent complexity, these models are not well
understood. While small adversarial inputs to such models are well known, the
structures of the representation space are not well characterized despite their
fundamental importance. In this paper, using the vision transformers as an
example due to the continuous nature of their input space, we show via analyses
and systematic experiments that the representation space consists of large
piecewise linear subspaces where there exist very different inputs sharing the
same representations, and at the same time, local normal spaces where there are
visually indistinguishable inputs having very different representations. The
empirical results are further verified using the local directional estimations
of the Lipschitz constants of the underlying models. Consequently, the
resulting representations change the results of downstream models, and such
models are subject to overgeneralization and with limited semantically
meaningful generalization capability.