Transformers are Universal Predictors
Sourya Basu Moulik Choraria Lav R. Varshney
We find limits to the Transformer architecture for language modeling and show
it has a universal prediction property in an information-theoretic sense. We
further analyze performance in non-asymptotic data regimes to understand the
role of various components of the Transformer architecture, especially in the
context of data-efficient training. We validate our theoretical analysis with
experiments on both synthetic and real datasets.