On the rate of convergence of an over-parametrized Transformer
  classifier learned by gradient descent
Michael Kohler Adam Krzyzak
One of the most recent and fascinating breakthroughs in artificial
intelligence is ChatGPT, a chatbot which can simulate human conversation.
ChatGPT is an instance of GPT4, which is a language model based on generative
gredictive gransformers. So if one wants to study from a theoretical point of
view, how powerful such artificial intelligence can be, one approach is to
consider transformer networks and to study which problems one can solve with
these networks theoretically. Here it is not only important what kind of models
these network can approximate, or how they can generalize their knowledge
learned by choosing the best possible approximation to a concrete data set, but
also how well optimization of such transformer network based on concrete data
set works. In this article we consider all these three different aspects
simultaneously and show a theoretical upper bound on the missclassification
probability of a transformer network fitted to the observed data. For
simplicity we focus in this context on transformer encoder networks which can
be applied to define an estimate in the context of a classification problem
involving natural language.