Attention-Only Transformers and Implementing MLPs with Attention Heads
Robert Huben Valerie Morris
The transformer architecture is widely used in machine learning models and
consists of two alternating sublayers: attention heads and MLPs. We prove that
an MLP neuron can be implemented by a masked attention head with internal
dimension 1 so long as the MLP's activation function comes from a restricted
class including SiLU and close approximations of ReLU and GeLU. This allows one
to convert an MLP-and-attention transformer into an attention-only transformer
at the cost of greatly increasing the number of attention heads. We also prove
that attention heads can perform the components of an MLP (linear
transformations and activation functions) separately. Finally, we prove that
attention heads can encode arbitrary masking patterns in their weight matrices
to within arbitrarily small error.