The Impact of LoRA on the Emergence of Clusters in Transformers
Hugo Koubbi Matthieu Boussard Louis Hernandez
In this paper, we employ the mathematical framework on Transformers developed
by
\citet{sander2022sinkformers,geshkovski2023emergence,geshkovski2023mathematical}
to explore how variations in attention parameters and initial token values
impact the structural dynamics of token clusters. Our analysis demonstrates
that while the clusters within a modified attention matrix dynamics can exhibit
significant divergence from the original over extended periods, they maintain
close similarities over shorter intervals, depending on the parameter
differences. This work contributes to the fine-tuning field through practical
applications to the LoRA algorithm \cite{hu2021lora,peft}, enhancing our
understanding of the behavior of LoRA-enhanced Transformer models.