The Impact of LoRA on the Emergence of Clusters in Transformers
This work provides insights for fine-tuning in LoRA-enhanced Transformer models, though it appears incremental in applying existing mathematical frameworks.
The paper analyzes how attention parameters and initial token values affect token cluster dynamics in Transformers, showing that clusters diverge significantly over long periods but remain similar over short intervals depending on parameter differences.
In this paper, we employ the mathematical framework on Transformers developed by \citet{sander2022sinkformers,geshkovski2023emergence,geshkovski2023mathematical} to explore how variations in attention parameters and initial token values impact the structural dynamics of token clusters. Our analysis demonstrates that while the clusters within a modified attention matrix dynamics can exhibit significant divergence from the original over extended periods, they maintain close similarities over shorter intervals, depending on the parameter differences. This work contributes to the fine-tuning field through practical applications to the LoRA algorithm \cite{hu2021lora,peft}, enhancing our understanding of the behavior of LoRA-enhanced Transformer models.