LoCO: Low-rank Compositional Rotation Fine-tuning
This work addresses the need for preserving geometric structure in pretrained representations during fine-tuning, offering a practical orthogonal PEFT method for high-dimensional feature spaces.
LoCO introduces a parameter-efficient fine-tuning method that uses low-rank skew-symmetric matrices and compositional rotation chains to construct orthogonal transformations, achieving superior or competitive performance across diffusion transformers, vision transformers, and language models while maintaining low computational complexity.
Parameter-efficient fine-tuning (PEFT) has emerged as an critical technique for adapting large-scale foundation models across natural language processing and computer vision. While existing methods such as low-rank adaptations achieve parameter efficiency via low-rank weight updates, they are limited in their ability to preserve the geometric structure of pretrained representations. We introduce Low-rank Compositional Orthogonal fine-tuning (LoCO), a novel PEFT method that constructs orthogonal transformations through low-rank skew-symmetric matrices and compositional rotation chains. We propose an approximation scheme that enables fully parallel computation of compositional rotations, making the approach practical for high-dimensional feature spaces. Our method maintains low computational complexity while maintaining orthogonality with controlled approximation error. We validate LoCO across diverse domains, including diffusion transformer fine-tuning, vision transformer adaptation, and language model adaptation. Our method demonstrates superior or competitive performance compared to both existing orthogonal and non-orthogonal methods.