Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
This work addresses a bottleneck in parameter-efficient fine-tuning for large language models, offering an incremental enhancement to existing methods.
The paper tackles the limited expressiveness of Low-Rank Adaptation (LoRA) in fine-tuning large language models by proposing LoRAN, a non-linear extension with structured activations like Sinter, resulting in consistent improvements over QLoRA in summarization and classification tasks.
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates. We further introduce Sinter, a sine-based activation that adds structured perturbations without increasing parameter count. Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA. Ablation studies reveal that Sinter outperforms standard activations such as Sigmoid, ReLU, and Tanh, highlighting the importance of activation design in lowrank tuning.