Null-LoRA: Low-Rank Adaptation on Null Space
This work addresses the need for more efficient fine-tuning methods in machine learning, particularly for adapting pre-trained models to downstream tasks, though it appears incremental as it builds on existing LoRA techniques.
The paper tackles the problem of parameter-efficient fine-tuning for large-scale models by proposing Null-LoRA, which performs low-rank adaptation in the null space to reduce redundancy and enhance effective rank. It achieves state-of-the-art results with fewer parameters in image-text retrieval and visual question answering tasks.
Parameter-efficient fine-tuning methods have gained considerable popularity for adapting large-scale models to downstream tasks, particularly LoRA and its variants. Existing methods perform low-rank adaptation over the full parameter space. However, fine-tuning within a subspace can achieve comparable effectiveness. Inspired by the observation that pre-trained models possess non-trivial null spaces, we propose Null-space based Low-Rank Adaptation (Null-LoRA). Null-LoRA effectively reduces redundancy and enhances effective rank by freezing portions of the low-rank matrices. To further improve parameter efficiency, Null-LoRA constrains the entire incremental update within the null space, maximizing the utilization of incremental updates to adapt to new task paradigms. Null-LoRA surpasses the state of the art with fewer parameters in extensive experiments across image-text retrieval and visual question answering tasks.