HOFT: Householder Orthogonal Fine-tuning
This work addresses efficiency issues in adapting foundation models for practitioners, though it appears incremental as it builds on existing orthogonal fine-tuning paradigms.
The authors tackled the inefficiency of orthogonal fine-tuning methods for foundation models by proposing HOFT and SHOFT, which reduce time and space complexity while achieving comparable or better results in tasks like commonsense reasoning and machine translation.
Adaptation of foundation models using low-rank methods is a widespread approach. Another way to adapt these models is to employ orthogonal fine-tuning methods, which are less time and memory efficient despite their good generalization properties. In this work, we propose Householder Orthogonal Fine-tuning (HOFT), a novel orthogonal fine-tuning method that aims to alleviate time and space complexity. Moreover, some theoretical properties of the orthogonal fine-tuning paradigm are explored. From this exploration, Scaled Householder Orthogonal Fine-tuning (SHOFT) is proposed. Both HOFT and SHOFT are evaluated in downstream tasks, namely commonsense reasoning, machine translation, subject-driven generation and mathematical reasoning. Compared with state-of-the-art adaptation methods, HOFT and SHOFT show comparable or better results.