Diving into Kronecker Adapters: Component Design Matters
This work addresses the need for more efficient fine-tuning methods in machine learning, but it is incremental as it builds on existing Kronecker adapter approaches.
The paper tackled the problem of optimizing Kronecker adapters for fine-tuning large models by analyzing how component structure affects capacity and alignment with full fine-tuning, resulting in the proposed CDKA method that shows effectiveness across NLP tasks.
Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget-aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various natural language processing tasks demonstrate the effectiveness of CDKA. Code is available at https://github.com/rainstonee/CDKA.