Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks
This work addresses the problem of efficient transfer learning for KANs, which is significant for researchers and practitioners working with KANs, particularly in science-related domains, and provides an incremental improvement over existing methods.
The authors tackled the problem of transfer learning for Kolmogorov--Arnold networks (KANs) and achieved efficient fine-tuning using low tensor-rank adaptation (LoTRA), resulting in reduced model size while maintaining performance. Experimental results validated the efficacy of LoTRA for solving partial differential equations (PDEs) and other tasks.
Kolmogorov--Arnold networks (KANs) have demonstrated their potential as an alternative to multi-layer perceptions (MLPs) in various domains, especially for science-related tasks. However, transfer learning of KANs remains a relatively unexplored area. In this paper, inspired by Tucker decomposition of tensors and evidence on the low tensor-rank structure in KAN parameter updates, we develop low tensor-rank adaptation (LoTRA) for fine-tuning KANs. We study the expressiveness of LoTRA based on Tucker decomposition approximations. Furthermore, we provide a theoretical analysis to select the learning rates for each LoTRA component to enable efficient training. Our analysis also shows that using identical learning rates across all components leads to inefficient training, highlighting the need for an adaptive learning rate strategy. Beyond theoretical insights, we explore the application of LoTRA for efficiently solving various partial differential equations (PDEs) by fine-tuning KANs. Additionally, we propose Slim KANs that incorporate the inherent low-tensor-rank properties of KAN parameter tensors to reduce model size while maintaining superior performance. Experimental results validate the efficacy of the proposed learning rate selection strategy and demonstrate the effectiveness of LoTRA for transfer learning of KANs in solving PDEs. Further evaluations on Slim KANs for function representation and image classification tasks highlight the expressiveness of LoTRA and the potential for parameter reduction through low tensor-rank decomposition.