Reimagining Linear Probing: Kolmogorov-Arnold Networks in Transfer Learning
It addresses the problem of modeling complex relationships in data for transfer learning practitioners, offering an incremental enhancement over existing methods.
This paper tackled the limitation of linear probing in transfer learning by replacing it with Kolmogorov-Arnold Networks (KAN), resulting in consistent performance improvements on CIFAR-10 with a ResNet-50 model.
This paper introduces Kolmogorov-Arnold Networks (KAN) as an enhancement to the traditional linear probing method in transfer learning. Linear probing, often applied to the final layer of pre-trained models, is limited by its inability to model complex relationships in data. To address this, we propose substituting the linear probing layer with KAN, which leverages spline-based representations to approximate intricate functions. In this study, we integrate KAN with a ResNet-50 model pre-trained on ImageNet and evaluate its performance on the CIFAR-10 dataset. We perform a systematic hyperparameter search, focusing on grid size and spline degree (k), to optimize KAN's flexibility and accuracy. Our results demonstrate that KAN consistently outperforms traditional linear probing, achieving significant improvements in accuracy and generalization across a range of configurations. These findings indicate that KAN offers a more powerful and adaptable alternative to conventional linear probing techniques in transfer learning.