On Training of Kolmogorov-Arnold Networks
This work addresses training stability issues for researchers using KANs, but it is incremental as it builds on recent introductions of KANs.
The paper tackled the training dynamics of Kolmogorov-Arnold Networks (KANs) compared to MLPs, finding that KANs are an effective alternative with better parameter efficiency on high-dimensional datasets but suffer from more unstable training.
Kolmogorov-Arnold Networks have recently been introduced as a flexible alternative to multi-layer Perceptron architectures. In this paper, we examine the training dynamics of different KAN architectures and compare them with corresponding MLP formulations. We train with a variety of different initialization schemes, optimizers, and learning rates, as well as utilize back propagation free approaches like the HSIC Bottleneck. We find that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics. Finally, we provide recommendations for improving training stability of larger KAN models.