Multilevel Training for Kolmogorov Arnold Networks
This work significantly improves the training efficiency and accuracy of Kolmogorov-Arnold networks, particularly benefiting researchers and practitioners using KANs for tasks like physics-informed neural networks, by providing a structured and faster training methodology.
This paper introduces a multilevel training approach for Kolmogorov-Arnold networks (KANs) by exploiting their structured nature. It establishes an equivalence between KANs with spline basis functions and multichannel MLPs, which motivates a multilevel training scheme using uniform refinement of spline knots and analytic geometric interpolation. This method achieves orders of magnitude improvement in accuracy over conventional KAN or MLP training, especially for physics-informed neural networks.
Algorithmic speedup of training common neural architectures is made difficult by the lack of structure guaranteed by the function compositions inherent to such networks. In contrast to multilayer perceptrons (MLPs), Kolmogorov-Arnold networks (KANs) provide more structure by expanding learned activations in a specified basis. This paper exploits this structure to develop practical algorithms and theoretical insights, yielding training speedup via multilevel training for KANs. To do so, we first establish an equivalence between KANs with spline basis functions and multichannel MLPs with power ReLU activations through a linear change of basis. We then analyze how this change of basis affects the geometry of gradient-based optimization with respect to spline knots. The KANs change-of-basis motivates a multilevel training approach, where we train a sequence of KANs naturally defined through a uniform refinement of spline knots with analytic geometric interpolation operators between models. The interpolation scheme enables a ``properly nested hierarchy'' of architectures, ensuring that interpolation to a fine model preserves the progress made on coarse models, while the compact support of spline basis functions ensures complementary optimization on subsequent levels. Numerical experiments demonstrate that our multilevel training approach can achieve orders of magnitude improvement in accuracy over conventional methods to train comparable KANs or MLPs, particularly for physics informed neural networks. Finally, this work demonstrates how principled design of neural networks can lead to exploitable structure, and in this case, multilevel algorithms that can dramatically improve training performance.