Improving Memory Efficiency for Training KANs via Meta Learning
This work addresses a key scalability bottleneck for researchers and practitioners using KANs, offering a more memory-efficient training method that narrows the cost gap with traditional MLPs, though it is incremental as it builds directly on the KAN framework.
The paper tackles the memory inefficiency and high training costs of Kolmogorov-Arnold Networks (KANs) by proposing MetaKANs, a meta-learning approach that generates KAN weights via a smaller meta-learner, achieving comparable or superior performance while significantly reducing trainable parameters and memory usage across tasks like symbolic regression and image classification.
Inspired by the Kolmogorov-Arnold representation theorem, KANs offer a novel framework for function approximation by replacing traditional neural network weights with learnable univariate functions. This design demonstrates significant potential as an efficient and interpretable alternative to traditional MLPs. However, KANs are characterized by a substantially larger number of trainable parameters, leading to challenges in memory efficiency and higher training costs compared to MLPs. To address this limitation, we propose to generate weights for KANs via a smaller meta-learner, called MetaKANs. By training KANs and MetaKANs in an end-to-end differentiable manner, MetaKANs achieve comparable or even superior performance while significantly reducing the number of trainable parameters and maintaining promising interpretability. Extensive experiments on diverse benchmark tasks, including symbolic regression, partial differential equation solving, and image classification, demonstrate the effectiveness of MetaKANs in improving parameter efficiency and memory usage. The proposed method provides an alternative technique for training KANs, that allows for greater scalability and extensibility, and narrows the training cost gap with MLPs stated in the original paper of KANs. Our code is available at https://github.com/Murphyzc/MetaKAN.