Automatic Grid Updates for Kolmogorov-Arnold Networks using Layer Histograms
This addresses a practical training overhead issue for researchers and practitioners using KANs, though it appears incremental as an enhancement to existing KAN methodology.
The paper tackles the problem of manual domain grid adjustments required during Kolmogorov-Arnold Network (KAN) training by introducing AdaptKAN, which uses layer histograms to automatically update domains in a data-driven manner. Results show AdaptKAN matches or exceeds prior KAN architectures and MLPs on tasks including learning scientific equations, image classification, control Lyapunov functions, and OOD detection.
Kolmogorov-Arnold Networks (KANs) are a class of neural networks that have received increased attention in recent literature. In contrast to MLPs, KANs leverage parameterized, trainable activation functions and offer several benefits including improved interpretability and higher accuracy on learning symbolic equations. However, the original KAN architecture requires adjustments to the domain discretization of the network (called the "domain grid") during training, creating extra overhead for the user in the training process. Typical KAN layers are not designed with the ability to autonomously update their domains in a data-driven manner informed by the changing output ranges of previous layers. As an added benefit, this histogram algorithm may also be applied towards detecting out-of-distribution (OOD) inputs in a variety of settings. We demonstrate that AdaptKAN exceeds or matches the performance of prior KAN architectures and MLPs on four different tasks: learning scientific equations from the Feynman dataset, image classification from frozen features, learning a control Lyapunov function, and detecting OOD inputs on the OpenOOD v1.5 benchmark.