Rethinking the Function of Neurons in KANs
This work addresses an incremental improvement for researchers and practitioners using KANs by optimizing neuron functions to enhance model performance and stability.
The paper tackled the problem of improving Kolmogorov-Arnold Networks (KANs) by replacing the sum function in neurons with the average function, resulting in significant performance enhancements on benchmark ML tasks and increased training stability.
The neurons of Kolmogorov-Arnold Networks (KANs) perform a simple summation motivated by the Kolmogorov-Arnold representation theorem, which asserts that sum is the only fundamental multivariate function. In this work, we investigate the potential for identifying an alternative multivariate function for KAN neurons that may offer increased practical utility. Our empirical research involves testing various multivariate functions in KAN neurons across a range of benchmark Machine Learning tasks. Our findings indicate that substituting the sum with the average function in KAN neurons results in significant performance enhancements compared to traditional KANs. Our study demonstrates that this minor modification contributes to the stability of training by confining the input to the spline within the effective range of the activation function. Our implementation and experiments are available at: \url{https://github.com/Ghaith81/dropkan}