MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network
This addresses the memory inefficiency of KANs for AI practitioners, though it is incremental as it builds on existing KAN methods.
The paper tackles the high parameter storage of Kolmogorov-Arnold Networks (KANs) by proposing MetaCluster, a framework that compresses KANs using clustering and meta-learning, achieving up to 80x reduction in storage without accuracy loss on datasets like MNIST and CIFAR.
Kolmogorov-Arnold Networks (KANs) replace scalar weights with per-edge vectors of basis coefficients, thereby boosting expressivity and accuracy but at the same time resulting in a multiplicative increase in parameters and memory. We propose MetaCluster, a framework that makes KANs highly compressible without sacrificing accuracy. Specifically, a lightweight meta-learner, trained jointly with the KAN, is used to map low-dimensional embedding to coefficient vectors, shaping them to lie on a low-dimensional manifold that is amenable to clustering. We then run K-means in coefficient space and replace per-edge vectors with shared centroids. Afterwards, the meta-learner can be discarded, and a brief fine-tuning of the centroid codebook recovers any residual accuracy loss. The resulting model stores only a small codebook and per-edge indices, exploiting the vector nature of KAN parameters to amortize storage across multiple coefficients. On MNIST, CIFAR-10, and CIFAR-100, across standard KANs and ConvKANs using multiple basis functions, MetaCluster achieves a reduction of up to 80$\times$ in parameter storage, with no loss in accuracy. Code will be released upon publication.