Function Spaces Without Kernels: Learning Compact Hilbert Space Representations
This work addresses the need for efficient and principled neural predictors with kernel-level guarantees, offering incremental improvements in scalability and adaptability for machine learning applications.
The paper tackled the problem of learning compact Hilbert space representations for neural networks by connecting function encoders to kernel methods, enabling scalable and adaptive models. It demonstrated that the same accuracy could be achieved with substantially fewer basis functions in benchmarks like polynomial tasks and nonlinear dynamical systems.
Function encoders are a recent technique that learn neural network basis functions to form compact, adaptive representations of Hilbert spaces of functions. We show that function encoders provide a principled connection to feature learning and kernel methods by defining a kernel through an inner product of the learned feature map. This kernel-theoretic perspective explains their ability to scale independently of dataset size while adapting to the intrinsic structure of data, and it enables kernel-style analysis of neural models. Building on this foundation, we develop two training algorithms that learn compact bases: a progressive training approach that constructively grows bases, and a train-then-prune approach that offers a computationally efficient alternative after training. Both approaches use principles from PCA to reveal the intrinsic dimension of the learned space. In parallel, we derive finite-sample generalization bounds using Rademacher complexity and PAC-Bayes techniques, providing inference time guarantees. We validate our approach on a polynomial benchmark with a known intrinsic dimension, and on nonlinear dynamical systems including a Van der Pol oscillator and a two-body orbital model, demonstrating that the same accuracy can be achieved with substantially fewer basis functions. This work suggests a path toward neural predictors with kernel-level guarantees, enabling adaptable models that are both efficient and principled at scale.