No More DeLuLu: Physics-Inspired Kernel Networks for Geometrically-Grounded Neural Computation
This work addresses the need for more principled and simplified neural architectures in machine learning, offering a novel kernel-based approach that could impact various AI applications, though it appears incremental in its current empirical validation.
The paper tackled the problem of simplifying neural network architectures by introducing the yat-product kernel, which combines quadratic alignment with inverse-square proximity, and used it to create Neural Matter Networks (NMNs) that replace traditional blocks with a single geometrically-grounded operation. The result showed that NMN-based classifiers matched linear baselines on MNIST and Aether-GPT2 achieved lower validation loss than GPT-2 with comparable parameters in language modeling.
We introduce the yat-product, a kernel operator combining quadratic alignment with inverse-square proximity. We prove it is a Mercer kernel, analytic, Lipschitz on bounded domains, and self-regularizing, admitting a unique RKHS embedding. Neural Matter Networks (NMNs) use yat-product as the sole non-linearity, replacing conventional linear-activation-normalization blocks with a single geometrically-grounded operation. This architectural simplification preserves universal approximation while shifting normalization into the kernel itself via the denominator, rather than relying on separate normalization layers. Empirically, NMN-based classifiers match linear baselines on MNIST while exhibiting bounded prototype evolution and superposition robustness. In language modeling, Aether-GPT2 achieves lower validation loss than GPT-2 with a comparable parameter budget while using yat-based attention and MLP blocks. Our framework unifies kernel learning, gradient stability, and information geometry, establishing NMNs as a principled alternative to conventional neural architectures.