Quantifying and Optimizing Simplicity via Polynomial Representations
This work provides a practical, quantitative simplicity measure and regularizer that improves generalization across diverse domains, addressing the need for a broadly applicable simplicity metric in deep learning.
The authors propose polynomial representations as a distribution-aware simplicity metric for neural networks, showing it predicts generalization better than existing proxies like sharpness. Their differentiable regularizer based on this metric consistently improves generalization across image classification, text classification, fine-tuning vision-language models, and reinforcement learning.
Deep networks often exhibit a preference for "simple" solutions, and such a simplicity bias is widely believed to play a key role in generalization. Yet a broadly applicable, quantitative measure of simplicity remains elusive. We introduce polynomial representations as a distribution-aware, low-dimensional surrogate for neural functions: we approximate a network's predictive behavior along data-dependent interpolation paths using orthogonal polynomial bases, yielding a compact functional representation. We show that the effective degree of this representation serves as a practical simplicity metric that is predictive of generalization across tasks and architectures, and consistently outperforms existing generalization proxies such as sharpness. Finally, polynomial representations naturally yield a differentiable simplicity regularizer, which consistently improves generalization in image and text classification, fine-tuning contrastive vision-language models, and reinforcement learning.