Convergence and Generalization of Anti-Regularization for Parametric Models
This addresses the problem of underfitting in limited-data scenarios for practitioners using parametric models, though it appears incremental as an enhancement to standard empirical risk minimization.
The paper tackles the problem of underfitting in small-sample regimes by introducing Anti-regularization, which amplifies model expressivity through a reversed-sign reward term that decays with sample size. Empirical results show it mitigates underfitting in regression and classification while preserving generalization and improving calibration.
Anti-regularization introduces a reward term with a reversed sign into the loss function, deliberately amplifying model expressivity in small-sample regimes while ensuring that the intervention gradually vanishes as the sample size grows through a power-law decay schedule. We formalize spectral safety conditions and trust-region constraints, and we design a lightweight safeguard that combines a projection operator with gradient clipping to guarantee stable intervention. Theoretical analysis extends to linear smoothers and the Neural Tangent Kernel regime, providing practical guidance on the choice of decay exponents through the balance between empirical risk and variance. Empirical results show that Anti-regularization mitigates underfitting in both regression and classification while preserving generalization and improving calibration. Ablation studies confirm that the decay schedule and safeguards are essential to avoiding overfitting and instability. As an alternative, we also propose a degrees-of-freedom targeting schedule that maintains constant per-sample complexity. Anti-regularization constitutes a simple and reproducible procedure that integrates seamlessly into standard empirical risk minimization pipelines, enabling robust learning under limited data and resource constraints by intervening only when necessary and vanishing otherwise.