The Implicit Bias of Logit Regularization
It provides theoretical insights into a widely used technique for improving calibration and generalization in classifiers, though it is incremental as it builds on existing understanding of logit regularization.
The paper analyzes logit regularization in linear classification, showing it induces an implicit bias of logit clustering that aligns the weight vector with Fisher's Linear Discriminant. This leads to halving the critical sample complexity and inducing grokking in a signal-plus-noise model, while improving generalization robustness to noise.
Logit regularization, the addition of a convex penalty directly in logit space, is widely used in modern classifiers, with label smoothing as a prominent example. While such methods often improve calibration and generalization, their mechanism remains under-explored. In this work, we analyze a general class of such logit regularizers in the context of linear classification, and demonstrate that they induce an implicit bias of logit clustering around finite per-sample targets. For Gaussian data, or whenever logits are sufficiently clustered, we prove that logit clustering drives the weight vector to align exactly with Fisher's Linear Discriminant. To demonstrate the consequences, we study a simple signal-plus-noise model in which this transition has dramatic effects: Logit regularization halves the critical sample complexity and induces grokking in the small-noise limit, while making generalization robust to noise. Our results extend the theoretical understanding of label smoothing and highlight the efficacy of a broader class of logit-regularization methods.