The Implicit Bias of Gradient Descent on Separable Multiclass Data
This work addresses a theoretical problem for researchers in machine learning by bridging the binary-multiclass gap in implicit bias analysis, though it is incremental as it extends existing frameworks and proofs.
The paper tackles the gap in theoretical analysis of implicit bias for multiclass classification by extending the exponential tail property to a broader class of losses, including cross-entropy, and proves that gradient descent on separable data converges to a maximum-margin solution, generalizing prior binary results to multiclass settings.
Implicit bias describes the phenomenon where optimization-based training algorithms, without explicit regularization, show a preference for simple estimators even when more complex estimators have equal objective values. Multiple works have developed the theory of implicit bias for binary classification under the assumption that the loss satisfies an exponential tail property. However, there is a noticeable gap in analysis for multiclass classification, with only a handful of results which themselves are restricted to the cross-entropy loss. In this work, we employ the framework of Permutation Equivariant and Relative Margin-based (PERM) losses [Wang and Scott, 2024] to introduce a multiclass extension of the exponential tail property. This class of losses includes not only cross-entropy but also other losses. Using this framework, we extend the implicit bias result of Soudry et al. [2018] to multiclass classification. Furthermore, our proof techniques closely mirror those of the binary case, thus illustrating the power of the PERM framework for bridging the binary-multiclass gap.