Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints
This work addresses the subtle mechanisms of implicit bias in gradient-based optimization for machine learning practitioners, but it appears incremental as it builds on existing frameworks like Normalized Steepest Descent.
The paper tackles the problem of understanding implicit bias in overparameterized models on multiclass separable data by introducing NucGD, a geometry-aware optimizer that enforces low-rank structures through nuclear norm constraints, and shows it connects with low-rank projection methods while enabling scalable training via an efficient SVD-free update rule.
Implicit bias induced by gradient-based algorithms is essential to the generalization of overparameterized models, yet its mechanisms can be subtle. This work leverages the Normalized Steepest Descent} (NSD) framework to investigate how optimization geometry shapes solutions on multiclass separable data. We introduce NucGD, a geometry-aware optimizer designed to enforce low rank structures through nuclear norm constraints. Beyond the algorithm itself, we connect NucGD with emerging low-rank projection methods, providing a unified perspective. To enable scalable training, we derive an efficient SVD-free update rule via asynchronous power iteration. Furthermore, we empirically dissect the impact of stochastic optimization dynamics, characterizing how varying levels of gradient noise induced by mini-batch sampling and momentum modulate the convergence toward the expected maximum margin solutions.Our code is accessible at: https://github.com/Tsokarsic/observing-the-implicit-bias-on-multiclass-seperable-data.