Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning
For theorists studying optimization and implicit bias in neural networks, this provides a unified framework linking mirror maps to feature learning sparsity.
This paper extends max-margin analysis of gradient flow to mirror flow in homogeneous neural networks, deriving a balance equation and characterizing the horizon function. It shows that different mirror maps can yield the same max-margin solution, convergence can be exponentially slow, and representations vary from sparse to dense.
We study the max-margin solutions reached by mirror flow in deep neural networks with homogeneous activation functions. Extending classical results on gradient flow, we derive a novel balance equation for mirror flow from convex duality, enabling a characterization of the horizon function governing the induced margin. We further establish max-margin characterizations together with convergence rates and norm growth estimates. Finally, we support our theory through experiments on synthetic datasets and standard vision tasks. Concretely, we show that: (1) distinct non-homogeneous mirror maps can induce the same max-margin solution; (2) convergence can be extremely slow, including exponentially slow regimes; and (3) although all considered mirror maps exhibit feature learning, they can produce markedly different representations, ranging from sparse to dense neuron activations. Together, these results provide a unified perspective on sparse and dense feature learning in homogeneous neural networks, highlighting how mirror maps shape both optimization dynamics and the geometry of the learned classifiers.