Implicit Bias in Deep Linear Discriminant Analysis
This provides theoretical insights into optimization geometry for discriminative metric-learning, which is incremental as it extends prior work on implicit bias to a new objective.
The paper tackles the problem of understanding implicit regularization in deep linear discriminant analysis (LDA), proving that under balanced initialization, the network architecture leads to multiplicative weight updates that conserve a specific quasi-norm.
While the Implicit Bias(or Implicit Regularization) of standard loss functions has been studied, the optimization geometry induced by discriminative metric-learning objectives remains largely unexplored.To the best of our knowledge, this paper presents an initial theoretical analysis of the implicit regularization induced by the Deep LDA,a scale invariant objective designed to minimize intraclass variance and maximize interclass distance. By analyzing the gradient flow of the loss on a L-layer diagonal linear network, we prove that under balanced initialization, the network architecture transforms standard additive gradient updates into multiplicative weight updates, which demonstrates an automatic conservation of the (2/L) quasi-norm.