Implicit Regularization with Polynomial Growth in Deep Tensor Factorization
This provides theoretical insight into deep learning regularization for tensor problems, though it appears incremental relative to prior work on matrix and shallow tensor factorization.
The paper investigates implicit regularization in deep tensor factorization, showing that its effect grows polynomially with network depth rather than quadratically as in shallow cases, which accurately matches experimental observations and improves estimation accuracy and convergence.
We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and 'shallow' tensor factorization via linear and certain type of non-linear neural networks promotes low-rank solutions with at most quadratic growth, we show that its effect in deep tensor factorization grows polynomially with the depth of the network. This provides a remarkably faithful description of the observed experimental behaviour. Using numerical experiments, we demonstrate the benefits of this implicit regularization in yielding a more accurate estimation and better convergence properties.