Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks
This work addresses the need for efficient learning in deep networks by potentially eliminating depth requirements, though it is incremental in combining known regularization techniques.
The paper tackled the problem of achieving low-rank solutions in matrix completion tasks without requiring deep networks by combining explicit regularization with adaptive gradient optimizers. The result was that a single-layer network achieved generalization error comparable to deep linear networks and performed competitively or outperformed other approaches across various parameter and data regimes.
Works on implicit regularization have studied gradient trajectories during the optimization process to explain why deep networks favor certain kinds of solutions over others. In deep linear networks, it has been shown that gradient descent implicitly regularizes toward low-rank solutions on matrix completion/factorization tasks. Adding depth not only improves performance on these tasks but also acts as an accelerative pre-conditioning that further enhances this bias towards low-rankedness. Inspired by this, we propose an explicit penalty to mirror this implicit bias which only takes effect with certain adaptive gradient optimizers (e.g. Adam). This combination can enable a degenerate single-layer network to achieve low-rank approximations with generalization error comparable to deep linear networks, making depth no longer necessary for learning. The single-layer network also performs competitively or out-performs various approaches for matrix completion over a range of parameter and data regimes despite its simplicity. Together with an optimizer's inductive bias, our findings suggest that explicit regularization can play a role in designing different, desirable forms of regularization and that a more nuanced understanding of this interplay may be necessary.