Differentiable Programming à la Moreau
This work provides a foundational mathematical framework for integrating Moreau envelopes into differentiable programming, which could benefit researchers developing and analyzing first-order optimization algorithms for deep learning.
This paper introduces a compositional calculus for Moreau envelopes, enabling their integration into differentiable programming. This framework allows for the mathematical optimization interpretation of various gradient back-propagation methods, particularly those involving virtual targets.
The notion of a Moreau envelope is central to the analysis of first-order optimization algorithms for machine learning. Yet, it has not been developed and extended to be applied to a deep network and, more broadly, to a machine learning system with a differentiable programming implementation. We define a compositional calculus adapted to Moreau envelopes and show how to integrate it within differentiable programming. The proposed framework casts in a mathematical optimization framework several variants of gradient back-propagation related to the idea of the propagation of virtual targets.