The structure of conservative gradient fields
This work provides a more complete mathematical understanding of generalized gradients for researchers working on the theoretical foundations of automatic differentiation and optimization in nonsmooth deep learning.
The paper investigates conservative gradient fields, which are enlarged generalized gradients useful for analyzing automatic differentiation in nonsmooth contexts and the convergence of deep learning algorithms. For semi-algebraic functions, the authors demonstrate that all conservative fields can be expressed as the sum of Clarke subdifferentials and normals to manifolds within Whitney stratifications.
The classical Clarke subdifferential alone is inadequate for understanding automatic differentiation in nonsmooth contexts. Instead, we can sometimes rely on enlarged generalized gradients called "conservative fields", defined through the natural path-wise chain rule: one application is the convergence analysis of gradient-based deep learning algorithms. In the semi-algebraic case, we show that all conservative fields are in fact just Clarke subdifferentials plus normals of manifolds in underlying Whitney stratifications.