A Granger-Causal Perspective on Gradient Descent with Application to Pruning
This provides a causal perspective for controlling gradient descent, with applications like pruning, though it appears incremental in exploring causality within existing optimization methods.
The paper tackles the causality aspect of gradient descent in neural networks, showing it has an implicit Granger-causal relationship between loss reduction and parameter changes, and applies this to pruning, where they observe a phase shift indicating optimal pruning and flatter minima leading to increased accuracy.
Stochastic Gradient Descent (SGD) is the main approach to optimizing neural networks. Several generalization properties of deep networks, such as convergence to a flatter minima, are believed to arise from SGD. This article explores the causality aspect of gradient descent. Specifically, we show that the gradient descent procedure has an implicit granger-causal relationship between the reduction in loss and a change in parameters. By suitable modifications, we make this causal relationship explicit. A causal approach to gradient descent has many significant applications which allow greater control. In this article, we illustrate the significance of the causal approach using the application of Pruning. The causal approach to pruning has several interesting properties - (i) We observe a phase shift as the percentage of pruned parameters increase. Such phase shift is indicative of an optimal pruning strategy. (ii) After pruning, we see that minima becomes "flatter", explaining the increase in accuracy after pruning weights.