Categorical Foundations of Gradient-Based Learning
This provides a foundational framework for understanding and comparing gradient-based learning methods, which is incremental in offering a new theoretical perspective.
The paper tackles the problem of unifying gradient-based machine learning algorithms by proposing a categorical semantics framework using lenses, parametrised maps, and reverse derivative categories, which encompasses various algorithms like ADAM and loss functions like MSE, and demonstrates its practical significance with a Python implementation.
We propose a categorical semantics of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach to gradient-based learning has examples generalising beyond the familiar continuous domains (modelled in categories of smooth maps) and can be realized in the discrete setting of boolean circuits. Finally, we demonstrate the practical significance of our framework with an implementation in Python.