Learning Stochastic Optimal Policies via Gradient Descent
This work addresses control problems in systems with stochastic dynamics, offering a more flexible approach for applications like finance, though it appears incremental as an extension of existing gradient-based methods.
The paper tackles stochastic optimal control by directly optimizing parametric control policies via gradient descent, extending classical techniques to less restrictive assumptions, and demonstrates performance on a portfolio optimization problem with proportional transaction costs.
We systematically develop a learning-based treatment of stochastic optimal control (SOC), relying on direct optimization of parametric control policies. We propose a derivation of adjoint sensitivity results for stochastic differential equations through direct application of variational calculus. Then, given an objective function for a predetermined task specifying the desiderata for the controller, we optimize their parameters via iterative gradient descent methods. In doing so, we extend the range of applicability of classical SOC techniques, often requiring strict assumptions on the functional form of system and control. We verify the performance of the proposed approach on a continuous-time, finite horizon portfolio optimization with proportional transaction costs.