Action-Driven Processes for Continuous-Time Control
This work provides a theoretical framework for continuous-time control, potentially benefiting researchers in reinforcement learning and spiking neural networks, but it appears incremental as it builds on existing control-as-inference ideas.
The paper tackles the problem of unifying stochastic processes and reinforcement learning through action-driven processes, showing that minimizing KL divergence between policy-driven and reward-driven distributions for these processes is equivalent to maximum entropy reinforcement learning.
At the heart of reinforcement learning are actions -- decisions made in response to observations of the environment. Actions are equally fundamental in the modeling of stochastic processes, as they trigger discontinuous state transitions and enable the flow of information through large, complex systems. In this paper, we unify the perspectives of stochastic processes and reinforcement learning through action-driven processes, and illustrate their application to spiking neural networks. Leveraging ideas from control-as-inference, we show that minimizing the Kullback-Leibler divergence between a policy-driven true distribution and a reward-driven model distribution for a suitably defined action-driven process is equivalent to maximum entropy reinforcement learning.