Exponential Family Model-Based Reinforcement Learning via Score Matching
This work addresses efficient learning in RL for settings with structured transition models, representing an incremental improvement through the application of score matching to reduce computational complexity.
The paper tackles the problem of model-based reinforcement learning with exponential family transition models by proposing an algorithm called SMRL that uses score matching for efficient parameter estimation, achieving an online regret bound of ̃O(d√(H^3T)).
We propose an optimistic model-based algorithm, dubbed SMRL, for finite-horizon episodic reinforcement learning (RL) when the transition model is specified by exponential family distributions with $d$ parameters and the reward is bounded and known. SMRL uses score matching, an unnormalized density estimation technique that enables efficient estimation of the model parameter by ridge regression. Under standard regularity assumptions, SMRL achieves $\tilde O(d\sqrt{H^3T})$ online regret, where $H$ is the length of each episode and $T$ is the total number of interactions (ignoring polynomial dependence on structural scale parameters).