EVAL: EigenVector-based Average-reward Learning
This work addresses average-reward RL for improved robustness and exploration, but it is incremental as it builds on existing entropy-regularized methods.
The paper tackles the entropy-regularized average-reward reinforcement learning problem by extending a tabular method to neural network function approximation, revealing theoretical insights and combining it with posterior policy iteration to handle non-regularized cases. Experimentally, it shows favorable stability and convergence rates on classic control benchmarks.
In reinforcement learning, two objective functions have been developed extensively in the literature: discounted and averaged rewards. The generalization to an entropy-regularized setting has led to improved robustness and exploration for both of these objectives. Recently, the entropy-regularized average-reward problem was addressed using tools from large deviation theory in the tabular setting. This method has the advantage of linearity, providing access to both the optimal policy and average reward-rate through properties of a single matrix. In this paper, we extend that framework to more general settings by developing approaches based on function approximation by neural networks. This formulation reveals new theoretical insights into the relationship between different objectives used in RL. Additionally, we combine our algorithm with a posterior policy iteration scheme, showing how our approach can also solve the average-reward RL problem without entropy-regularization. Using classic control benchmarks, we experimentally find that our method compares favorably with other algorithms in terms of stability and rate of convergence.