Active Measure Reinforcement Learning for Observation Cost Minimization
This addresses a practical issue in sequential decision-making tasks like medical treatment, where observation costs matter, but it is an incremental improvement over existing RL methods.
The paper tackles the problem of minimizing observation costs in reinforcement learning by proposing the active measure RL framework (Amrl), where agents learn to maximize a costed return balancing rewards and costs; results show that Amrl-Q achieves a higher costed return than standard methods while learning at a similar rate.
Standard reinforcement learning (RL) algorithms assume that the observation of the next state comes instantaneously and at no cost. In a wide variety of sequential decision making tasks ranging from medical treatment to scientific discovery, however, multiple classes of state observations are possible, each of which has an associated cost. We propose the active measure RL framework (Amrl) as an initial solution to this problem where the agent learns to maximize the costed return, which we define as the discounted sum of rewards minus the sum of observation costs. Our empirical evaluation demonstrates that Amrl-Q agents are able to learn a policy and state estimator in parallel during online training. During training the agent naturally shifts from its reliance on costly measurements of the environment to its state estimator in order to increase its reward. It does this without harm to the learned policy. Our results show that the Amrl-Q agent learns at a rate similar to standard Q-learning and Dyna-Q. Critically, by utilizing an active strategy, Amrl-Q achieves a higher costed return.