Value Driven Representation for Human-in-the-Loop Reinforcement Learning
This addresses the challenge for system designers in interactive adaptive systems like intelligent tutoring, though it is incremental as it builds on existing RL methods.
The paper tackles the problem of helping system designers choose observation spaces for human-in-the-loop reinforcement learning systems, presenting the value driven representation (VDR) algorithm that iteratively augments the observation space to capture near-optimal policies, with evaluation on standard benchmarks showing significant improvement over prior baselines.
Interactive adaptive systems powered by Reinforcement Learning (RL) have many potential applications, such as intelligent tutoring systems. In such systems there is typically an external human system designer that is creating, monitoring and modifying the interactive adaptive system, trying to improve its performance on the target outcomes. In this paper we focus on algorithmic foundation of how to help the system designer choose the set of sensors or features to define the observation space used by reinforcement learning agent. We present an algorithm, value driven representation (VDR), that can iteratively and adaptively augment the observation space of a reinforcement learning agent so that is sufficient to capture a (near) optimal policy. To do so we introduce a new method to optimistically estimate the value of a policy using offline simulated Monte Carlo rollouts. We evaluate the performance of our approach on standard RL benchmarks with simulated humans and demonstrate significant improvement over prior baselines.