Offline Policy Comparison under Limited Historical Agent-Environment Interactions
This addresses a practical challenge for real-world RL applications where data constraints hinder accurate evaluation, offering an incremental improvement over existing methods.
The paper tackles the problem of biased policy evaluation in reinforcement learning when historical data is limited, proposing policy comparison as an alternative and introducing the Limited Data Estimator (LDE) for reliable ranking, with theoretical and experimental validation.
We address the challenge of policy evaluation in real-world applications of reinforcement learning systems where the available historical data is limited due to ethical, practical, or security considerations. This constrained distribution of data samples often leads to biased policy evaluation estimates. To remedy this, we propose that instead of policy evaluation, one should perform policy comparison, i.e. to rank the policies of interest in terms of their value based on available historical data. In addition we present the Limited Data Estimator (LDE) as a simple method for evaluating and comparing policies from a small number of interactions with the environment. According to our theoretical analysis, the LDE is shown to be statistically reliable on policy comparison tasks under mild assumptions on the distribution of the historical data. Additionally, our numerical experiments compare the LDE to other policy evaluation methods on the task of policy ranking and demonstrate its advantage in various settings.