Off-Policy Evaluation for Action-Dependent Non-Stationary Environments
It addresses a fundamental challenge in sequential decision-making for applications affected by non-stationarity, representing an incremental step by building on existing evaluation methods.
The paper tackles the problem of evaluating policies in non-stationary environments where changes occur due to external factors or interactions, proposing the OPEN algorithm to estimate past performance changes with lower bias and variance, and shows promising results for predicting future performances in real-world inspired domains.
Methods for sequential decision-making are often built upon a foundational assumption that the underlying decision process is stationary. This limits the application of such methods because real-world problems are often subject to changes due to external factors (passive non-stationarity), changes induced by interactions with the system itself (active non-stationarity), or both (hybrid non-stationarity). In this work, we take the first steps towards the fundamental challenge of on-policy and off-policy evaluation amidst structured changes due to active, passive, or hybrid non-stationarity. Towards this goal, we make a higher-order stationarity assumption such that non-stationarity results in changes over time, but the way changes happen is fixed. We propose, OPEN, an algorithm that uses a double application of counterfactual reasoning and a novel importance-weighted instrument-variable regression to obtain both a lower bias and a lower variance estimate of the structure in the changes of a policy's past performances. Finally, we show promising results on how OPEN can be used to predict future performances for several domains inspired by real-world applications that exhibit non-stationarity.