Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting
This addresses a limitation in IRL for modeling human-like agents, but it is incremental as it extends existing identifiability characterizations to non-exponential discounting.
The paper tackles the problem of inferring agent preferences from observed behavior in inverse reinforcement learning (IRL) for agents with non-exponential discounting, such as hyperbolic discounting, and shows that IRL generally cannot infer enough information to identify the correct optimal policy, indicating its insufficiency for adequately characterizing preferences.
The aim of inverse reinforcement learning (IRL) is to infer an agent's preferences from observing their behaviour. Usually, preferences are modelled as a reward function, $R$, and behaviour is modelled as a policy, $π$. One of the central difficulties in IRL is that multiple preferences may lead to the same observed behaviour. That is, $R$ is typically underdetermined by $π$, which means that $R$ is only partially identifiable. Recent work has characterised the extent of this partial identifiability for different types of agents, including optimal and Boltzmann-rational agents. However, work so far has only considered agents that discount future reward exponentially: this is a serious limitation, especially given that extensive work in the behavioural sciences suggests that humans are better modelled as discounting hyperbolically. In this work, we newly characterise partial identifiability in IRL for agents with non-exponential discounting: our results are in particular relevant for hyperbolical discounting, but they also more generally apply to agents that use other types of (non-exponential) discounting. We significantly show that generally IRL is unable to infer enough information about $R$ to identify the correct optimal policy, which entails that IRL alone can be insufficient to adequately characterise the preferences of such agents.