Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs
This addresses the problem of scaling reinforcement learning for downstream tasks in complex, non-Markovian settings, representing an incremental advancement over prior Markovian-focused methods.
The paper tackles the challenge of discovering agent-centric state representations in high-dimensional non-Markovian environments, showing that generalized inverse models can be adapted for this task, with empirical analysis revealing that past actions can either enhance success or cause dramatic failure depending on usage.
Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We complement these findings with a thorough empirical study on the agent-centric state discovery abilities of the different alternatives we put forward. Particularly notable is our analysis of past actions, where we show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly.