Learning to Perceive the World Through Control: Empowerment-Based Representation Learning
This work addresses the problem of learning efficient, control-relevant representations for reinforcement learning agents in high-dimensional environments, which is crucial for improving sample efficiency and generalization.
This paper investigates whether reinforcement learning agents can learn representations that focus solely on control-relevant features in high-dimensional environments. It demonstrates that maximizing empowerment, an objective for unsupervised skill learning, leads agents to learn distinct forward and backward representations that are both invariant to control-irrelevant features, effectively creating an implicit, control-centric world model.
In many practical reinforcement learning environments, observations are far higher-dimensional than the variables that matter for control. In this work, we ask: can we learn representations that capture only control-relevant features of the environment? We study this question through the empowerment objective, which maximizes an agent's influence over the environment and is widely used for unsupervised skill learning. We show that empowerment agents induce two distinct representations -- forward and backward -- that capture complementary aspects of the state, and both of which are invariant to control-irrelevant features. Thus, empowerment maximization leads agents to learn an implicit, control-centric model of the world. Our analysis highlights the importance of learning representations through interaction rather than from passive datasets: interaction aimed at maximizing control is essential for learning useful invariance properties, a perspective that aligns closely with the causal learning literature.