Learning telic-controllable state representations
This work addresses a foundational issue in reinforcement learning for agents, focusing on how goals shape state representations, but it appears incremental as it builds on existing concepts of controllability and representation learning.
The paper tackles the problem of learning state representations in reinforcement learning by coupling descriptive and prescriptive aspects through telic states, introducing a framework for telic-controllability to balance granularity and policy complexity, and demonstrating it with a simulated navigation task.
Computational models of purposeful behavior comprise both descriptive and prescriptive aspects, used respectively to ascertain and evaluate situations in the world. In reinforcement learning, prescriptive reward functions are assumed to depend on predefined and fixed descriptive state representations. Alternatively, these two aspects may emerge interdependently: goals can shape the acquired state representations and vice versa. Here, we present a computational framework for state representation learning in bounded agents, where descriptive and prescriptive aspects are coupled through the notion of goal-directed, or telic, states. We introduce the concept of telic-controllability to characterize the tradeoff between the granularity of a telic state representation and the policy complexity required to reach all telic states. We propose an algorithm for learning telic-controllable state representations, illustrating it using a simulated navigation task. Our framework highlights the role of deliberate ignorance -- knowing what to ignore -- for learning state representations that balance goal flexibility and cognitive complexity.