From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
This addresses the representation problem for reinforcement learning agents that need to exploit factored structure from pixel inputs, though it is incremental as it builds on existing contrastive learning and factored MDP concepts.
The paper tackled the problem of learning independently controllable state variables from high-dimensional observations in reinforcement learning, and introduced Action-Controllable Factorization (ACF), which outperformed baseline disentanglement algorithms on benchmarks like Taxi, FourRooms, and MiniGrid-DoorKey.
Algorithms that exploit factored Markov decision processes are far more sample-efficient than factor-agnostic methods, yet they assume a factored representation is known a priori -- a requirement that breaks down when the agent sees only high-dimensional observations. Conversely, deep reinforcement learning handles such inputs but cannot benefit from factored structure. We address this representation problem with Action-Controllable Factorization (ACF), a contrastive learning approach that uncovers independently controllable latent variables -- state components each action can influence separately. ACF leverages sparsity: actions typically affect only a subset of variables, while the rest evolve under the environment's dynamics, yielding informative data for contrastive training. ACF recovers the ground truth controllable factors directly from pixel observations on three benchmarks with known factored structure -- Taxi, FourRooms, and MiniGrid-DoorKey -- consistently outperforming baseline disentanglement algorithms.