Towards Learning Controllable Representations of Physical Systems
This work addresses the challenge of principled representation evaluation for control in reinforcement learning, offering incremental improvements over existing methods.
The paper tackles the problem of evaluating learned representations for control in dynamical systems by proposing two metrics—temporal smoothness and high mutual information between true state and representation—to predict reinforcement learning performance, demonstrating this on a simulated peg-in-hole task with autoencoder variants.
Learned representations of dynamical systems reduce dimensionality, potentially supporting downstream reinforcement learning (RL). However, no established methods predict a representation's suitability for control and evaluation is largely done via downstream RL performance, slowing representation design. Towards a principled evaluation of representations for control, we consider the relationship between the true state and the corresponding representations, proposing that ideally each representation corresponds to a unique true state. This motivates two metrics: temporal smoothness and high mutual information between true state/representation. These metrics are related to established representation objectives, and studied on Lagrangian systems where true state, information requirements, and statistical properties of the state can be formalized for a broad class of systems. These metrics are shown to predict reinforcement learning performance in a simulated peg-in-hole task when comparing variants of autoencoder-based representations.