Symmetry Detection in Trajectory Data for More Meaningful Reinforcement Learning Representations
This work addresses the challenge of extracting meaningful representations for RL systems, which can improve interpretability and efficiency, though it appears incremental as it builds on existing symmetry detection concepts.
The paper tackles the problem of automatically detecting symmetries in reinforcement learning systems from raw trajectory data, enabling the creation of compressed and semantically meaningful representations. The method successfully identifies symmetries in simulated RL use cases, such as a pusher robot and a UAV flying in wind, without requiring active control.
Knowledge of the symmetries of reinforcement learning (RL) systems can be used to create compressed and semantically meaningful representations of a low-level state space. We present a method of automatically detecting RL symmetries directly from raw trajectory data without requiring active control of the system. Our method generates candidate symmetries and trains a recurrent neural network (RNN) to discriminate between the original trajectories and the transformed trajectories for each candidate symmetry. The RNN discriminator's accuracy for each candidate reveals how symmetric the system is under that transformation. This information can be used to create high-level representations that are invariant to all symmetries on a dataset level and to communicate properties of the RL behavior to users. We show in experiments on two simulated RL use cases (a pusher robot and a UAV flying in wind) that our method can determine the symmetries underlying both the environment physics and the trained RL policy.