Efficient Embedding of Semantic Similarity in Control Policies via Entangled Bisimulation
This addresses the problem of visual distractions in reinforcement learning for robotics or simulation tasks, offering an incremental improvement over existing bisimulation methods.
The paper tackles the challenge of learning generalizable policies from visual input with distractions in reinforcement learning by introducing entangled bisimulation, a metric that can be estimated without bias in continuous spaces. It shows meaningful improvements over previous methods on the Distracting Control Suite, even when combined with data augmentation.
Learning generalizeable policies from visual input in the presence of visual distractions is a challenging problem in reinforcement learning. Recently, there has been renewed interest in bisimulation metrics as a tool to address this issue; these metrics can be used to learn representations that are, in principle, invariant to irrelevant distractions by measuring behavioural similarity between states. An accurate, unbiased, and scalable estimation of these metrics has proved elusive in continuous state and action scenarios. We propose entangled bisimulation, a bisimulation metric that allows the specification of the distance function between states, and can be estimated without bias in continuous state and action spaces. We show how entangled bisimulation can meaningfully improve over previous methods on the Distracting Control Suite (DCS), even when added on top of data augmentation techniques.