Learning and Understanding a Disentangled Feature Representation for Hidden Parameters in Reinforcement Learning
This work addresses the challenge of analyzing latent variables in RL systems, which is incremental as it builds on existing model-based RL techniques.
The paper tackles the problem of identifying and understanding hidden parameters in reinforcement learning environments by developing an unsupervised method that maps trajectories into a disentangled feature space, demonstrating it on four hidden parameters across three RL environments.
Hidden parameters are latent variables in reinforcement learning (RL) environments that are constant over the course of a trajectory. Understanding what, if any, hidden parameters affect a particular environment can aid both the development and appropriate usage of RL systems. We present an unsupervised method to map RL trajectories into a feature space where distance represents the relative difference in system behavior due to hidden parameters. Our approach disentangles the effects of hidden parameters by leveraging a recurrent neural network (RNN) world model as used in model-based RL. First, we alter the standard world model training algorithm to isolate the hidden parameter information in the world model memory. Then, we use a metric learning approach to map the RNN memory into a space with a distance metric approximating a bisimulation metric with respect to the hidden parameters. The resulting disentangled feature space can be used to meaningfully relate trajectories to each other and analyze the hidden parameter. We demonstrate our approach on four hidden parameters across three RL environments. Finally we present two methods to help identify and understand the effects of hidden parameters on systems.