From Parameters to Behavior: Unsupervised Compression of the Policy Space
This addresses the problem of sample inefficiency in DRL, particularly in multi-task settings, by reducing redundancy in policy optimization, though it is an incremental improvement over existing methods.
The paper tackles the sample inefficiency of Deep Reinforcement Learning by compressing the high-dimensional policy parameter space into a low-dimensional latent space using an unsupervised generative model, achieving compression up to five orders of magnitude while retaining expressivity and enabling task-specific adaptation.
Despite its recent successes, Deep Reinforcement Learning (DRL) is notoriously sample-inefficient. We argue that this inefficiency stems from the standard practice of optimizing policies directly in the high-dimensional and highly redundant parameter space $Θ$. This challenge is greatly compounded in multi-task settings. In this work, we develop a novel, unsupervised approach that compresses the policy parameter space $Θ$ into a low-dimensional latent space $\mathcal{Z}$. We train a generative model $g:\mathcal{Z}\toΘ$ by optimizing a behavioral reconstruction loss, which ensures that the latent space is organized by functional similarity rather than proximity in parameterization. We conjecture that the inherent dimensionality of this manifold is a function of the environment's complexity, rather than the size of the policy network. We validate our approach in continuous control domains, showing that the parameterization of standard policy networks can be compressed up to five orders of magnitude while retaining most of its expressivity. As a byproduct, we show that the learned manifold enables task-specific adaptation via Policy Gradient operating in the latent space $\mathcal{Z}$.