Learning Policy Representations for Steerable Behavior Synthesis
This work addresses the challenge of flexible policy control in reinforcement learning, offering a novel approach for behavior synthesis, though it is incremental in its method development.
The paper tackles the problem of learning representations for policies in Markov decision processes to enable behavior steering at test time, resulting in a model that can steer policies to satisfy unseen value function constraints without additional training.
Given a Markov decision process (MDP), we seek to learn representations for a range of policies to facilitate behavior steering at test time. As policies of an MDP are uniquely determined by their occupancy measures, we propose modeling policy representations as expectations of state-action feature maps with respect to occupancy measures. We show that these representations can be approximated uniformly for a range of policies using a set-based architecture. Our model encodes a set of state-action samples into a latent embedding, from which we decode both the policy and its value functions corresponding to multiple rewards. We use variational generative approach to induce a smooth latent space, and further shape it with contrastive learning so that latent distances align with differences in value functions. This geometry permits gradient-based optimization directly in the latent space. Leveraging this capability, we solve a novel behavior synthesis task, where policies are steered to satisfy previously unseen value function constraints without additional training.