Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
This work addresses the challenge of designing practical asymmetric reinforcement learning methods for partially observable environments, offering a more flexible approach that is incremental over existing methods.
The paper tackles the problem of reinforcement learning in partially observable environments by proposing an informed asymmetric actor-critic framework that allows conditioning the critic on arbitrary privileged signals without full-state access, showing improved learning efficiency and value estimation in benchmark tasks.
Reinforcement learning in partially observable environments requires agents to act under uncertainty from noisy, incomplete observations. Asymmetric actor-critic methods leverage privileged information during training to improve learning under these conditions. However, existing approaches typically assume full-state access during training. In this work, we challenge this assumption by proposing a novel actor-critic framework, called informed asymmetric actor-critic, that enables conditioning the critic on arbitrary privileged signals without requiring access to the full state. We show that policy gradients remain unbiased under this formulation, extending the theoretical foundation of asymmetric methods to the more general case of privileged partial information. To quantify the impact of such signals, we propose informativeness measures based on kernel methods and return prediction error, providing practical tools for evaluating training-time signals. We validate our approach empirically on benchmark navigation tasks and synthetic partially observable environments, showing that our informed asymmetric method improves learning efficiency and value estimation when informative privileged inputs are available. Our findings challenge the necessity of full-state access and open new directions for designing asymmetric reinforcement learning methods that are both practical and theoretically sound.