Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
This addresses the problem of zero-shot generalization for AI agents, offering a novel approach that could enhance adaptability in robotics and gaming, though it appears incremental in its method combination.
The paper tackles the challenge of enabling agents to generalize to unseen environments without additional data collection by learning a neural function that decomposes sparse rewards into contingency-aware observations for per-step rewards, achieving strong performance on tasks like Super Mario Bros and robotic control.
It is a long-standing challenge to enable an intelligent agent to learn in one environment and generalize to an unseen environment without further data collection and finetuning. In this paper, we consider a zero shot generalization problem setup that complies with biological intelligent agents' learning and generalization processes. The agent is first presented with previous experiences in the training environment, along with task description in the form of trajectory-level sparse rewards. Later when it is placed in the new testing environment, it is asked to perform the task without any interaction with the testing environment. We find this setting natural for biological creatures and at the same time, challenging for previous methods. Behavior cloning, state-of-art RL along with other zero-shot learning methods perform poorly on this benchmark. Given a set of experiences in the training environment, our method learns a neural function that decomposes the sparse reward into particular regions in a contingency-aware observation as a per step reward. Based on such decomposed rewards, we further learn a dynamics model and use Model Predictive Control (MPC) to obtain a policy. Since the rewards are decomposed to finer-granularity observations, they are naturally generalizable to new environments that are composed of similar basic elements. We demonstrate our method on a wide range of environments, including a classic video game -- Super Mario Bros, as well as a robotic continuous control task. Please refer to the project page for more visualized results.