Simple Emergent Action Representations from Multi-Task Policy Training
This work addresses the problem of inefficient and non-adaptable action planning in reinforcement learning for robotics and AI systems, offering a novel approach that is incremental but promising for improving composability.
The paper tackles the challenge of understanding and utilizing high-dimensional motor actions in deep reinforcement learning by showing that meaningful action representations emerge from multi-task policy training with state and task embeddings, enabling effective intra-action interpolation and inter-action composition with limited additional learning and superior task adaptation in Mujoco locomotion tasks.
The low-level sensory and motor signals in deep reinforcement learning, which exist in high-dimensional spaces such as image observations or motor torques, are inherently challenging to understand or utilize directly for downstream tasks. While sensory representations have been extensively studied, the representations of motor actions are still an area of active exploration. Our work reveals that a space containing meaningful action representations emerges when a multi-task policy network takes as inputs both states and task embeddings. Moderate constraints are added to improve its representation ability. Therefore, interpolated or composed embeddings can function as a high-level interface within this space, providing instructions to the agent for executing meaningful action sequences. Empirical results demonstrate that the proposed action representations are effective for intra-action interpolation and inter-action composition with limited or no additional learning. Furthermore, our approach exhibits superior task adaptation ability compared to strong baselines in Mujoco locomotion tasks. Our work sheds light on the promising direction of learning action representations for efficient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space. Project page: https://sites.google.com/view/emergent-action-representation/