Hyper-Decision Transformer for Efficient Online Policy Adaptation
This work addresses efficient online policy adaptation for reinforcement learning, particularly in object manipulation tasks, representing an incremental improvement over existing methods.
The paper tackles the challenge of adapting Decision Transformers to unseen tasks efficiently, proposing Hyper-Decision Transformer (HDT) that uses a hyper-network to initialize an adaptation module from few demonstrations, resulting in faster adaptation with only 0.5% parameter fine-tuning and outperforming state-of-the-art baselines in task success rates.
Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To address this challenge, we propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in a data- and parameter-efficient manner. To achieve such a goal, we propose to augment the base DT with an adaptation module, whose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and initializes the adaptation module accordingly. This initialization enables HDT to efficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT's generalization capability on object manipulation tasks. We find that with a single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally, we explore a more challenging setting where expert actions are not available, and we show that HDT outperforms state-of-the-art baselines in terms of task success rates by a large margin.