TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
This work addresses data-efficient and continual adaptation for robotics and control domains, representing an incremental improvement by applying existing parameter-efficient methods to a new application area.
The paper tackles the challenge of adapting large pretrained models to new control tasks with limited data by introducing TAIL, a framework using parameter-efficient fine-tuning techniques like LoRA, which achieves the best performance with only 1% of trainable parameters compared to full fine-tuning while avoiding catastrophic forgetting.
The full potential of large pretrained models remains largely untapped in control domains like robotics. This is mainly because of the scarcity of data and the computational challenges associated with training or fine-tuning these large models for such applications. Prior work mainly emphasizes either effective pretraining of large models for decision-making or single-task adaptation. But real-world problems will require data-efficient, continual adaptation for new control tasks. Recognizing these constraints, we introduce TAIL (Task-specific Adapters for Imitation Learning), a framework for efficient adaptation to new control tasks. Inspired by recent advancements in parameter-efficient fine-tuning in language domains, we explore efficient fine-tuning techniques -- e.g., Bottleneck Adapters, P-Tuning, and Low-Rank Adaptation (LoRA) -- in TAIL to adapt large pretrained models for new tasks with limited demonstration data. Our extensive experiments in large-scale language-conditioned manipulation tasks comparing prevalent parameter-efficient fine-tuning techniques and adaptation baselines suggest that TAIL with LoRA can achieve the best post-adaptation performance with only 1\% of the trainable parameters of full fine-tuning, while avoiding catastrophic forgetting and preserving adaptation plasticity in continual learning settings.