Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
This work addresses the challenge of maintaining generalization in robot learning by preventing disruption of pretrained features, which is incremental as it builds on existing vision-language models with specific adaptations.
The paper tackles the problem of preserving pretrained representations in vision-language-action models during fine-tuning to enhance generalization in robot manipulation, resulting in improved robustness to visual perturbations, generalization to novel instructions and environments, and overall task success compared to baselines.
Vision-language-action (VLA) models finetuned from vision-language models (VLMs) hold the promise of leveraging rich pretrained representations to build generalist robots across diverse tasks and environments. However, direct fine-tuning on robot data often disrupts these representations and limits generalization. We present a framework that better preserves pretrained features while adapting them for robot manipulation. Our approach introduces three components: (i) a dual-encoder design with one frozen vision encoder to retain pretrained features and another trainable for task adaptation, (ii) a string-based action tokenizer that casts continuous actions into character sequences aligned with the model's pretraining domain, and (iii) a co-training strategy that combines robot demonstrations with vision-language datasets emphasizing spatial reasoning and affordances. Evaluations in simulation and on real robots show that our method improves robustness to visual perturbations, generalization to novel instructions and environments, and overall task success compared to baselines.