TTF-VLA: Temporal Token Fusion via Pixel-Attention Integration for Vision-Language-Action Models
This addresses the issue of visual noise and temporal coherence in VLA models for robotic manipulation, offering an incremental enhancement through selective fusion techniques.
The paper tackles the problem of Vision-Language-Action models discarding temporal information in robotic manipulation tasks by proposing Temporal Token Fusion, a training-free method that integrates historical and current visual representations, resulting in average improvements of 4.0 percentage points on LIBERO, 4.8% relative improvement on SimplerEnv, and 8.7% relative improvement on real robot tasks.
Vision-Language-Action (VLA) models process visual inputs independently at each timestep, discarding valuable temporal information inherent in robotic manipulation tasks. This frame-by-frame processing makes models vulnerable to visual noise while ignoring the substantial coherence between consecutive frames in manipulation sequences. We propose Temporal Token Fusion (TTF), a training-free approach that intelligently integrates historical and current visual representations to enhance VLA inference quality. Our method employs dual-dimension detection combining efficient grayscale pixel difference analysis with attention-based semantic relevance assessment, enabling selective temporal token fusion through hard fusion strategies and keyframe anchoring to prevent error accumulation. Comprehensive experiments across LIBERO, SimplerEnv, and real robot tasks demonstrate consistent improvements: 4.0 percentage points average on LIBERO (72.4\% vs 68.4\% baseline), cross-environment validation on SimplerEnv (4.8\% relative improvement), and 8.7\% relative improvement on real robot tasks. Our approach proves model-agnostic, working across OpenVLA and VLA-Cache architectures. Notably, TTF reveals that selective Query matrix reuse in attention mechanisms enhances rather than compromises performance, suggesting promising directions for direct KQV matrix reuse strategies that achieve computational acceleration while improving task success rates.