EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies
For researchers developing generalist mobile manipulation policies, EBench provides a diagnostic tool to identify specific strengths and weaknesses beyond aggregate success rates, enabling targeted model improvements.
EBench is a simulation benchmark with 26 tasks that diagnoses generalist mobile manipulation policies across 5 capability and 4 generalization dimensions. Evaluation of models like π0, π0.5, XVLA, and InternVLA-A1 reveals that models with similar overall success rates have distinct capability profiles, e.g., π0.5 has the highest test success rate and train-test retention, while InternVLA-A1 excels in mobile manipulation but fails on dexterous tasks.
We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including $π_0$, $π_{0.5}$, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: $π_{0.5}$ achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.