SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation
For robot learning researchers, this provides a method to curate human video data for dexterous manipulation, showing that selective data mining can significantly outperform random data mixing.
SiMDex is a similarity-based data mining framework that selects task-relevant egocentric human video samples for VLA post-training in dexterous manipulation. Using only ~1.49M mined samples (<5% of a ~32M pool), it improves success rate from 47.7% to 61.1% over a baseline with random human data.
Recent years have witnessed an explosive trend of scaling ego-centric human videos for robot manipulation, yet it remains unclear which data actually benefits dexterous manipulation. We present SiMDex, a similarity-based data mining framework that casts human data selection for VLA post-training in dexterous manipulation as a recommendation problem. For each robot demonstration, SiMDex employs a three-layer recall-ranking-re-ranking pipeline to extract task-relevant subsets from a pool of ~32M egocentric human samples, operating in a morphology-agnostic action space that requires no changes to VLA architecture or training. Against a strong baseline trained with an equal amount of randomly sampled human data, SiMDex uses only ~1.49M mined samples (<5% of the pool) yet improves the overall success rate from 47.7% to 61.1%, showing that selective curation outperforms indiscriminate data mixing.