EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning
This addresses the problem of accurate 3D hand reconstruction for applications in egocentric vision, such as hand-object interaction analysis, with incremental improvements in robustness and generalization.
The paper tackles the challenge of robust 3D hand reconstruction in egocentric vision by introducing EgoHandICL, an in-context learning framework that improves semantic alignment, visual consistency, and robustness, showing consistent gains over state-of-the-art methods on datasets like ARCTIC and EgoExo4D.
Robust 3D hand reconstruction in egocentric vision is challenging due to depth ambiguity, self-occlusion, and complex hand-object interactions. Prior methods mitigate these issues by scaling training data or adding auxiliary cues, but they often struggle in unseen contexts. We present EgoHandICL, the first in-context learning (ICL) framework for 3D hand reconstruction that improves semantic alignment, visual consistency, and robustness under challenging egocentric conditions. EgoHandICL introduces complementary exemplar retrieval guided by vision-language models (VLMs), an ICL-tailored tokenizer for multimodal context, and a masked autoencoder (MAE)-based architecture trained with hand-guided geometric and perceptual objectives. Experiments on ARCTIC and EgoExo4D show consistent gains over state-of-the-art methods. We also demonstrate real-world generalization and improve EgoVLM hand-object interaction reasoning by using reconstructed hands as visual prompts. Code and data: https://github.com/Nicous20/EgoHandICL