Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos
This addresses the need for detailed mistake analysis in egocentric video understanding, offering incremental improvements by extending existing datasets and methods.
The paper tackles the problem of fine-grained understanding of human mistakes in egocentric videos by introducing the Mistake Attribution (MATT) task, which attributes mistakes to instruction text or video, determining what, when, and where mistakes occur, and results in datasets up to 100 times larger than prior ones and a model that outperforms various baselines.
We introduce Mistake Attribution (MATT), a task for fine-grained understanding of human mistakes in egocentric video. Unlike prior mistake understanding work, which lacks fine-grained output, MATT concretely attributes mistakes to the input instruction text or the attempt video. MATT determines what part of the instruction is violated (semantic role), when the deviation becomes irreversible (the Point-of-No-Return, PNR), and where the mistake appears in the PNR frame. We develop MisEngine, a data engine that automatically constructs attribution-rich mistake samples from existing datasets and inherits their annotations. Applied to large egocentric corpora, MisEngine yields EPIC-KITCHENS-M and Ego4D-M, two datasets that are up to two orders of magnitude larger than prior mistake datasets. We then present MisFormer, a unified attention-based model for mistake attribution across semantic (what), temporal (when), and spatial (where) dimensions, trained using MisEngine supervision. Experiments on our new datasets and prior benchmarks show that MisFormer outperforms strong video-language, temporal localization, hand-object interaction, and mistake-detection baselines.