A Probabilistic Jump-Diffusion Framework for Open-World Egocentric Activity Recognition
It addresses the challenge of recognizing activities in unconstrained, real-world egocentric video data, which is crucial for applications like assistive technologies and robotics, though it appears incremental by building on existing methods with structured priors and VLMs.
The paper tackles open-world egocentric activity recognition by introducing ProbRes, a probabilistic jump-diffusion framework that balances exploration and exploitation to infer unseen activities, achieving state-of-the-art performance on benchmark datasets like GTEA Gaze and EPIC-Kitchens.
Open-world egocentric activity recognition poses a fundamental challenge due to its unconstrained nature, requiring models to infer unseen activities from an expansive, partially observed search space. We introduce ProbRes, a Probabilistic Residual search framework based on jump-diffusion that efficiently navigates this space by balancing prior-guided exploration with likelihood-driven exploitation. Our approach integrates structured commonsense priors to construct a semantically coherent search space, adaptively refines predictions using Vision-Language Models (VLMs) and employs a stochastic search mechanism to locate high-likelihood activity labels while minimizing exhaustive enumeration efficiently. We systematically evaluate ProbRes across multiple openness levels (L0--L3), demonstrating its adaptability to increasing search space complexity. In addition to achieving state-of-the-art performance on benchmark datasets (GTEA Gaze, GTEA Gaze+, EPIC-Kitchens, and Charades-Ego), we establish a clear taxonomy for open-world recognition, delineating the challenges and methodological advancements necessary for egocentric activity understanding.