MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipation
This addresses the problem of reliable long-term dense anticipation of human activities in real-world scenarios, representing an incremental improvement over existing SSM approaches.
The paper tackles the limitation of static forget-gates in State Space Models for human activity anticipation by introducing MixANT, which dynamically selects contextually relevant forget-gate matrices using a mixture of experts approach. The method outperforms state-of-the-art methods on three datasets (50Salads, Breakfast, and Assembly101) across all evaluation settings.
We present MixANT, a novel architecture for stochastic long-term dense anticipation of human activities. While recent State Space Models (SSMs) like Mamba have shown promise through input-dependent selectivity on three key parameters, the critical forget-gate ($\textbf{A}$ matrix) controlling temporal memory remains static. We address this limitation by introducing a mixture of experts approach that dynamically selects contextually relevant $\textbf{A}$ matrices based on input features, enhancing representational capacity without sacrificing computational efficiency. Extensive experiments on the 50Salads, Breakfast, and Assembly101 datasets demonstrate that MixANT consistently outperforms state-of-the-art methods across all evaluation settings. Our results highlight the importance of input-dependent forget-gate mechanisms for reliable prediction of human behavior in diverse real-world scenarios.