Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition
For practitioners of multi-modal action recognition, this work provides a simple yet effective post-hoc OOD detection method that can be combined with existing training-time approaches.
The paper tackles multi-modal out-of-distribution (OOD) detection for action recognition, proposing a post-hoc detector that leverages the relationship between multi-modal and uni-modal predictions combined with feature-space scores. The method consistently improves performance over state-of-the-art on the MultiOOD benchmark.
The incorporation of additional modalities into action recognition models increases their performance across a wide range of settings. However, how this additional information can contribute to making the models more robust remains underexplored, particularly for the case of multi-modal out-of-distribution (OOD) detection. While methods exist that regularize the multi-modal training process with OOD detection in mind, they still apply off-the-shelf OOD detectors designed for the uni-modal case during inference, discarding important information. Based on an interesting relationship we find between the multi-modal and uni-modal predictions, we propose to use this signal to build a post-hoc detector explicitly designed for the multi-modal scenario. We combine this new source of information with a feature-space score, which detects off-manifold samples in the multi-modal space, and normalize them by the multi-modal logits. In doing so, the proposed hybrid detector is compatible with existing training-time approaches and consistently improves performance. Experiments on a wide range of established datasets from the MultiOOD benchmark show that, on average, our approach outperforms the state of the art. Our results show the importance of explicitly considering the different modalities at inference time for multi-modal OOD detection.