Uncertainty aware audiovisual activity recognition using deep Bayesian variational inference
This work addresses the need for reliable uncertainty estimation in multimodal deep learning applications, which is incremental by combining deterministic and variational layers for deeper Bayesian networks.
The paper tackles the problem of predictive uncertainty in multimodal audiovisual activity recognition by proposing an uncertainty-aware Bayesian fusion framework, resulting in a 10.2% improvement in precision-recall AUC on a subset of the Moments-in-Time dataset compared to non-Bayesian baselines.
Deep neural networks (DNNs) provide state-of-the-art results for a multitude of applications, but the approaches using DNNs for multimodal audiovisual applications do not consider predictive uncertainty associated with individual modalities. Bayesian deep learning methods provide principled confidence and quantify predictive uncertainty. Our contribution in this work is to propose an uncertainty aware multimodal Bayesian fusion framework for activity recognition. We demonstrate a novel approach that combines deterministic and variational layers to scale Bayesian DNNs to deeper architectures. Our experiments using in- and out-of-distribution samples selected from a subset of Moments-in-Time (MiT) dataset show a more reliable confidence measure as compared to the non-Bayesian baseline and the Monte Carlo dropout (MC dropout) approximate Bayesian inference. We also demonstrate the uncertainty estimates obtained from the proposed framework can identify out-of-distribution data on the UCF101 and MiT datasets. In the multimodal setting, the proposed framework improved precision-recall AUC by 10.2% on the subset of MiT dataset as compared to non-Bayesian baseline.