Predicting Mood Disorder Symptoms with Remotely Collected Videos Using an Interpretable Multimodal Dynamic Attention Fusion Network
This work addresses mood disorder diagnosis for patients and clinicians by providing a novel, interpretable approach, though it is incremental in improving upon existing multimodal methods.
The researchers tackled the problem of identifying mood disorder symptoms by developing an interpretable multimodal classification method using audio, video, and text from a smartphone app, achieving better performance than existing methods with static embeddings on a dataset of 3002 participants.
We developed a novel, interpretable multimodal classification method to identify symptoms of mood disorders viz. depression, anxiety and anhedonia using audio, video and text collected from a smartphone application. We used CNN-based unimodal encoders to learn dynamic embeddings for each modality and then combined these through a transformer encoder. We applied these methods to a novel dataset - collected by a smartphone application - on 3002 participants across up to three recording sessions. Our method demonstrated better multimodal classification performance compared to existing methods that employed static embeddings. Lastly, we used SHapley Additive exPlanations (SHAP) to prioritize important features in our model that could serve as potential digital markers.