Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder
This work addresses the time-consuming and expert-dependent manual coding of clinical videos for autism research, offering a computational tool to augment diagnostic support.
The paper tackled the problem of analyzing long-form clinical videos of children with Autism Spectrum Disorder by proposing a multimodal pipeline using foundation models across speech, video, and text, which improved performance on tasks like activity recognition and abnormal behavior detection compared to unimodal approaches.
Clinical videos in the context of Autism Spectrum Disorder are often long-form interactions between children and caregivers/clinical professionals, encompassing complex verbal and non-verbal behaviors. Objective analyses of these videos could provide clinicians and researchers with nuanced insights into the behavior of children with Autism Spectrum Disorder. Manually coding these videos is a time-consuming task and requires a high level of domain expertise. Hence, the ability to capture these interactions computationally can augment the manual effort and enable supporting the diagnostic procedure. In this work, we investigate the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions. We propose a unified methodology to combine multiple modalities by using large language models as reasoning agents. We evaluate their performance on two tasks with different information granularity: activity recognition and abnormal behavior detection. We find that the proposed multimodal pipeline provides robustness to modality-specific limitations and improves performance on the clinical video analysis compared to unimodal settings.