Show from Tell: Audio-Visual Modelling in Clinical Settings
This work addresses the problem of learning anatomical representations from noisy clinical audio-visual data for healthcare applications, representing an incremental advance in self-supervised learning for medical imaging.
The paper tackled the challenge of audio-visual modeling in clinical settings by proposing a multi-modal self-supervised learning framework that learns medical representations without human annotation, resulting in improved performance for automated downstream clinical tasks, even outperforming fully-supervised solutions.
Auditory and visual signals usually present together and correlate with each other, not only in natural environments but also in clinical settings. However, the audio-visual modelling in the latter case can be more challenging, due to the different sources of audio/video signals and the noise (both signal-level and semantic-level) in auditory signals -- usually speech. In this paper, we consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations that benefit various clinical tasks, without human expert annotation. A simple yet effective multi-modal self-supervised learning framework is proposed for this purpose. The proposed approach is able to localise anatomical regions of interest during ultrasound imaging, with only speech audio as a reference. Experimental evaluations on a large-scale clinical multi-modal ultrasound video dataset show that the proposed self-supervised method learns good transferable anatomical representations that boost the performance of automated downstream clinical tasks, even outperforming fully-supervised solutions.