SurgMAE: Masked Autoencoders for Long Surgical Video Analysis
This work addresses the need for scalable and efficient tools in surgical workflow analysis by reducing reliance on labeled data, though it is incremental as it adapts existing MAE methods to a new domain.
The paper tackles the problem of reducing annotation costs for deep learning models in long surgical video analysis by proposing SurgMAE, a masked autoencoder with a novel spatio-temporal masking strategy, which outperforms baselines in low-data regimes and shows generalizability on non-surgical datasets.
There has been a growing interest in using deep learning models for processing long surgical videos, in order to automatically detect clinical/operational activities and extract metrics that can enable workflow efficiency tools and applications. However, training such models require vast amounts of labeled data which is costly and not scalable. Recently, self-supervised learning has been explored in computer vision community to reduce the burden of the annotation cost. Masked autoencoders (MAE) got the attention in self-supervised paradigm for Vision Transformers (ViTs) by predicting the randomly masked regions given the visible patches of an image or a video clip, and have shown superior performance on benchmark datasets. However, the application of MAE in surgical data remains unexplored. In this paper, we first investigate whether MAE can learn transferrable representations in surgical video domain. We propose SurgMAE, which is a novel architecture with a masking strategy based on sampling high spatio-temporal tokens for MAE. We provide an empirical study of SurgMAE on two large scale long surgical video datasets, and find that our method outperforms several baselines in low data regime. We conduct extensive ablation studies to show the efficacy of our approach and also demonstrate it's superior performance on UCF-101 to prove it's generalizability in non-surgical datasets as well.