OR-Action: Multi-Role Video Understanding with Fine-Grained Actions
This work provides a new benchmark and method for fine-grained action recognition in operating rooms, addressing the need for temporal modeling in cluttered, multi-role environments.
The authors introduce the first action-centric benchmark for operating room video understanding, defining a fine-grained, multi-role action taxonomy from scene graph state changes. They show that current scene graph methods struggle with temporal modeling and propose a vision-only temporal model that significantly outperforms graph-based methods, along with a multi- to single-view feature alignment strategy that improves single-view multi-role action recognition.
Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance, yet remains difficult due to clutter, occlusions, and limited sensing. The prevailing approach to model this environment is scene graphs as an interpretable representation of OR interactions. Converting their frame-wise relational predictions into temporally extended, fine-grained actions however, is challenging without explicit temporal modeling. To enable a principled temporal evaluation of current OR understanding methods, we introduce the first action-centric benchmark built on a publicly available ego-exocentric OR dataset by defining a fine-grained, multi-role action taxonomy and generating dense action segments via distillation from ground-truth scene graph state changes. Experiments on this benchmark show that current scene graph prediction methods struggle to model temporal structure, even when adding explicit modeling through Graph Neural Networks. We therefore introduce a vision-only temporal model that outperforms graph-based methods significantly when using all available egocentric video as input. Building on this model we also introduce a novel multi- to single-view feature alignment strategy that improves single-view performance on multi-role action recognition, mitigating the need for extensive egocentric video capture. Benchmark and code will be released upon acceptance.