Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation
This work addresses the lack of large annotated datasets for video semantic segmentation in specialized fields like medicine, offering a semi-supervised approach to leverage temporal clues and anatomical constraints, though it is incremental as it builds on existing autoencoder methods.
The authors tackled the problem of video semantic segmentation in surgical videos by introducing the Temporally Constrained Neural Networks (TCNN) framework, which uses autoencoder networks to provide spatial and temporal supervisory signals, resulting in consistent improvements on sparsely labeled datasets like Endoscapes and CaDIS with no extra computational cost at inference.
A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised framework used for video semantic segmentation of surgical videos. In this work, we show that autoencoder networks can be used to efficiently provide both spatial and temporal supervisory signals to train deep learning models. We test our method on a newly introduced video dataset of laparoscopic cholecystectomy procedures, Endoscapes, and an adaptation of a public dataset of cataract surgeries, CaDIS. We demonstrate that lower-dimensional representations of predicted masks can be leveraged to provide a consistent improvement on both sparsely labeled datasets with no additional computational cost at inference time. Further, the TCNN framework is model-agnostic and can be used in conjunction with other model design choices with minimal additional complexity.