Mind the Gap: Disentangling Performance Bottlenecks in Video Instance Segmentation
This work provides a systematic diagnostic tool for researchers to identify and target the core challenge of robust long-term temporal association in VIS.
The paper introduces a diagnostic framework to isolate performance bottlenecks in Video Instance Segmentation (VIS), revealing that tracking instability is the primary bottleneck for online methods, with gaps exceeding 20 AP under heavy occlusion, while semantic classification impact diminishes where tracking fails. The analysis shows that temporal fragility is algorithmic rather than representational.
In Video Instance Segmentation (VIS), classification, segmentation, and tracking objectives are jointly evaluated, but their individual contributions to performance loss remain opaque. We introduce a diagnostic framework that formulates identity and class assignment as an Integer Linear Program (ILP), yielding a model-agnostic oracle that hierarchically isolates each error source. Applied to seven VIS methods spanning online and offline paradigms across YouTube-VIS 2019/2021 and a diagnostic subset of OVIS, our analysis reveals a consistent picture. Tracking instability is a critical bottleneck for online methods, with gaps exceeding 20 AP under heavy occlusion, and grows sharply with video length and instance density. While semantic classification contributes meaningfully on standard benchmarks, its impact becomes negligible where tracking fails most. Although stronger backbones substantially lift default scores, they leave AP tracking gaps largely intact, confirming that temporal fragility is algorithmic rather than purely representational. To complement the oracle, we introduce TrackLens, a visual tool that translates gap magnitude into observable, query-level failure modes. Together, these tools provide a systematic foundation for targeting VIS's core challenge: robust long-term temporal association.