SPOC: Spatially-Progressing Object State Change Segmentation in Video
This addresses the limitation of existing temporal localization methods for detailed action progress and spatial localization, with potential applications in robotics, though it is incremental in advancing video understanding.
The paper tackles the problem of object state change segmentation in video by introducing a spatially-progressing task to segment actionable and transformed regions at the pixel-level, achieving validation on two datasets with promising results for localizing where and how fast objects change.
Object state changes in video reveal critical information about human and agent activity. However, existing methods are limited to temporal localization of when the object is in its initial state (e.g., the unchopped avocado) versus when it has completed a state change (e.g., the chopped avocado), which limits applicability for any task requiring detailed information about the progress of the actions and its spatial localization. We propose to deepen the problem by introducing the spatially-progressing object state change segmentation task. The goal is to segment at the pixel-level those regions of an object that are actionable and those that are transformed. We introduce the first model to address this task, designing a VLM-based pseudo-labeling approach, state-change dynamics constraints, and a novel WhereToChange benchmark built on in-the-wild Internet videos. Experiments on two datasets validate both the challenge of the new task as well as the promise of our model for localizing exactly where and how fast objects are changing in video. We further demonstrate useful implications for tracking activity progress to benefit robotic agents. Project page: https://vision.cs.utexas.edu/projects/spoc-spatially-progressing-osc