CASR: Refining Action Segmentation via Marginalizing Frame-levle Causal Relationships
This work addresses the challenge of improving action segmentation accuracy and interpretability for video analysis applications, representing an incremental advancement by refining existing models through causal abstraction.
The paper tackles the problem of refining Temporal Action Segmentation (TAS) by addressing noisy frame-level causal relationships that hinder macro action semantics, proposing CASR to marginalize these relationships and improve segmentation results, with experiments showing significant performance gains and enhanced causal interpretability on mainstream datasets.
Integrating deep learning and causal discovery has increased the interpretability of Temporal Action Segmentation (TAS) tasks. However, frame-level causal relationships exist many complicated noises outside the segment-level, making it infeasible to directly express macro action semantics. Thus, we propose Causal Abstraction Segmentation Refiner (CASR), which can refine TAS results from various models by enhancing video causality in marginalizing frame-level casual relationships. Specifically, we define the equivalent frame-level casual model and segment-level causal model, so that the causal adjacency matrix constructed from marginalized frame-level causal relationships has the ability to represent the segmnet-level causal relationships. CASR works out by reducing the difference in the causal adjacency matrix between we constructed and pre-segmentation results of backbone models. In addition, we propose a novel evaluation metric Causal Edit Distance (CED) to evaluate the causal interpretability. Extensive experimental results on mainstream datasets indicate that CASR significantly surpasses existing various methods in action segmentation performance, as well as in causal explainability and generalization.