REVEX: A Unified Framework for Removal-Based Explainable Artificial Intelligence in Video
This work addresses the need for explainable AI in video analysis, but it is incremental as it adapts existing methods rather than introducing a fundamentally new approach.
The authors tackled the problem of explainable AI for video by developing REVEX, a unified framework that adapts six existing methods to video, revealing that Video LIME achieved up to 31% lower deletion and 30% higher insertion values, while Video RISE had 10% lower average drop, but localization metrics like pointing game accuracy reached only 53% and IoU-based metrics stayed below 20%.
We developed REVEX, a removal-based video explanations framework. This work extends fine-grained explanation frameworks for computer vision data and adapts six existing techniques to video by adding temporal information and local explanations. The adapted methods were evaluated across networks, datasets, image classes, and evaluation metrics. By decomposing explanation into steps, strengths and weaknesses were revealed in the studied methods, for example, on pixel clustering and perturbations in the input. Video LIME outperformed other methods with deletion values up to 31\% lower and insertion up to 30\% higher, depending on method and network. Video RISE achieved superior performance in the average drop metric, with values 10\% lower. In contrast, localization-based metrics revealed low performance across all methods, with significant variation depending on network. Pointing game accuracy reached 53\%, and IoU-based metrics remained below 20\%. Drawing on the findings across XAI methods, we further examine the limitations of the employed XAI evaluation metrics and highlight their suitability in different applications.