Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation
This addresses the problem of generating accessible audio descriptions for movies and TV series, offering a training-free solution that integrates expert knowledge.
The paper tackles automatic Audio Description generation for edited videos by proposing a two-stage framework that uses shots as fundamental units and incorporates film grammar devices, achieving state-of-the-art performance among training-free approaches and surpassing fine-tuned methods on several benchmarks.
Our objective is the automatic generation of Audio Descriptions (ADs) for edited video material, such as movies and TV series. To achieve this, we propose a two-stage framework that leverages "shots" as the fundamental units of video understanding. This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures, to guide AD generation. Our method is compatible with both open-source and proprietary Visual-Language Models (VLMs), integrating expert knowledge from add-on modules without requiring additional training of the VLMs. We achieve state-of-the-art performance among all prior training-free approaches and even surpass fine-tuned methods on several benchmarks. To evaluate the quality of predicted ADs, we introduce a new evaluation measure -- an action score -- specifically targeted to assessing this important aspect of AD. Additionally, we propose a novel evaluation protocol that treats automatic frameworks as AD generation assistants and asks them to generate multiple candidate ADs for selection.