LLM-AD: Large Language Model based Audio Description System
This work addresses the challenge of making video content more accessible for visually impaired individuals by reducing the labor-intensive nature of traditional AD production, though it is incremental as it builds on existing multimodal models.
The paper tackles the problem of automating Audio Description (AD) generation for video accessibility by introducing a pipeline using GPT-4V, which eliminates the need for training and achieves performance comparable to learning-based methods with a CIDEr score of 20.5 on the MAD dataset.
The development of Audio Description (AD) has been a pivotal step forward in making video content more accessible and inclusive. Traditionally, AD production has demanded a considerable amount of skilled labor, while existing automated approaches still necessitate extensive training to integrate multimodal inputs and tailor the output from a captioning style to an AD style. In this paper, we introduce an automated AD generation pipeline that harnesses the potent multimodal and instruction-following capacities of GPT-4V(ision). Notably, our methodology employs readily available components, eliminating the need for additional training. It produces ADs that not only comply with established natural language AD production standards but also maintain contextually consistent character information across frames, courtesy of a tracking-based character recognition module. A thorough analysis on the MAD dataset reveals that our approach achieves a performance on par with learning-based methods in automated AD production, as substantiated by a CIDEr score of 20.5.