Extend the FFmpeg Framework to Analyze Media Content
This work addresses the need for enhanced media analysis capabilities in multimedia applications using FFmpeg, but it is incremental as it builds on existing frameworks and tools.
The paper tackled the problem of integrating AI-based video analytics into the FFmpeg framework by developing new plugins, resulting in thread-optimized performance and acceleration across platforms like CPU and GPU, with the OpenVINO backend being pushed into the FFmpeg mainstream repository.
This paper introduces a new set of video analytics plugins developed for the FFmpeg framework. Multimedia applications that increasingly utilize the FFmpeg media features for its comprehensive media encoding, decoding, muxing, and demuxing capabilities can now additionally analyze the video content based on AI models. The plugins are thread optimized for best performance overcoming certain FFmpeg threading limitations. The plugins utilize the Intel OpenVINO Toolkit inference engine as the backend. The analytics workloads are accelerated on different platforms such as CPU, GPU, FPGA or specialized analytics accelerators. With our reference implementation, the feature of OpenVINO as inference backend has been pushed into FFmpeg mainstream repository. We plan to submit more patches later.