Make a Video Call with LLM: A Measurement Campaign over Five Mainstream Apps
This work addresses the need for performance evaluation in the emerging domain of AI video chat, which is incremental but provides foundational benchmarking for researchers and developers.
The paper tackles the lack of systematic performance characterization for AI video chat systems by proposing a comprehensive benchmark across quality, latency, internal mechanisms, and system overhead, and evaluates five mainstream AI video chatbots to provide a baseline and identify system bottlenecks.
In 2025, Large Language Model (LLM) services have launched a new feature -- AI video chat -- allowing users to interact with AI agents via real-time video communication (RTC), just like chatting with real people. Despite its significance, no systematic study has characterized the performance of existing AI video chat systems. To address this gap, this paper proposes a comprehensive benchmark with carefully designed metrics across four dimensions: quality, latency, internal mechanisms, and system overhead. Using custom testbeds, we further evaluate five mainstream AI video chatbots with this benchmark. This work provides the research community a baseline of real-world performance and identifies unique system bottlenecks. In the meantime, our benchmarking results also open up several research questions for future optimizations of AI video chatbots.