End-to-End Learning-based Video Streaming Enhancement Pipeline: A Generative AI Approach
This addresses the problem of bandwidth efficiency in video streaming for users, though it is incremental as it builds on existing codecs and generative AI techniques.
The paper tackles the challenge of balancing video quality and smooth playback in streaming by introducing ELVIS, an end-to-end architecture that uses server-side encoding and client-side generative in-painting to remove and reconstruct redundant data, achieving up to 11 VMAF points improvement over baselines.
The primary challenge of video streaming is to balance high video quality with smooth playback. Traditional codecs are well tuned for this trade-off, yet their inability to use context means they must encode the entire video data and transmit it to the client. This paper introduces ELVIS (End-to-end Learning-based VIdeo Streaming Enhancement Pipeline), an end-to-end architecture that combines server-side encoding optimizations with client-side generative in-painting to remove and reconstruct redundant video data. Its modular design allows ELVIS to integrate different codecs, inpainting models, and quality metrics, making it adaptable to future innovations. Our results show that current technologies achieve improvements of up to 11 VMAF points over baseline benchmarks, though challenges remain for real-time applications due to computational demands. ELVIS represents a foundational step toward incorporating generative AI into video streaming pipelines, enabling higher quality experiences without increased bandwidth requirements.