Low Bandwidth Video-Chat Compression using Deep Generative Models
This work aims to make video chat accessible to hundreds of millions of people facing poor connectivity or high data costs, representing a strong specific gain in a domain-specific application.
This paper addresses the challenge of video chat for users with poor connectivity by reconstructing faces on the receiver's device using facial landmarks transmitted from the sender. The authors developed a mobile-compatible architecture that achieves video calling at a few kbits per second, which is an order of magnitude lower bandwidth than current alternatives.
To unlock video chat for hundreds of millions of people hindered by poor connectivity or unaffordable data costs, we propose to authentically reconstruct faces on the receiver's device using facial landmarks extracted at the sender's side and transmitted over the network. In this context, we discuss and evaluate the benefits and disadvantages of several deep adversarial approaches. In particular, we explore quality and bandwidth trade-offs for approaches based on static landmarks, dynamic landmarks or segmentation maps. We design a mobile-compatible architecture based on the first order animation model of Siarohin et al. In addition, we leverage SPADE blocks to refine results in important areas such as the eyes and lips. We compress the networks down to about 3MB, allowing models to run in real time on iPhone 8 (CPU). This approach enables video calling at a few kbits per second, an order of magnitude lower than currently available alternatives.