Revising Second Order Terms in Deep Animation Video Coding
This work addresses a specific bottleneck in deep animation video coding for low-bitrate video communication, offering incremental improvements over existing methods.
The paper tackles the limitation of the First Order Motion Model in generating facial animations with strong head rotations by replacing Jacobian transformations with a global rotation, achieving 40% to 80% bitrate savings on P-frames while improving performance on such movements.
First Order Motion Model is a generative model that animates human heads based on very little motion information derived from keypoints. It is a promising solution for video communication because first it operates at very low bitrate and second its computational complexity is moderate compared to other learning based video codecs. However, it has strong limitations by design. Since it generates facial animations by warping source-images, it fails to recreate videos with strong head movements. This works concentrates on one specific kind of head movements, namely head rotations. We show that replacing the Jacobian transformations in FOMM by a global rotation helps the system to perform better on items with head-rotations while saving 40% to 80% of bitrate on P-frames. Moreover, we apply state-of-the-art normalization techniques to the discriminator to stabilize the adversarial training which is essential for generating visually appealing videos. We evaluate the performance by the learned metics LPIPS and DISTS to show the success our optimizations.