Introducing Explicit Gaze Constraints to Face Swapping
This work addresses a specific bottleneck in face swapping technology, enhancing realism for applications in entertainment and human-computer interaction, while also aiding Deepfake detection efforts.
The paper tackled the problem of inaccurate gaze reconstruction in face swapping by proposing a novel loss function that leverages gaze prediction during training, resulting in significant improvements in gaze accuracy for all tested methods.
Face swapping combines one face's identity with another face's non-appearance attributes (expression, head pose, lighting) to generate a synthetic face. This technology is rapidly improving, but falls flat when reconstructing some attributes, particularly gaze. Image-based loss metrics that consider the full face do not effectively capture the perceptually important, yet spatially small, eye regions. Improving gaze in face swaps can improve naturalness and realism, benefiting applications in entertainment, human computer interaction, and more. Improved gaze will also directly improve Deepfake detection efforts, serving as ideal training data for classifiers that rely on gaze for classification. We propose a novel loss function that leverages gaze prediction to inform the face swap model during training and compare against existing methods. We find all methods to significantly benefit gaze in resulting face swaps.