Sharp Multiple Instance Learning for DeepFake Video Detection
This addresses security concerns in multimedia and computer vision by improving detection of manipulated videos, though it is incremental as it builds on existing multiple instance learning methods.
The paper tackles the problem of detecting DeepFake videos with partial face attacks, where only video-level labels are available and not all faces are manipulated, by proposing a sharp multiple instance learning (S-MIL) framework that builds direct mapping from instance embeddings to bag prediction to relieve gradient vanishing. Experiments on the FFPMS and DFDC datasets show S-MIL is superior for partially attacked video detection and achieves state-of-the-art performance on single-frame image detection tasks.
With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.