DiffMI: Breaking Face Recognition Privacy via Diffusion-Driven Training-Free Model Inversion
This work addresses privacy vulnerabilities for users of face recognition systems, offering a more efficient and adaptable attack method, though it is incremental as it builds on existing model inversion techniques.
The paper tackles the problem of privacy risks in face recognition systems by proposing DiffMI, a diffusion-driven, training-free model inversion attack that achieves 84.42%--92.87% attack success rates and outperforms prior training-free GAN-based approaches by 4.01%--9.82%.
Face recognition poses serious privacy risks due to its reliance on sensitive and immutable biometric data. While modern systems mitigate privacy risks by mapping facial images to embeddings (commonly regarded as privacy-preserving), model inversion attacks reveal that identity information can still be recovered, exposing critical vulnerabilities. However, existing attacks are often computationally expensive and lack generalization, especially those requiring target-specific training. Even training-free approaches suffer from limited identity controllability, hindering faithful reconstruction of nuanced or unseen identities. In this work, we propose DiffMI, the first diffusion-driven, training-free model inversion attack. DiffMI introduces a novel pipeline combining robust latent code initialization, a ranked adversarial refinement strategy, and a statistically grounded, confidence-aware optimization objective. DiffMI applies directly to unseen target identities and face recognition models, offering greater adaptability than training-dependent approaches while significantly reducing computational overhead. Our method achieves 84.42%--92.87% attack success rates against inversion-resilient systems and outperforms the best prior training-free GAN-based approach by 4.01%--9.82%. The implementation is available at https://github.com/azrealwang/DiffMI.