SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation
For researchers in machine unlearning, SUPREME reduces the computational cost of multi-seed evaluations, enabling more rigorous benchmarking.
SUPREME is a multi-GPU framework for evaluating image unlearning methods that distributes training, unlearning, and evaluation across multiple GPUs, enabling ten-seed evaluations on Pins Face Recognition with ResNet18 and ViT.
Machine unlearning removes the influence of specific training data from a trained model without retraining it from scratch. Evaluating an unlearning method requires repeating training, unlearning, and evaluation across multiple seeds, which is computationally expensive. To our knowledge, existing image classification unlearning frameworks run on a single GPU, which limits how many seeds can be evaluated in reasonable time. We introduce SUPREME, an open-source framework that distributes these stages across multiple GPUs. SUPREME makes three contributions: a registry-based design for adding new methods, metrics, models, and scenarios; a multi-GPU architecture supporting multiple accelerators and precision modes; and a demonstration on Pins Face Recognition using ResNet18 and ViT under full-class and random-sample unlearning across ten seeds. The framework is available at https://github.com/pedroandreou/supreme-unlearning.