PUMA: Performance Unchanged Model Augmentation for Training Data Removal
This addresses the need for efficient and performance-preserving data removal in machine learning, offering a novel solution that avoids retraining and mitigates generalization loss, though it is incremental in the context of existing data removal techniques.
The paper tackles the problem of removing marked training data from a model without degrading its performance, and presents PUMA, which reweights remaining data to maintain generalization while effectively removing unique characteristics, as shown by fooling membership attacks and resisting performance degradation.
Preserving the performance of a trained model while removing unique characteristics of marked training data points is challenging. Recent research usually suggests retraining a model from scratch with remaining training data or refining the model by reverting the model optimization on the marked data points. Unfortunately, aside from their computational inefficiency, those approaches inevitably hurt the resulting model's generalization ability since they remove not only unique characteristics but also discard shared (and possibly contributive) information. To address the performance degradation problem, this paper presents a novel approach called Performance Unchanged Model Augmentation~(PUMA). The proposed PUMA framework explicitly models the influence of each training data point on the model's generalization ability with respect to various performance criteria. It then complements the negative impact of removing marked data by reweighting the remaining data optimally. To demonstrate the effectiveness of the PUMA framework, we compared it with multiple state-of-the-art data removal techniques in the experiments, where we show the PUMA can effectively and efficiently remove the unique characteristics of marked training data without retraining the model that can 1) fool a membership attack, and 2) resist performance degradation. In addition, as PUMA estimates the data importance during its operation, we show it could serve to debug mislabelled data points more efficiently than existing approaches.