Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models
This addresses the 'right to be forgotten' for users in federated learning systems, offering an exact solution in a specific but incremental regime.
The paper tackles the problem of exact unlearning in federated settings with frozen foundation models and ridge-regression heads, achieving deterministic retrain-equivalence guarantees and matching centralized retraining within 10^{-9} relative Frobenius error at orders-of-magnitude lower cost.
Foundation models are commonly deployed as frozen feature extractors with a small trainable head to adapt to private, user-generated data in federated settings. The ``right to be forgotten'' requires removing the influence of specific samples or users from the trained model on demand. Existing federated unlearning methods target general deep models and rely on approximate reconstruction or selective retraining, making exactness costly or elusive. We study this problem in a practically relevant but under-explored regime: a frozen foundation model with a ridge-regression head. The exact optimum depends on the data only through two additive sufficient statistics, which we turn into a communication protocol supporting an arbitrary stream of add and delete requests via fixed-size messages. The server maintains a head that is, in exact arithmetic, pointwise identical to centralized retraining after every request. We provide deterministic retrain-equivalence guarantees, order and partition invariance, two server-side variants, and a Bayesian certificate of zero KL divergence. Experiments on four benchmarks confirm the guarantees: both variants match centralized ridge retraining to within $10^{-9}$ relative Frobenius error and complete each request at orders-of-magnitude lower cost than federated retraining baselines.