Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models
This addresses unlearning requests from regulators and users who lack access to massive training data, though it is incremental as it builds on existing unlearning concepts.
The paper tackles the problem of machine unlearning for foundation models by proposing a shift from data-tracing to knowledge-tracing, based on practical needs and cognitive insights, and illustrates this with a case study on a vision-language model.
Machine unlearning removes certain training data points and their influence on AI models (e.g., when a data owner revokes their decision to allow models to learn from the data). In this position paper, we propose to lift data-tracing machine unlearning to knowledge-tracing for foundation models (FMs). We support this position based on practical needs and insights from cognitive studies. Practically, tracing data cannot meet the diverse unlearning requests for FMs, which may be from regulators, enterprise users, product teams, etc., having no access to FMs' massive training data. Instead, it is convenient for these parties to issue an unlearning request about the knowledge or capability FMs (should not) possess. Cognitively, knowledge-tracing unlearning aligns with how the human brain forgets more closely than tracing individual training data points. Finally, we provide a concrete case study about a vision-language FM to illustrate how an unlearner might instantiate the knowledge-tracing machine unlearning paradigm.