Data Attribution in Large Language Models via Bidirectional Gradient Optimization
It addresses the need for interpretability and accountability in LLMs by enabling precise identification of influential training data, which is critical for governance and data provenance.
The paper proposes a method for training data attribution in auto-regressive LLMs using bidirectional gradient optimization, achieving state-of-the-art performance on influence metrics for both factual and stylistic attribution.
Large Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance. Understanding which training data most influenced a model's output remains a fundamental open problem. We address this challenge through training data attribution (TDA) for auto-regressive LLMs by expanding upon the inverse formulation: How would training data be affected if the model had seen the generated output during training? Our method perturbs the base model using bidirectional gradient optimization (gradient ascent and descent) on a generated text sample and measures the resulting change in loss across training samples. Our framework supports attribution at arbitrary data granularity, enabling both factual and stylistic attribution. We evaluate our method against baselines on pretrained models with known datasets, and show that it outperforms previous work on influence metrics, thereby enhancing model interpretability, an essential requirement for accountable AI systems.