Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?
For researchers using open models to interpret closed APIs, this work highlights a critical limitation: prediction-level agreement is insufficient for transferring mechanistic insights.
The paper investigates whether open language models can serve as surrogates for closed models in mechanistic interpretability, finding that prediction fidelity overstates attribution fidelity—models agreeing on answers often disagree on reasons—and that white-box signals are weakly predictive of causal attributions.
Mechanistic interpretability (MI) requires full access to model internals, yet the APIs for most widely deployed language models at best expose log-probabilities over output tokens. This creates a surrogate problem: when do measurements made on open models allow us to make claims about a closed model? We evaluate surrogate fidelity at the prediction, attribution, and representation levels. For binary classification tasks, log-odds provide an API-compatible scalar readout of the model's representation space, and leave-one-out attributions provide insight into model behavior. Across eleven models spanning four families (Llama, Qwen, GPT, and Gemini), we find that prediction fidelity substantially overstates attribution fidelity: models that agree on what the answer is often disagree on why. We document an access-validity inversion: white-box signals like attention patterns and perturbation magnitudes are highly stable across models but only weakly predictive of causal attributions, which black-box input ablations capture by design. Mechanistic insight does not automatically transfer to closed targets, and prediction-level agreement is insufficient to warrant such transfer. Code and results are available at https://github.com/facebookresearch/surrogate.