Black-box language model explanation by context length probing
This work addresses the need for improved explainability in large language models, which is crucial for users and developers, but it appears incremental as it builds on existing probing methods.
The authors tackled the problem of explaining black-box language models by introducing context length probing, a model-agnostic technique that assigns importance scores to contexts based on prediction changes with varying context lengths, and they applied it to large pre-trained models to analyze long-range dependencies.
The increasingly widespread adoption of large language models has highlighted the need for improving their explainability. We present context length probing, a novel explanation technique for causal language models, based on tracking the predictions of a model as a function of the length of available context, and allowing to assign differential importance scores to different contexts. The technique is model-agnostic and does not rely on access to model internals beyond computing token-level probabilities. We apply context length probing to large pre-trained language models and offer some initial analyses and insights, including the potential for studying long-range dependencies. The source code and an interactive demo of the method are available.