Probing the Embedding Space of Transformers via Minimal Token Perturbations
This work provides a tool for interpretability in AI, but it is incremental as it builds on existing assumptions about Transformer layers.
The paper tackled the problem of understanding information propagation in Transformer models by studying minimal token perturbations on the embedding space, finding that rare tokens cause larger shifts and input information becomes more intermixed in deeper layers, validating the use of early layers for model explanations.
Understanding how information propagates through Transformer models is a key challenge for interpretability. In this work, we study the effects of minimal token perturbations on the embedding space. In our experiments, we analyze the frequency of which tokens yield to minimal shifts, highlighting that rare tokens usually lead to larger shifts. Moreover, we study how perturbations propagate across layers, demonstrating that input information is increasingly intermixed in deeper layers. Our findings validate the common assumption that the first layers of a model can be used as proxies for model explanations. Overall, this work introduces the combination of token perturbations and shifts on the embedding space as a powerful tool for model interpretability.