Tracr-Injection: Distilling Algorithms into Pre-trained Language Models
This addresses the problem of enhancing symbolic reasoning in language models for AI researchers, though it is incremental as it builds on existing RASP and distillation techniques.
The paper tackles the mismatch between theoretical symbolic capabilities of transformers and their practical learnability from unsupervised data by proposing tracr-injection, a method to distill algorithms written in RASP directly into pre-trained language models, resulting in an interpretable subspace and improved out-of-distribution performance compared to a baseline.
Motivated by the surge of large language models, there has been a push to formally characterize the symbolic abilities intrinsic to the transformer architecture. A programming language, called RASP, has been proposed, which can be directly compiled into transformer weights to implement these algorithms. However, the tasks that can be implemented in RASP are often uncommon to learn from natural unsupervised data, showing a mismatch between theoretical capabilities of the transformer architecture, and the practical learnability of these capabilities from unsupervised data. We propose tracr-injection, a method that allows us to distill algorithms written in RASP directly into a pre-trained language model. We showcase our method by injecting 3 different algorithms into a language model. We show how our method creates an interpretable subspace within the model's residual stream, which can be decoded into the variables present in the code of the RASP algorithm. Additionally, we found that the proposed method can improve out-of-distribution performance compared to our baseline, indicating that indeed a more symbolic mechanism is taking place in the inner workings of the model. We release the code used to run our experiments.