Neural Decompiling of Tracr Transformers
This work addresses the interpretability challenge in neural networks for researchers, though it is incremental as it focuses on a specific compiler-generated dataset.
The paper tackles the problem of explaining transformer inner workings by developing a model to decompile Tracr-compiled transformer weights back into RASP programs, achieving exact reproductions on over 30% of test objects and functional equivalence in more than 70% of cases.
Recently, the transformer architecture has enabled substantial progress in many areas of pattern recognition and machine learning. However, as with other neural network models, there is currently no general method available to explain their inner workings. The present paper represents a first step towards this direction. We utilize \textit{Transformer Compiler for RASP} (Tracr) to generate a large dataset of pairs of transformer weights and corresponding RASP programs. Based on this dataset, we then build and train a model, with the aim of recovering the RASP code from the compiled model. We demonstrate that the simple form of Tracr compiled transformer weights is interpretable for such a decompiler model. In an empirical evaluation, our model achieves exact reproductions on more than 30\% of the test objects, while the remaining 70\% can generally be reproduced with only few errors. Additionally, more than 70\% of the programs, produced by our model, are functionally equivalent to the ground truth, and therefore a valid decompilation of the Tracr compiled transformer weights.