Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks
This work addresses the fundamental challenge of mechanistic interpretability in AI for researchers, though it is incremental in building on symbolic AI insights.
The paper tackles the problem of understanding how transformer networks achieve symbol processing through in-context learning, by developing a high-level language (PSL) to write symbolic programs and compiling them into interpretable transformer architectures, demonstrating that PSL is Turing Universal.
Large Language Models (LLMs) have demonstrated impressive abilities in symbol processing through in-context learning (ICL). This success flies in the face of decades of predictions that artificial neural networks cannot master abstract symbol manipulation. We seek to understand the mechanisms that can enable robust symbol processing in transformer networks, illuminating both the unanticipated success, and the significant limitations, of transformers in symbol processing. Borrowing insights from symbolic AI on the power of Production System architectures, we develop a high-level language, PSL, that allows us to write symbolic programs to do complex, abstract symbol processing, and create compilers that precisely implement PSL programs in transformer networks which are, by construction, 100% mechanistically interpretable. We demonstrate that PSL is Turing Universal, so the work can inform the understanding of transformer ICL in general. The type of transformer architecture that we compile from PSL programs suggests a number of paths for enhancing transformers' capabilities at symbol processing. (Note: The first section of the paper gives an extended synopsis of the entire paper.)