Internalizing Tools as Morphisms in Graded Transformers
This work addresses the challenge of making transformers more interpretable and capable of symbolic reasoning, which is incremental as it builds on existing graded transformer and tool-use paradigms.
The paper tackles the problem of integrating symbolic computation into transformers by introducing a graded formulation where symbolic operations are realized as typed block maps activated by a differentiable routing policy, achieving sparse and interpretable behavior through a self-supervised graded utility functional. It unifies symbolic computation, geometry, and self-supervised learning within the graded transformer formalism, subsuming prior external-tool paradigms as a special case.
We introduce a graded formulation of internal symbolic computation for transformers. The hidden space is endowed with a grading $V=\bigoplus_{g\in G}V_g$, and symbolic operations are realized as typed block maps (morphisms) $φ_{h\leftarrow g}:V_g\to V_h$ that are activated selectively by a differentiable routing policy. A self-supervised \emph{graded utility functional}, defined as the loss reduction induced by a candidate morphism, governs activation and yields sparse, interpretable behavior. We develop the algebraic and geometric foundations: an internal model category whose objects are homogeneous components and whose morphisms are admissible grade transitions; adjoint pairs encoding typed round trips; and information-geometric interpretations in terms of KL gain, mirror descent with Bregman divergences, and Fisher natural gradients. Methodologically, we specify a utility--aware routing mechanism and objective that remain fully end-to-end differentiable. Analytic case studies and lightweight sanity checks illustrate selective morphic activation on hybrid symbolic-linguistic tasks. The framework unifies symbolic computation, geometry, and self--supervised learning within the \emph{graded transformer} formalism \cite{sh-89,sh-95}, while subsuming prior external-tool paradigms (e.g., Toolformer \cite{toolformer2023}) as a special case via functorial internalization.