Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention
This work addresses interpretability, adaptability, and scalability issues in Transformers for AI research, though it appears incremental as it builds on existing architectures with a novel theoretical insight.
The paper tackles the challenge of monolithic Transformer architectures by introducing a modular design that decouples knowledge and reasoning using a generalized cross-attention mechanism, with a theoretical derivation showing the Feed-Forward Network as a specialized case of this mechanism.
Transformers have achieved remarkable success across diverse domains, but their monolithic architecture presents challenges in interpretability, adaptability, and scalability. This paper introduces a novel modular Transformer architecture that explicitly decouples knowledge and reasoning through a generalized cross-attention mechanism to a globally shared knowledge base with layer-specific transformations, specifically designed for effective knowledge retrieval. Critically, we provide a rigorous mathematical derivation demonstrating that the Feed-Forward Network (FFN) in a standard Transformer is a specialized case (a closure) of this generalized cross-attention, revealing its role in implicit knowledge retrieval and validating our design. This theoretical framework provides a new lens for understanding FFNs and lays the foundation for future research exploring enhanced interpretability, adaptability, and scalability, enabling richer interplay with external knowledge bases and other systems.