Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks
This addresses efficiency and effectiveness in language modeling, but appears incremental as it builds on existing attention and state space methods.
The authors tackled the problem of improving language model architectures by proposing inner function attention with dynamic mask and cross-domain mixture of experts, resulting in a more efficient and effective model called Wonderful Matrices for complex language tasks.
We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadratic causal self-attention and state space dual algorithms. We propose inner function attention with dynamic mask, which can improve the expressiveness of the attention algorithm and avoid the sequence noise significantly affecting the accuracy of the attention score. We also design cross domain mixture of experts, which can improve the granularity of the sparse activation feedforward network while maintaining the efficiency of parameter utilization and retrieval. The combination of these methods constitutes our foundation model architecture: Wonderful Matrices. We conduct experiments on the language modeling task and find that Wonderful Matrices are more efficient and effective in handling complex language tasks.