A general tensor-structured compression scheme for efficient large language models
For practitioners deploying large language models, MixT offers a general compression method that significantly reduces computational and memory costs with minimal accuracy loss.
The paper introduces Tensor Mixture (MixT), a general tensor-structured compression scheme for large language models that replaces dense linear layers with mixtures of tensor operators. On LLaMA2-7B, MixT reduces parameters by 47.5%, inference FLOPs by 37.1%, and peak inference memory by 60.4% while largely preserving MMLU accuracy.
Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5\%, inference FLOPs by 37.1\%, training FLOPs by 52.1\% and peak inference memory by 60.4\%, demonstrating its practical potential for lower-cost LLM compression.