Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
This work addresses the need for efficient deployment of large hybrid MoE models under high user throughput constraints, enabling practical interactive serving and ultra-long-context applications.
Nemotron-Labs-3-Puzzle-75B-A9B is a compressed hybrid MoE LLM that achieves 2x higher server throughput than its parent model on a single 8xB200 node under interactive workloads, and increases 1M-token concurrency from 1 to 8 requests on a single H100 GPU, while retaining strong downstream accuracy.
We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability.