Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
For practitioners deploying LLMs with long chain-of-thought reasoning, PIPO reduces decoding latency without sacrificing output quality, addressing a critical bottleneck in autoregressive inference.
PIPO unifies input-side latent compression and output-side multi-token prediction to reduce inference cost in LLMs, achieving up to 2.64× first-token-latency and 2.07× per-token-latency speedups while improving pass@4 by up to +7.15 points on benchmarks like AIME 2025 and GPQA-Diamond.
Long chain-of-thought reasoning has made autoregressive decoding the dominant inference cost of modern large language models. Existing methods target either the input side (latent compression) or the output side (speculative decoding and multi-token prediction, MTP), but the two lines of work have been pursued independently. Moreover, output-side methods must incur an expensive verifier pass to validate the unreliable draft tokens predicted by MTP. To address these issues, we propose \textbf{Pair-In, Pair-Out (PIPO)}, which unifies both sides by viewing a latent compressor and an MTP head as mirror-image operations: the compressor folds two input tokens into one latent representation, while the MTP head unfolds one hidden state into one additional output token. To remove the verifier cost without sacrificing reliability, PIPO trains a lightweight confidence head that decides whether draft tokens should be accepted. We observe that On-Policy Distillation (OPD) naturally matches the rejection-sampling criterion of speculative decoding, so the confidence head can be trained alongside OPD with negligible extra cost. Experiments on AIME 2025, GPQA-Diamond, LiveCodeBench v6, and LongBench v2 with Qwen3.5-4B and 9B backbones show that PIPO improves pass@4 over regular decoding by up to $+7.15$ points, while delivering up to $2.64\times$ first-token-latency and $2.07\times$ per-token-latency speedups.