DLLG: Dynamic Logit-Level Gating of LLM Experts
For practitioners combining multiple LLMs, DLLG offers a more robust and scalable alternative to existing methods that suffer from premature routing, fragile proxies, or parameter interference.
DLLG introduces a dynamic logit-level gating framework that learns token-level fusion weights from sparse response-level supervision, enabling robust integration of multiple specialized LLMs without token-level labels or expert retraining. It consistently outperforms routing, heuristic ensembling, and parameter-merging baselines across diverse reasoning and code benchmarks.
Leveraging multiple specialized LLMs can combine complementary strengths, but existing approaches trade adaptability for stability: routing commits prematurely, heuristic ensembling depends on fragile proxies, and parameter merging introduces interference. We propose DLLG (Dynamic Logit-Level Gating), a dynamic logit-level ensembling framework that learns token-level expert fusion from sparse response-level supervision. A lightweight gating module predicts step-wise fusion weights, linking trajectory-level correctness to generation without token-level labels or expert retraining. Across diverse reasoning and code benchmarks, DLLG consistently outperforms strong routing, heuristic ensembling, and parameter-merging baselines across model scales, highlighting learned logit-level fusion as a robust and scalable paradigm for integrating specialized experts.