$n$-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models
This work addresses the challenge of efficient collaboration among language models for reasoning tasks, offering a compact mechanism with observational insights into expert utilization, though it is incremental as it builds on existing RLVR methods.
The paper tackles the problem of enabling structured reasoning in small, specialized language models (SLMs) without large monolithic LLMs by introducing soft hidden-state collaboration, where multiple heterogeneous frozen SLM experts are integrated through their internal representations via a trainable attention interface, resulting in competitive performance with strong single-model RLVR baselines on Reasoning Gym and GSM8K.
Recent progress in reinforcement learning with verifiable rewards (RLVR) shows that small, specialized language models (SLMs) can exhibit structured reasoning without relying on large monolithic LLMs. We introduce soft hidden-state collaboration, where multiple heterogeneous frozen SLM experts are integrated through their internal representations via a trainable attention interface. Experiments on Reasoning Gym and GSM8K show that this latent integration is competitive with strong single-model RLVR baselines. Ablations further reveal a dual mechanism of expert utilization: for simpler arithmetic domains, performance gains can largely be explained by static expert preferences, whereas more challenging settings induce increasingly concentrated and structured expert attention over training, indicating emergent specialization in how the router connects to relevant experts. Overall, hidden-state collaboration provides a compact mechanism for leveraging frozen experts, while offering an observational window into expert utilization patterns and their evolution under RLVR.