TARo: Token-level Adaptive Routing for LLM Test-time Alignment
This addresses the need for lightweight, efficient alignment for reasoning tasks in LLMs, offering a novel approach beyond preference optimization.
The paper tackles the problem of improving reasoning in large language models without expensive post-training by proposing Token-level Adaptive Routing (TARo), a test-time alignment method that enhances reasoning performance by up to +22.4% over the base model and +8.4% over existing methods.
Large language models (LLMs) exhibit strong reasoning capabilities but typically require expensive post-training to reach high performance. Recent test-time alignment methods offer a lightweight alternative, but have been explored mainly for preference alignment rather than reasoning. To bridge this gap, we propose, Token-level Adaptive Routing (TARo), which steers frozen LLMs toward structured reasoning entirely at inference time. Specifically, we first train reward models on step-wise mathematical traces to capture fine-grained logical consistency signals, then introduce a learnable token-level router that automatically controls the guidance of the reward model to the base model. Extensive experiments show that TARo significantly improves reasoning performance by up to +22.4% over base model and +8.4% over existing token-level test-time alignment methods, while also boosting out-of-distribution clinical reasoning (MedXpertQA) and instruction following (AlpacaEval). Furthermore, TARo also generalizes from small to large backbones without retraining, extending test-time alignment from preference optimization to robust, cross-domain reasoning.