Timber: Training-free Instruct Model Refining with Base via Effective Rank
This work addresses the trade-off in post-training for large language models, offering a practical refinement strategy without retraining, though it is incremental.
The paper tackles the superficiality of post-training that converts pretrained Base models into Instruct models, showing that effective rank changes negligibly and proposing Timber, a training-free method to enhance exploration while preserving exploitation, which improves Pass@k performance on Llama and Qwen series.
Post-training, which elicits a pretrained Base model into the corresponding Instruct model, is widely considered to be superficial. In this work, we first reinforce this hypothesis by providing novel quantitative evidence from the weight level that the effective rank (eRank) remains negligibly changed. However, this superficiality also suffers a critical trade-off, improving the exploitation capabilities at the cost of limiting its exploration. To tackle this issue, we propose Timber, a simple yet effective training-free method that enhances the exploration capability of the Instruct model while preserving its exploitation. The key insight is to partially revert Instruct towards the paired Base model by subtle yet targeted refinement of the weight deltas. Extensive experiments on Llama and Qwen series demonstrate that Timber consistently improves vanilla Instruct models, particularly on Pass@k performance. Our findings offer new insights into the post-training stage at the weight level and practical strategies to refine the Instruct model without training.