Tina: Tiny Reasoning Models via LoRA
This work addresses the problem of high computational costs for developing reasoning models in AI, offering a highly efficient solution that is incremental but with substantial practical impact.
The paper tackles the problem of achieving strong reasoning abilities in language models cost-effectively by introducing Tina, a family of tiny reasoning models that use parameter-efficient updates via LoRA during reinforcement learning on a 1.5B parameter base model. The result is models that achieve competitive or superior reasoning performance to SOTA RL reasoning models with the same base, at a 260x cost reduction, including a >20% reasoning performance increase and 43.33% Pass@1 accuracy on AIME24 at only $9 USD post-training cost.
How cost-effectively can strong reasoning abilities be achieved in language models? Driven by this fundamental question, we present Tina, a family of tiny reasoning models achieved with high cost-efficiency. Notably, Tina demonstrates that substantial reasoning performance can be developed using only minimal resources, by applying parameter-efficient updates during reinforcement learning (RL), using low-rank adaptation (LoRA), to an already tiny 1.5B parameter base model. This minimalist approach produces models that achieve reasoning performance which is competitive with, and sometimes surpasses, SOTA RL reasoning models built upon the same base model. Crucially, this is achieved at a tiny fraction of the computational post-training cost employed by existing SOTA models. In fact, the best Tina model achieves a >20\% reasoning performance increase and 43.33\% Pass@1 accuracy on AIME24, at only \$9 USD post-training and evaluation cost (i.e., an estimated 260x cost reduction). Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA. We validate this across multiple open-source reasoning datasets and various ablation settings starting with a single, fixed set of hyperparameters. Furthermore, we hypothesize that this effectiveness and efficiency stem from LoRA rapidly adapting the model to the structural format of reasoning rewarded by RL, while largely preserving the base model's underlying knowledge. In service of accessibility and open research, we fully open-source all code, training logs, and model weights \& checkpoints.