Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering
This addresses the challenge of making smaller, more practical language models effective for software engineers, though it is incremental as it builds on test-time scaling methods.
The paper tackles the problem of improving smaller language models' performance on real-world software engineering tasks like GitHub issue resolution, where they struggle compared to larger models, by proposing Evolutionary Test-Time Scaling (EvoScale) to reduce sampling needs and achieve sample efficiency. The result shows that their 32B model matches or exceeds the performance of models over 100B parameters on SWE-Bench-Verified with few samples.
Language models (LMs) perform well on standardized coding benchmarks but struggle with real-world software engineering tasks such as resolving GitHub issues in SWE-Bench, especially when model parameters are less than 100B. While smaller models are preferable in practice due to their lower computational cost, improving their performance remains challenging. Existing approaches primarily rely on supervised fine-tuning (SFT) with high-quality data, which is expensive to curate at scale. An alternative is test-time scaling: generating multiple outputs, scoring them using a verifier, and selecting the best one. Although effective, this strategy often requires excessive sampling and costly scoring, limiting its practical application. We propose Evolutionary Test-Time Scaling (EvoScale), a sample-efficient method that treats generation as an evolutionary process. By iteratively refining outputs via selection and mutation, EvoScale shifts the output distribution toward higher-scoring regions, reducing the number of samples needed to find correct solutions. To reduce the overhead from repeatedly sampling and selection, we train the model to self-evolve using reinforcement learning (RL). Rather than relying on external verifiers at inference time, the model learns to self-improve the scores of its own generations across iterations. Evaluated on SWE-Bench-Verified, EvoScale enables our 32B model, Satori-SWE-32B, to match or exceed the performance of models with over 100B parameters while using a few samples. Code, data, and models will be fully open-sourced.