Parameter Golf: What Really Works?
For practitioners training small models under tight resource budgets, this work identifies which optimization techniques consistently improve performance across competitive submissions.
The Parameter Golf challenge asked participants to train the best language model under strict constraints (16 MB artifact, <10 min on 8xH100). Analyzing 1,430 submissions, the leaderboard score improved by 13.6% (from 1.2244 to 1.058 BPB) through a combination of 84 techniques, though most individual gains were <1%.
How far can a language model improve under a strict artifact budget? Parameter Golf posed this question as an open community challenge in which participants trained the best language model, with the complete artifact (training code + compressed weights) required to fit within 16 MB and be trained in under ten minutes on 8xH100 SXM GPUs. Quality was measured in bits-per-byte (BPB), the average number of bits required to encode each byte of unseen text. We analyze 2,037 pull requests and 1,430 clean scored submissions from the contest, build a taxonomy of 84 optimization techniques, and measure each technique's contribution to BPB. The verified leaderboard score dropped from 1.2244 to 1.058 BPB across three phases -- a 13.6% reduction, despite individual techniques rarely improving BPB by more than 1%. We show that most gains in techniques shrink across competitive submissions, isolating the few methods that improve performance across stacks.