Open Weight AI Models Require Proportional Evaluation Approaches
For policymakers, funders, and researchers evaluating AI safety, this paper highlights a critical gap in current evaluation practices for open-weight models.
Open-weight AI models (OWMs) pose distinct risks not addressed by existing evaluation practices designed for closed-weight models. The authors propose four proportional evaluation approaches and find that only 1 out of 37 OWM families released in 2025-2026 satisfies all four, with most fulfilling none.
Open-weight AI models (OWMs), or models released with publicly-available weights, are distributing rapidly and approaching the performance levels of leading closed-weight AI models (CWMs). While OWMs offer substantial scientific and economic benefits, their release introduces distinct risk factors for which existing evaluation practices, largely designed for CWM deployment, fail to account. In this paper, we argue that these risk factors demand distinct proportional evaluation (PE) approaches: evaluating without system-level safeguards (PE1), assessing robustness to modifications that undo model-level safeguards (PE2), testing selective capability amplification (PE3), and proxying worst-case misuse (PE4). We systematically review current evaluation practices of OWMs released in 2025 through April 2026, finding that only one of the 37 families of models reviewed fulfills PE1-4 and most do not fulfill any. This paper targets policymakers, funders, and researchers involved in AI evaluation. As OWMs grow increasingly capable, their evaluation warrants close attention from developers, funders, and governance bodies alike.