When Does Pairing Seeds Reduce Variance? Evidence from a Multi-Agent Economic Simulation
This work addresses the issue of statistical efficiency in evaluating machine learning systems for researchers and practitioners, offering an incremental improvement over standard practices.
The paper tackles the problem of variance in comparative evaluation of machine learning systems by analyzing the use of shared random seeds to induce matched stochastic realisations, showing that this approach reduces variance when outcomes are positively correlated. In a multi-agent economic simulator, paired evaluation with shared seeds revealed systematic differences in aggregate and distributional outcomes that were statistically inconclusive under independent evaluation at fixed budgets.
Machine learning systems appear stochastic but are deterministically random, as seeded pseudorandom number generators produce identical realisations across repeated executions. Standard evaluation practice typically treats runs across alternatives as independent and does not exploit shared sources of randomness. This paper analyses the statistical structure of comparative evaluation under shared random seeds. Under this design, competing systems are evaluated using identical seeds, inducing matched stochastic realisations and yielding strict variance reduction whenever outcomes are positively correlated at the seed level. We demonstrate these effects using an extended learning-based multi-agent economic simulator, where paired evaluation exposes systematic differences in aggregate and distributional outcomes that remain statistically inconclusive under independent evaluation at fixed budgets.