Magnus Ekman

Magnus Ekman

Simulation remains a cornerstone of computer architecture research, yet full end-to-end application execution is prohibitively time-consuming. The industry-standard solution, SimPoint, mitigates this cost by selecting a small number of representative code regions that capture program phase behavior. In this work, we take a fresh look at phase behavior in the SPEC CPU 2017 Integer suite to assess how pronounced such behavior truly is and what accuracy can be expected from typical SimPoint usage. Based on previously published data, we argue that common SimPoint counts can induce substantial estimation errors. To explore this further, we recast SimPoint as a stratified sampling problem, which enables the derivation of a conservative confidence interval. The analysis indicates that significant errors are expected, and our empirical analysis confirms this: with 20 SimPoints, two applications exhibit 40-60% performance prediction error. We decompose SimPoint into its two fundamental components - stratification (clustering) and sample-unit selection (centroid choice) - and analyze their individual effects on accuracy. We then extend the approach into a two-phase (double) sampling scheme, in which a large preliminary random sample enables improved stratification and more representative region selection. Using this method, the maximum per-application error drops to 3%. Finally, we demonstrate that the proposed two-phase stratified framework achieves an order-of-magnitude reduction in required sample size compared to simple random sampling while maintaining a tight analytical confidence interval, suggesting a practical path toward statistically grounded and efficient architectural simulation.

Magnus Ekman

Computer system simulation studies routinely rely on executing a limited number of short application regions, since full end-to-end simulation is prohibitively time-consuming. To preserve representativeness, existing methods employ either random sampling or phase-based characterization to identify representative regions. In this work, we revisit random sampling in the context of computer architecture simulation. To assess how the confidence level varies with different micro-architectural configurations, we examine how the sample standard deviation relates to the sample mean. We show that the ranked set sampling (RSS) technique - well established in the statistical literature - maps naturally to architectural simulation and yields significantly tighter confidence intervals than simple random sampling. Across our experiments, RSS reduces the confidence interval width by up to 50%. We further introduce a repeated subsampling scheme that identifies representative simulation regions for future studies. For a fixed sample size, this approach reduces the maximum observed error from 35% to 10%. Evaluating two selection criteria, we find that more informed subsample selection provides additional accuracy gains. Overall, our method achieves an average error below 2% and a maximum error of 3.5% across individual SPEC CPU 2017 Integer applications when simulating 30 regions of 1 million instructions each.