Beyond Application End-Point Results: Quantifying Statistical Robustness of MCMC Accelerators
This work addresses a critical gap for hardware designers and domain experts by providing a methodology to assess statistical properties beyond accuracy, though it is incremental as it builds on existing evaluation practices.
The paper tackles the problem of incomplete evaluation methods for probabilistic accelerators like MCMC, which often focus only on end-point accuracy, by proposing a framework with three metrics for statistical robustness and applying it to an MCMC accelerator to expose design issues and achieve comparable robustness to software with minimal hardware changes.
Statistical machine learning often uses probabilistic algorithms, such as Markov Chain Monte Carlo (MCMC), to solve a wide range of problems. Probabilistic computations, often considered too slow on conventional processors, can be accelerated with specialized hardware by exploiting parallelism and optimizing the design using various approximation techniques. Current methodologies for evaluating correctness of probabilistic accelerators are often incomplete, mostly focusing only on end-point result quality ("accuracy"). It is important for hardware designers and domain experts to look beyond end-point "accuracy" and be aware of the hardware optimizations impact on other statistical properties. This work takes a first step towards defining metrics and a methodology for quantitatively evaluating correctness of probabilistic accelerators beyond end-point result quality. We propose three pillars of statistical robustness: 1) sampling quality, 2) convergence diagnostic, and 3) goodness of fit. We apply our framework to a representative MCMC accelerator and surface design issues that cannot be exposed using only application end-point result quality. Applying the framework to guide design space exploration shows that statistical robustness comparable to floating-point software can be achieved by slightly increasing the bit representation, without floating-point hardware requirements.