Bounds on Bayes Factors for Binomial A/B Testing
This work provides a theoretical link for statisticians and practitioners in fields like conversion testing, but it is incremental as it builds on existing tools from information geometry.
The paper tackles the problem of connecting Bayesian and frequentist approaches in binomial A/B testing by showing that Bayes factors are controlled by the Jensen-Shannon divergence of success ratios, which can be bounded by the Welch statistic, resulting in Bayesian sample bounds that almost match frequentist bounds.
Bayes factors, in many cases, have been proven to bridge the classic -value based significance testing and bayesian analysis of posterior odds. This paper discusses this phenomena within the binomial A/B testing setup (applicable for example to conversion testing). It is shown that the bayes factor is controlled by the \emph{Jensen-Shannon divergence} of success ratios in two tested groups, which can be further bounded by the Welch statistic. As a result, bayesian sample bounds almost match frequentionist's sample bounds. The link between Jensen-Shannon divergence and Welch's test as well as the derivation are an elegant application of tools from information geometry.