Not all Jensen-Shannon Divergence Estimators are Equal
For practitioners evaluating synthetic tabular data, this paper highlights that current JSD estimation protocols are incomparable and provides a correction for a known bias.
The paper shows that Jensen-Shannon divergence estimates for synthetic tabular data are highly protocol-dependent, with marginal estimators ignoring dependencies and classifier-based estimators showing sensitivity to class imbalance and dimensionality. They derive a posterior correction for prior-shift bias and provide guidelines for meaningful comparison.
The Jensen-Shannon divergence is widely reported as a scalar measure of fidelity for synthetic tabular data. Yet, in practice, it is estimated from finite samples using protocols that are often underspecified. This creates a measurement problem. Although the population divergence is well defined, the empirical value depends on the estimator family, sampling protocol, calibration, dimensionality, and class balance. We show that different protocols can yield non-comparable values: marginal-based estimators ignore dependencies in the joint distribution and can severely underestimate divergence, while classifier-based estimators capture joint structure but exhibit strong estimator dependence. We systematically study this behavior across controlled settings with reference divergences and real-world synthetic tabular benchmarks. Our analysis reveals dependence blindness in marginal estimators, prior-shift bias under class imbalance, and estimator sensitivity in high dimensions. To address prior shift, we derive a closed-form posterior correction for classifier-based Jensen-Shannon estimation. Our results show that empirical Jensen-Shannon divergence values are inherently protocol-dependent, making explicit specification of the estimation procedure necessary for meaningful comparison. We provide practical guidelines and an open-source tool for estimator-aware Jensen-Shannon evaluation.