Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference
For NLI researchers, a simple debiasing method that reduces spurious correlations without significant accuracy loss.
Neural NLI models overfit dataset artifacts; Product-of-Experts training reduces bias reliance by 4.71% while nearly preserving accuracy (89.10% vs. 89.30%).
Neural NLI models overfit dataset artifacts instead of truly reasoning. A hypothesis-only model gets 57.7% in SNLI, showing strong spurious correlations, and 38.6% of the baseline errors are the result of these artifacts. We propose Product-of-Experts (PoE) training, which downweights examples where biased models are overconfident. PoE nearly preserves accuracy (89.10% vs. 89.30%) while cutting bias reliance by 4.71% (bias agreement 49.85% to 45%). An ablation finds lambda = 1.5 that best balances debiasing and accuracy. Behavioral tests still reveal issues with negation and numerical reasoning.