Validation of massively-parallel adaptive testing using dynamic control matching
This addresses the challenge for businesses running many simultaneous and adaptive marketing tests, but it is incremental as it builds on existing causal inference and control group techniques.
The paper tackles the problem of evaluating causal effects in dynamic parallel A/B/n testing, where traditional methods fail due to continuous adaptation, by proposing a method using a matched-synthetic control group that adapts alongside tests.
A/B testing is a widely-used paradigm within marketing optimization because it promises identification of causal effects and because it is implemented out of the box in most messaging delivery software platforms. Modern businesses, however, often run many A/B/n tests at the same time and in parallel, and package many content variations into the same messages, not all of which are part of an explicit test. Whether as the result of many teams testing at the same time, or as part of a more sophisticated reinforcement learning (RL) approach that continuously adapts tests and test condition assignment based on previous results, dynamic parallel testing cannot be evaluated the same way traditional A/B tests are evaluated. This paper presents a method for disentangling the causal effects of the various tests under conditions of continuous test adaptation, using a matched-synthetic control group that adapts alongside the tests.