Inform Product Change through Experimentation with Data-Driven Behavioral Segmentation
This work addresses the need for finer analysis in online experimentation to inform product changes, though it appears incremental as it builds on existing segmentation approaches.
The paper tackles the problem of understanding why treatment effects occur in online controlled experiments by introducing a framework for creating user behavioral segments based on engagement with product components, and demonstrates with a real-world example that this analysis provided deep, actionable insights for product decision-making.
Online controlled experimentation is widely adopted for evaluating new features in the rapid development cycle for web products and mobile applications. Measurement of the overall experiment sample is a common practice to quantify the overall treatment effect. In order to understand why the treatment effect occurs in a certain way, segmentation becomes a valuable approach to a finer analysis of experiment results. This paper introduces a framework for creating and utilizing user behavioral segments in online experimentation. By using the data of user engagement with individual product components as input, this method defines segments that are closely related to the features being evaluated in the product development cycle. With a real-world example, we demonstrate that the analysis with such behavioral segments offered deep, actionable insights that successfully informed product decision-making.