Interview-Informed Generative Agents for Product Discovery: A Validation Study
This work addresses the problem of using LLMs for product discovery, showing incremental progress by validating their limits and potential in design research workflows.
The study investigated whether interview-informed generative agents could simulate user responses in concept testing, finding that while agents approximated population-level response distributions, they failed to replicate specific individual responses.
Large language models (LLMs) have shown strong performance on standardized social science instruments, but their value for product discovery remains unclear. We investigate whether interview-informed generative agents can simulate user responses in concept testing scenarios. Using in-depth workflow interviews with knowledge workers, we created personalized agents and compared their evaluations of novel AI concepts against the same participants' responses. Our results show that agents are distribution-calibrated but identity-imprecise: they fail to replicate the specific individual they are grounded in, yet approximate population-level response distributions. These findings highlight both the potential and the limits of LLM simulation in design research. While unsuitable as a substitute for individual-level insights, simulation may provide value for early-stage concept screening and iteration, where distributional accuracy suffices. We discuss implications for integrating simulation responsibly into product development workflows.