LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text
For survey researchers and experience analysts, this provides a baseline method to infer ratings from text and highlights a meaningful gap between narrative-driven predictions and holistic self-reports.
GPT-4.1 predicted fan experience ratings from open-ended text within one point for 67% of 10,000 responses (36% exact match), with near-deterministic scoring (87% exact agreement across runs). Predicted ratings correlated strongly with overall experience (r=0.82) but were systematically ~1 point lower than self-reported ratings, revealing a construct difference between salient moments and overall judgment.
We tasked GPT-4.1 to read what baseball fans wrote about their game-day experience and predict the overall experience rating each fan gave on a 0-10 survey scale. The model received only the text of a single open-ended response. These AI predictions were compared with the actual experience ratings captured by the survey instrument across approximately 10,000 fan responses from five Major League Baseball teams. In total two-thirds of predicted ratings fell within one point of self-reported fan ratings (67% within +/-1, 36% exact match), and the predicted measurement was near-deterministic across three independent scoring runs (87% exact agreement, 99.9% within +/-1). Predicted ratings aligned most strongly with the overall experience rating (r = 0.82) rather than with any specific aspect of the game-day experience such as parking, concessions, staff, etc. However, predictions were systematically lower than self-reported ratings by approximately one point, and this gap was not driven by any single aspect. Rather, our analysis shows that self-reported ratings capture the fan's verdict, an overall evaluative judgment that integrates the entire experience. While predicted ratings quantify the impact of salient moments characterized as memorable, emotionally intense, unusual, or actionable. Each measure contains information the other misses. These baseline results establish that a simple, unoptimized prompt can directionally predict how fans rate their experience from the text a fan wrote and that a gap between the two numbers can be interpreted as a construct difference worth preserving rather than an error to eliminate.