Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings
For researchers and practitioners in online review analysis, this work provides a causal inference approach to isolate the impact of correlated aspects on overall ratings, though the enhancements are incremental.
The paper introduces a methodology based on CausalBERT to disentangle the effect of each aspect (e.g., school administration, benchmark performance) on overall online review ratings, validated on over 600K reviews of U.S. K-12 schools. The enhanced CausalBERT with temperature scaling, hyperparameter optimization, and interpretability methods yields more reliable estimates, identifying administration and benchmark performance as significant drivers.
Online reviews provide valuable insights into the perceived quality of facets of a product or service. While aspect-based sentiment analysis has focused on extracting these facets from reviews, there is less work understanding the impact of each aspect on overall perception. This is particularly challenging given correlations among aspects, making it difficult to isolate the effects of each. This paper introduces a methodology based on recent advances in text-based causal analysis, specifically CausalBERT, to disentangle the effect of each factor on overall review ratings. We enhance CausalBERT with three key improvements: temperature scaling for better calibrated treatment assignment estimates; hyperparameter optimization to reduce confound overadjustment; and interpretability methods to characterize discovered confounds. In this work, we treat the textual mentions in reviews as proxies for real-world attributes. We validate our approach on real and semi-synthetic data from over 600K reviews of U.S. K-12 schools. We find that the proposed enhancements result in more reliable estimates, and that perception of school administration and performance on benchmarks are significant drivers of overall school ratings.