Splits! A Flexible Dataset and Evaluation Framework for Sociocultural Linguistic Investigation
This provides a systematic tool for researchers in sociocultural linguistics to accelerate discovery, though it is incremental as it builds on existing data and methods.
The authors tackled the challenge of limited computational study of sociocultural linguistic phenomena by introducing Splits!, a 9.7 million-post dataset from Reddit with 53,000 users across 6 demographic groups and 89 topics, and a framework that reduces statistically significant findings needing manual inspection by 1.5-1.8x.
Variation in language use, shaped by speakers' sociocultural background and specific context of use, offers a rich lens into cultural perspectives, values, and opinions. However, the computational study of these Sociocultural Linguistic Phenomena (SLP) has often been limited to bespoke analyses of specific groups or topics, hindering the pace of scientific discovery. To address this, we introduce Splits!, a 9.7 million-post dataset from Reddit designed for systematic and flexible research. The dataset contains posts from over 53,000 users across 6 demographic groups, organized into 89 discussion topics to enable comparative analysis. We validate Splits! via self-identification and by successfully replicating several known SLPs from existing literature. We complement this dataset with a framework that leverages efficient retrieval methods to rapidly validate potential SLPs (PSLPs) by automatically evaluating whether a given hypothesis is supported by our data. Crucially, to distinguish between novel and obvious insights, the framework incorporates a human-validated measure of a hypothesis's ``unexpectedness.'' We demonstrate that the two-stage process reduces the number of statistically significant findings requiring manual inspection by a factor of 1.5-1.8x, streamlining the discovery of promising phenomena for further investigation.