ConceptCarve: Dynamic Realization of Evidence
This work addresses the challenge of scalable evidence retrieval for abstract concepts in social media analysis, which is incremental as it builds on existing retrieval and LLM methods.
The paper tackles the problem of finding evidence for human opinions and behaviors at scale on social media, such as linking gun ownership to perceptions of freedom, by introducing ConceptCarve, a framework that uses traditional retrievers and LLMs to dynamically characterize the search space during retrieval. The result shows that ConceptCarve surpasses traditional retrieval systems in evidence retrieval within social media communities and produces interpretable representations for analyzing complex thought patterns across communities.
Finding evidence for human opinion and behavior at scale is a challenging task, often requiring an understanding of sophisticated thought patterns among vast online communities found on social media. For example, studying how gun ownership is related to the perception of Freedom, requires a retrieval system that can operate at scale over social media posts, while dealing with two key challenges: (1) identifying abstract concept instances, (2) which can be instantiated differently across different communities. To address these, we introduce ConceptCarve, an evidence retrieval framework that utilizes traditional retrievers and LLMs to dynamically characterize the search space during retrieval. Our experiments show that ConceptCarve surpasses traditional retrieval systems in finding evidence within a social media community. It also produces an interpretable representation of the evidence for that community, which we use to qualitatively analyze complex thought patterns that manifest differently across the communities.