The Effect of Document Selection on Query-focused Text Analysis
For researchers performing query-focused text analysis, this work establishes data selection as a methodological decision rather than a practical necessity, providing guidance on effective selection strategies.
The paper systematically evaluates seven document selection methods across four text analysis techniques on two datasets with 26 queries, finding that semantic or hybrid retrieval are robust go-to approaches that avoid pitfalls of weaker strategies and unnecessary compute overhead.
Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints preclude analyzing all documents, yet little work has examined effects of selection strategy choices. We systematically evaluate seven selection methods (from random selection to hybrid retrieval) on outputs from four text analyses methods (LDA, BERTopic, TopicGPT, HiCode) over two datasets with 26 open-ended queries. Our evaluation reveals practice guidance: semantic or hybrid retrieval offer strong go-to approaches that avoid the pitfalls of weaker selection strategies and the unnecessary compute overhead of more complicated ones. Overall, our evaluation framework establishes data selection as a methodological decision, rather than a practical necessity, inviting the development of new strategies.