Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks
This work addresses the need for standardized guidelines in Computational Social Science to improve model performance on social knowledge tasks, though it is incremental as it synthesizes existing methods rather than introducing new ones.
The paper tackled the problem of establishing best practices for using Large Language Models in Computational Social Science by evaluating performance on 23 social knowledge tasks, finding that selecting models with larger vocabularies and pre-training corpora, using AI-enhanced prompting over zero-shot, and fine-tuning on task-specific data are effective strategies.
Large Language Models are expressive tools that enable complex tasks of text understanding within Computational Social Science. Their versatility, while beneficial, poses a barrier for establishing standardized best practices within the field. To bring clarity on the values of different strategies, we present an overview of the performance of modern LLM-based classification methods on a benchmark of 23 social knowledge tasks. Our results point to three best practices: select models with larger vocabulary and pre-training corpora; avoid simple zero-shot in favor of AI-enhanced prompting; fine-tune on task-specific data, and consider more complex forms instruction-tuning on multiple datasets only when only training data is more abundant.