Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation
This work addresses the problem of assessing psychological traits using LLMs for researchers and practitioners in NLP, but it is incremental as it builds on existing methods.
The study evaluated GPT-3's zero-shot ability to estimate Big 5 personality traits from social media posts, finding that it performs close to a state-of-the-art model with trait knowledge in prompts for broad classification but drops to near a simple baseline for fine-grained classification.
Very large language models (LLMs) perform extremely well on a spectrum of NLP tasks in a zero-shot setting. However, little is known about their performance on human-level NLP problems which rely on understanding psychological concepts, such as assessing personality traits. In this work, we investigate the zero-shot ability of GPT-3 to estimate the Big 5 personality traits from users' social media posts. Through a set of systematic experiments, we find that zero-shot GPT-3 performance is somewhat close to an existing pre-trained SotA for broad classification upon injecting knowledge about the trait in the prompts. However, when prompted to provide fine-grained classification, its performance drops to close to a simple most frequent class (MFC) baseline. We further analyze where GPT-3 performs better, as well as worse, than a pretrained lexical model, illustrating systematic errors that suggest ways to improve LLMs on human-level NLP tasks.