Linguistic Properties of Truthful Response
This addresses the challenge of truthfulness detection in AI-generated text for researchers and developers, but it is incremental due to limited scope and dataset constraints.
The study tackled the problem of detecting untruthful responses in LLMs by analyzing 220 handcrafted linguistic features, finding that linguistic profiles are similar across GPT-3 model sizes and training SVMs to classify truthfulness based on style alone, though dataset limitations constrain the results.
We investigate the phenomenon of an LLM's untruthful response using a large set of 220 handcrafted linguistic features. We focus on GPT-3 models and find that the linguistic profiles of responses are similar across model sizes. That is, how varying-sized LLMs respond to given prompts stays similar on the linguistic properties level. We expand upon this finding by training support vector machines that rely only upon the stylistic components of model responses to classify the truthfulness of statements. Though the dataset size limits our current findings, we show the possibility that truthfulness detection is possible without evaluating the content itself. But at the same time, the limited scope of our experiments must be taken into account in interpreting the results.