LLMs on Drugs: Language Models Are Few-Shot Consumers
This addresses the issue of prompt sensitivity for LLM users, showing that persona text can destroy reliability, but it is incremental as it builds on known persona effects.
The study tackled the problem of how psychoactive framings in prompts affect large language models (LLMs) by benchmarking them on ARC-Challenge, finding that prompts like alcohol reduced accuracy from 0.45 to 0.10, significantly degrading performance.
Large language models (LLMs) are sensitive to the personas imposed on them at inference time, yet prompt-level "drug" interventions have never been benchmarked rigorously. We present the first controlled study of psychoactive framings on GPT-5-mini using ARC-Challenge. Four single-sentence prompts -- LSD, cocaine, alcohol, and cannabis -- are compared against a sober control across 100 validation items per condition, with deterministic decoding, full logging, Wilson confidence intervals, and Fisher exact tests. Control accuracy is 0.45; alcohol collapses to 0.10 (p = 3.2e-8), cocaine to 0.21 (p = 4.9e-4), LSD to 0.19 (p = 1.3e-4), and cannabis to 0.30 (p = 0.041), largely because persona prompts disrupt the mandated "Answer: <LETTER>" template. Persona text therefore behaves like a "few-shot consumable" that can destroy reliability without touching model weights. All experimental code, raw results, and analysis scripts are available at https://github.com/lexdoudkin/llms-on-drugs.