Demystifying optimized prompts in language models
This work addresses the interpretability and robustness issues of language models for AI researchers and practitioners, but it is incremental as it builds on existing knowledge about prompt optimization.
The paper investigated the composition and internal processing of optimized prompts in language models, finding they consist of rare punctuation and noun tokens and are distinguishable from natural language through sparse activations, with consistent representation formation across model families.
Modern language models (LMs) are not robust to out-of-distribution inputs. Machine generated (``optimized'') prompts can be used to modulate LM outputs and induce specific behaviors while appearing completely uninterpretable. In this work, we investigate the composition of optimized prompts, as well as the mechanisms by which LMs parse and build predictions from optimized prompts. We find that optimized prompts primarily consist of punctuation and noun tokens which are more rare in the training data. Internally, optimized prompts are clearly distinguishable from natural language counterparts based on sparse subsets of the model's activations. Across various families of instruction-tuned models, optimized prompts follow a similar path in how their representations form through the network.