System 2 Attention (is something you might need too)
This addresses a key issue in LLMs for improving reliability in tasks with opinion or irrelevant information, though it is an incremental improvement over existing attention mechanisms.
The paper tackles the problem of soft attention in Transformer-based LLMs incorporating irrelevant context, which harms next token generation, by introducing System 2 Attention (S2A) that regenerates input to include only relevant portions before attending; S2A outperforms standard attention-based LLMs on tasks like QA and math word problems, increasing factuality and objectivity while decreasing sycophancy.
Soft attention in Transformer-based Large Language Models (LLMs) is susceptible to incorporating irrelevant information from the context into its latent representations, which adversely affects next token generations. To help rectify these issues, we introduce System 2 Attention (S2A), which leverages the ability of LLMs to reason in natural language and follow instructions in order to decide what to attend to. S2A regenerates the input context to only include the relevant portions, before attending to the regenerated context to elicit the final response. In experiments, S2A outperforms standard attention-based LLMs on three tasks containing opinion or irrelevant information, QA, math word problems and longform generation, where S2A increases factuality and objectivity, and decreases sycophancy.