Low-Perplexity LLM-Generated Sequences and Where To Find Them
This addresses transparency and accountability issues for LLM users and developers, though it is incremental in building on existing data analysis methods.
The paper tackled the problem of understanding how LLM training data influences outputs by analyzing low-perplexity sequences, finding that many cannot be traced to the training corpus and quantifying verbatim recall for those that do.
As Large Language Models (LLMs) become increasingly widespread, understanding how specific training data shapes their outputs is crucial for transparency, accountability, privacy, and fairness. To explore how LLMs leverage and replicate their training data, we introduce a systematic approach centered on analyzing low-perplexity sequences - high-probability text spans generated by the model. Our pipeline reliably extracts such long sequences across diverse topics while avoiding degeneration, then traces them back to their sources in the training data. Surprisingly, we find that a substantial portion of these low-perplexity spans cannot be mapped to the corpus. For those that do match, we quantify the distribution of occurrences across source documents, highlighting the scope and nature of verbatim recall and paving a way toward better understanding of how LLMs training data impacts their behavior.