Memory Limitations of Prompt Tuning in Transformers
This provides a fundamental understanding of transformer limitations for handling long sequences, but it is incremental as it builds on existing empirical observations.
The paper tackled the theoretical limitations of prompt tuning in transformers, proving that memorization scales linearly with prompt length and demonstrating inherent memory constraints that cause performance degradation with extended contexts.
Despite the empirical success of prompt tuning in adapting pretrained language models to new tasks, theoretical analyses of its capabilities remain limited. Existing theoretical work primarily addresses universal approximation properties, demonstrating results comparable to standard weight tuning. In this paper, we explore a different aspect of the theory of transformers: the memorization capability of prompt tuning. We provide two principal theoretical contributions. First, we prove that the amount of information memorized by a transformer cannot scale faster than linearly with the prompt length. Second, and more importantly, we present the first formal proof of a phenomenon empirically observed in large language models: performance degradation in transformers with extended contexts. We rigorously demonstrate that transformers inherently have limited memory, constraining the amount of information they can retain, regardless of the context size. This finding offers a fundamental understanding of the intrinsic limitations of transformer architectures, particularly their ability to handle long sequences.