LIFT: Improving Long Context Understanding of Large Language Models through Long Input Fine-Tuning
This addresses the problem of limited context windows in LLMs for applications requiring long document processing, though it is an incremental improvement over existing fine-tuning methods.
The paper tackles the challenge of long context understanding in large language models by introducing Long Input Fine-Tuning (LIFT), a framework that fine-tunes long inputs into model parameters, enabling short-context models to answer questions without requiring the information in the context during inference, with results showing improved performance on long-context tasks.
Long context understanding remains challenging for large language models due to their limited context windows. This paper presents Long Input Fine-Tuning (LIFT), a novel framework for long-context modeling that can improve the long-context performance of arbitrary (short-context) LLMs by dynamically adapting model parameters based on the long input. Importantly, LIFT, rather than endlessly extending the context window size to accommodate increasingly longer inputs in context, chooses to store and absorb the long input in parameter. By fine-tuning the long input into model parameters, LIFT allows short-context LLMs to answer questions even when the required information is not provided in the context during inference. Furthermore, to enhance LIFT performance while maintaining the original in-context learning (ICL) capabilities, we introduce Gated Memory, a specialized attention adapter that automatically balances long input memorization and ICL. We provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research.