Flora: Effortless Context Construction to Arbitrary Length and Scale
This addresses the problem of costly and limited long-context training for LLM developers, offering a more efficient method, though it appears incremental as it builds on existing instruction-tuning approaches.
The paper tackles the challenge of handling long contexts in Large Language Models (LLMs) by introducing Flora, an effortless strategy that constructs long contexts from short instructions, resulting in enhanced performance on long-context benchmarks with minimal compromise on short-context tasks, as demonstrated on models like Llama3-8B-Instruct and QwQ-32B.
Effectively handling long contexts is challenging for Large Language Models (LLMs) due to the rarity of long texts, high computational demands, and substantial forgetting of short-context abilities. Recent approaches have attempted to construct long contexts for instruction tuning, but these methods often require LLMs or human interventions, which are both costly and limited in length and diversity. Also, the drop in short-context performances of present long-context LLMs remains significant. In this paper, we introduce Flora, an effortless (human/LLM-free) long-context construction strategy. Flora can markedly enhance the long-context performance of LLMs by arbitrarily assembling short instructions based on categories and instructing LLMs to generate responses based on long-context meta-instructions. This enables Flora to produce contexts of arbitrary length and scale with rich diversity, while only slightly compromising short-context performance. Experiments on Llama3-8B-Instruct and QwQ-32B show that LLMs enhanced by Flora excel in three long-context benchmarks while maintaining strong performances in short-context tasks. Our data-construction code is available at \href{https://github.com/txchen-USTC/Flora}{https://github.com/txchen-USTC/Flora}.