LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
This addresses a key bottleneck in long-context modeling for LLMs, offering a practical solution for efficient data selection, though it is incremental as it builds on existing attention mechanisms.
The paper tackles the challenge of measuring the quality of long-context training data for LLMs by proposing LADM, a framework that uses attention-based dependency measurement to select high-quality data, resulting in significant performance boosts on multiple long-context tasks with only 1B tokens for training.
Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However, it still remains an open challenge to measure the quality of long-context training data. To address this issue, we propose a Long-context data selection framework with Attention-based Dependency Measurement (LADM), which can efficiently identify high-quality long-context data from a large-scale, multi-domain pre-training corpus. LADM leverages the retrieval capabilities of the attention mechanism to capture contextual dependencies, ensuring a comprehensive quality measurement of long-context data. Experimental results show that our LADM framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training.