Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models
This research addresses a practical problem for LLM developers by exploring an understudied aspect of training efficiency and model capabilities, though it appears incremental in nature.
The study investigated how document-packing strategies affect the latent multi-hop reasoning abilities of large language models, finding that packing can improve performance compared to training on individual documents, though at the cost of increased compute.
The standard practice for training large language models involves packing multiple documents together to optimize computational efficiency. However, the impact of this process on the models' capabilities remains largely unexplored. To address this gap, we investigate how different document-packing strategies influence the latent multi-hop reasoning abilities of LLMs. Our findings indicate that packing can improve model performance compared to training on individual documents, at the expense of more compute. To further understand the underlying mechanisms, we conduct an ablation study, identifying key factors that explain the advantages of packing. Ultimately, our research deepens the understanding of LLM training dynamics and provides practical insights for optimizing model development.