Trillion 7B Technical Report
This addresses the high computational cost of multilingual LLM training for researchers and developers, though it appears incremental in optimizing existing approaches.
The paper tackles the problem of building token-efficient multilingual large language models by introducing Trillion-7B, which achieves competitive performance across 27 benchmarks in four languages while using only 10% of its 2T training tokens for multilingual data and costing $148K for full training.
We introduce Trillion-7B, the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient and effective knowledge transfer from English to target languages like Korean and Japanese. Combined with optimized data mixtures, language-specific filtering, and tailored tokenizer construction, Trillion-7B achieves competitive performance while dedicating only 10\% of its 2T training tokens to multilingual data and requiring just 59.4K H100 GPU hours (\$148K) for full training. Comprehensive evaluations across 27 benchmarks in four languages demonstrate Trillion-7B's robust multilingual performance and exceptional cross-lingual consistency.