Compass-V2 Technical Report
This addresses the problem of language and domain bias in LLMs for users in Southeast Asia and e-commerce, though it is incremental as it builds on existing MoE and multilingual techniques.
The authors tackled the underrepresentation of Southeast Asian languages and limited e-commerce focus in LLMs by introducing Compass-v2, a lightweight Mixture-of-Experts model with 30B total and 5B active parameters, which achieved state-of-the-art performance in these areas among sub-30B models while reducing inference costs.
Predominant LLMs focus on high-resource languages while leaving low-resource languages, particularly those in Southeast Asia (SEA), underrepresented. In addition, those models are general-purpose and pay limited attention to the e-commerce domain. To overcome these limitations, we introduce Compass-v2, a lightweight Mixture-of-Experts (MoE) model specifically designed for Southeast Asian languages and e-commerce applications. To balance model performance and inference cost, the model is designed with 30B total parameters and 5B active parameters, incorporating both fine-grained and shared expert modules. To enhance multilingual performance, we curated and constructed a high-quality, industry-leading SEA dataset, to the best of our knowledge. To boost performance in the e-commerce domain, we built a dataset comprising hundreds of billions of tokens, sourced through external data mining and internal platform collection. Besides, we pioneered a hybrid reasoning model that supports both fast thinking and deep thinking within a unified framework to enhance the reasoning capabilities, diverging from the conventional industry practice of deploying two separate models. Through extensive experimental evaluations, our model demonstrates state-of-the-art SEA multilingual and e-commerce performance among sub-30B models, while maintaining significantly lower inference cost.