ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining
This provides a practical solution for organizations lacking infrastructure to deploy large-scale models, though it appears incremental as it applies an existing adaptation method to sLLMs.
The paper tackles the challenge of deploying large language models in enterprise settings by developing ixi-GEN, a method using Domain Adaptive Continual Pretraining (DACP) on small LLMs (sLLMs). The result shows that ixi-GEN models achieve substantial gains in target-domain performance while preserving general capabilities, offering a cost-efficient and scalable solution.
The emergence of open-source large language models (LLMs) has expanded opportunities for enterprise applications; however, many organizations still lack the infrastructure to deploy and maintain large-scale models. As a result, small LLMs (sLLMs) have become a practical alternative despite inherent performance limitations. While Domain Adaptive Continual Pretraining (DACP) has been explored for domain adaptation, its utility in commercial settings remains under-examined. In this study, we validate the effectiveness of a DACP-based recipe across diverse foundation models and service domains, producing DACP-applied sLLMs (ixi-GEN). Through extensive experiments and real-world evaluations, we demonstrate that ixi-GEN models achieve substantial gains in target-domain performance while preserving general capabilities, offering a cost-efficient and scalable solution for enterprise-level deployment.