Some Like It Small: Czech Semantic Embedding Models for Industry Applications
This work addresses the need for efficient Czech language processing in industry settings, though it is incremental as it applies known techniques like knowledge distillation to a specific domain.
The paper tackled the problem of developing small Czech sentence embedding models for resource-constrained industry applications, achieving competitive performance with approximately 8 times smaller size and 5 times faster speed compared to larger models.
This article focuses on the development and evaluation of Small-sized Czech sentence embedding models. Small models are important components for real-time industry applications in resource-constrained environments. Given the limited availability of labeled Czech data, alternative approaches, including pre-training, knowledge distillation, and unsupervised contrastive fine-tuning, are investigated. Comprehensive intrinsic and extrinsic analyses are conducted, showcasing the competitive performance of our models compared to significantly larger counterparts, with approximately 8 times smaller size and 5 times faster speed than conventional Base-sized models. To promote cooperation and reproducibility, both the models and the evaluation pipeline are made publicly accessible. Ultimately, this article presents practical applications of the developed sentence embedding models in Seznam.cz, the Czech search engine. These models have effectively replaced previous counterparts, enhancing the overall search experience for instance, in organic search, featured snippets, and image search. This transition has yielded improved performance.