ECR: Manifold-Guided Semantic Cues for Compact Language Models
This addresses the issue of semantic drift in compact models for deployment under efficiency or privacy constraints, though it is incremental as it builds on existing compression methods.
The paper tackled the problem of embedding space collapse in compact language models, which causes semantic drift and hampers downstream tasks, by introducing the Embedding Consistency Regulation (ECR) framework. The result showed that ECR stabilizes training and preserves semantic structure across tasks and languages on a 100K multilingual corpus, enabling low-capacity models to learn cleaner manifolds than conventional baselines.
Compact models often lose the structure of their embedding space. The issue shows up when the capacity is tight or the data spans several languages. Such collapse makes it difficult for downstream tasks to build on the resulting representation. Existing compression methods focus on aligning model outputs at a superficial level but fail to preserve the underlying manifold structure. This mismatch often leads to semantic drift in the compact model, causing both task behavior and linguistic properties to deviate from the reference model. To address those issues, we provide a new framework called Embedding Consistency Regulation (ECR). This framework first derives a set of semantic anchors from teacher embeddings (computed once offline). Then, the compact model learns to maintain consistent geometry around these anchors, without relying on matching logits or internal features. ECR adds only a small projection step at inference, without altering the decoding architecture or its runtime behavior. In experiments on a 100K multilingual corpus, ECR consistently stabilizes training and preserves semantic structure across tasks and languages. It also produces a more compact and task-aligned representation space, enabling low-capacity models to learn cleaner manifolds than conventional baselines. ECR works without teacher outputs and is compatible with, but independent of, distillation. Taken together, our results show that ECR helps compact models better follow task requirements and makes them easier to deploy under strict efficiency or privacy limits.