Pre-Training on Software Engineering Texts: Effects on Domain Adaptation and General-Language Understanding
This provides practical guidance for practitioners adapting LMs to software engineering text, but the findings are incremental and confirm the expected advantage of continual pre-training.
The paper investigates how to adapt language models to software engineering text, comparing continual pre-training (CPT) against pre-training from scratch (PTS). It finds that CPT yields small domain gains while preserving general-language understanding, whereas PTS incurs large penalties on both axes, being competitive only for small models with abundant tokens.
Generalist and code-focused Language Models (LMs) are increasingly applied to software engineering (SE), yet whether they are optimized for understanding SE textual artifacts (e.g., issues, commit messages, developer discussions) remains unclear, as most evidence comes from code-focused benchmarks. We study how to adapt encoder and decoder LMs to SE text, comparing continual pre-training (CPT) against pre-training from scratch (PTS) on a new SE corpus, and evaluating both domain adaptation (SELU) and general-language understanding (SuperGLUE). To keep the comparisons fair, we control pre-training under constant-token and compute-matched budgets. We find that across families and sizes, reusing an existing LM dominates training a domain-native one from scratch: CPT yields small and mostly inconclusive domain gains while leaving general-language understanding essentially unchanged, whereas PTS pays a large and usually decisive penalty on both axes and becomes competitive only for small LMs under a token-rich budget. We distill these results into practical guidance for adapting LMs to SE text and release our corpus and pre-trained LMs in our replication kit.