David Lo

Wendkûuni C. Ouédraogo, Yinghua Li, Xueqi Dang et al.

Bug resolution is a cross-artifact process: natural-language reports must expose actionable behavioral cues that can be reproduced by tests and corrected by patches. Yet the extent to which these signals are preserved across artifacts remains largely unquantified. We introduce Desc2Fix, a framework for measuring semantic alignment between bug reports, triggering tests, and developer-written fixes. Alignment is operationalized through structured behavioral anchors (e.g., reproduction steps, API/exception cues, expected vs. actual behavior), deterministic similarity metrics (ROUGE, SBERT, CodeBERT, OpenAI embeddings), and LLM-based judgments grounded in coverage, correctness, and specificity. We analyze 2,857 report-test-patch triplets from Defects4J and SWT-Bench using GPT-4o and DeepSeek-Chat. LLMs reliably extract structured signals (up to 90% completeness) and exhibit strong cross-model consistency, yielding a stable semantic input contract for downstream reasoning. However, alignment is highly representation-sensitive: lexical similarity alone is insufficient; full diffs provide the most stable basis for judging report-patch correspondence; and structured summaries trade surface similarity for predicate-level salience. Across more than 182,000 LLM alignment ratings, both models show systematic optimism relative to humans (1-2 points on 5-point scales) and only modest rank agreement, motivating bias-aware evaluation. Our results demonstrate that behavioral alignment is measurable but not reducible to similarity, and that structured anchors combined with embedding-based proxies provide reproducible signals for ranking and filtering tests and candidate patches. By transforming alignment into a controllable engineering signal, Desc2Fix enables more reliable test generation, semantics-aware fault localization, principled patch ranking, and improved bug report authoring.

Weifeng Sun, Ye Fan, Yuchen Chen et al.

Large language models (LLMs) excel at general-purpose code generation, yet how well they handle scientific code remains an open question. Existing datasets and benchmarks are limited in scale, domain coverage, or executable verification, leaving the true gap between current LLMs and reliable scientific code generators inadequately assessed. To address these limitations, we present SciCodePile, the largest scientific code corpus to date, constructed from 37,737 public repositories and collectively comprising 128GB of code that spans multiple computational science disciplines. From this corpus, we further curate an executable benchmark of 200 tasks, each equipped with a sandboxed execution environment and an automated test harness for functional verification. We evaluate 15 LLMs from both open-source and closed-source families on three tasks: prefix-to-suffix completion, fill-in-the-middle infilling, and executable code generation. Results show that scientific code generation remains highly challenging: The best CodeBLEU reaches only 38.13 and 38.37 on the two completion tasks, while the strongest model achieves just 12.30\% Pass@1 on the executable benchmark, underscoring how far current models remain from reliable scientific code generation. To demonstrate the training utility of SciCodePile, we further show that continued pretraining on our corpus improves CodeBLEU by $\times$2.84 on scientific code completion, and instruction tuning on our data improves Pass@1 by $\times$4.79 on the executable benchmark. All code and data are available at https://huggingface.co/SciCodePile.