scicode-lint: Detecting Methodology Bugs in Scientific Python Code with LLM-Generated Patterns
This addresses the need for sustainable, automated methodology checking in scientific software, especially as AI-generated code increases, by reducing reliance on manual engineering for pattern updates.
The paper tackles the problem of detecting methodology bugs in scientific Python code, which produce plausible but incorrect results, by introducing scicode-lint, a tool that uses LLM-generated patterns for automated checking. It achieves up to 65% precision at 100% recall for preprocessing leakage detection on Kaggle notebooks and 97.7% accuracy across 66 patterns in controlled tests.
Methodology bugs in scientific Python code produce plausible but incorrect results that traditional linters and static analysis tools cannot detect. Several research groups have built ML-specific linters, demonstrating that detection is feasible. Yet these tools share a sustainability problem: dependency on specific pylint or Python versions, limited packaging, and reliance on manual engineering for every new pattern. As AI-generated code increases the volume of scientific software, the need for automated methodology checking (such as detecting data leakage, incorrect cross-validation, and missing random seeds) grows. We present scicode-lint, whose two-tier architecture separates pattern design (frontier models at build time) from execution (small local model at runtime). Patterns are generated, not hand-coded; adapting to new library versions costs tokens, not engineering hours. On Kaggle notebooks with human-labeled ground truth, preprocessing leakage detection reaches 65% precision at 100% recall; on 38 published scientific papers applying AI/ML, precision is 62% (LLM-judged) with substantial variation across pattern categories; on a held-out paper set, precision is 54%. On controlled tests, scicode-lint achieves 97.7% accuracy across 66 patterns.