Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach
This addresses shortcut reasoning in language models, which can hinder generalization, but it is incremental as it builds on existing gradient-based methods.
The paper tackles the problem of language models relying on shortcuts like pattern matching instead of logical reasoning by proposing Shortcut-Aware Reasoning Training (SART), a gradient-aware framework that improves accuracy by +16.5% and robustness by +40.2% over baselines on reasoning benchmarks.
Large language models exhibit strong reasoning capabilities, yet often rely on shortcuts such as surface pattern matching and answer memorization rather than genuine logical inference. We propose Shortcut-Aware Reasoning Training (SART), a gradient-aware framework that detects and mitigates shortcut-promoting samples via ShortcutScore and gradient surgery. Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration, and modifies training dynamics accordingly. Experiments on controlled reasoning benchmarks show that SART achieves +16.5% accuracy and +40.2% robustness over the strongest baseline, significantly improving generalization under distribution shifts. Code is available at: https://github.com/fuyanjie/short-cut-aware-data-centric-reasoning.