SafeMath: Inference-time Safety improves Math Accuracy
This addresses safety risks in educational settings for children by mitigating biased or unethical content in math problems, representing an incremental improvement in safety alignment for LLMs.
The paper tackles the problem of harmful content in mathematical word problems by introducing the ToxicGSM dataset and proposing SafeMath, a safety alignment technique that reduces harmful outputs while maintaining or improving mathematical reasoning performance, with results showing effective safety without sacrificing accuracy.
Recent research points toward LLMs being manipulated through adversarial and seemingly benign inputs, resulting in harmful, biased, or policy-violating outputs. In this paper, we study an underexplored issue concerning harmful and toxic mathematical word problems. We show that math questions, particularly those framed as natural language narratives, can serve as a subtle medium for propagating biased, unethical, or psychologically harmful content, with heightened risks in educational settings involving children. To support a systematic study of this phenomenon, we introduce ToxicGSM, a dataset of 1.9k arithmetic problems in which harmful or sensitive context is embedded while preserving mathematically well-defined reasoning tasks. Using this dataset, we audit the behaviour of existing LLMs and analyse the trade-offs between safety enforcement and mathematical correctness. We further propose SafeMath -- a safety alignment technique that reduces harmful outputs while maintaining, and in some cases improving, mathematical reasoning performance. Our results highlight the importance of disentangling linguistic harm from math reasoning and demonstrate that effective safety alignment need not come at the cost of accuracy. We release the source code and dataset at https://github.com/Swagnick99/SafeMath/tree/main.