Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks?
This addresses the problem of efficiently supervising LLMs for complex reasoning tasks, offering practical strategies for data augmentation and training, though it appears incremental in optimizing existing supervision methods.
The paper investigates how weak teacher models can effectively supervise LLMs on hard math reasoning tasks, finding that training on high-error-rate hard task supervision (e.g., 90% error) can outperform perfect easier subtask supervision, and identifying step-wise error rates as a critical factor influencing performance with a 30% accuracy gap on the MATH benchmark.
How can "weak teacher models" such as average human annotators or existing AI systems, effectively supervise LLMs to improve performance on hard reasoning tasks, especially those that challenge and requires expertise or daily practice from the teacher models? In this paper, we seek for empirical answers to this question by investigating various data-driven strategies that offer supervision data at different quality levels upon tasks of varying complexity. Two intuitive strategies emerge for teacher models to provide supervision during alignment training: 1) using lower-quality supervision from complete tasks that match the difficulty of the target reasoning tasks, and 2) leveraging higher-quality supervision from easier subtasks that are less challenging. Interestingly, we find that even when the outcome error rate for hard task supervision is high (e.g., 90\%), training on such data can outperform perfectly correct supervision of easier subtasks on multiple hard math benchmarks. We further identify a more critical factor influencing training performance: step-wise error rates, which indicate the severity of errors in solutions. Specifically, training on hard task supervision with the same outcome error rates but disparate step-wise error rates can lead to a 30\% accuracy gap on MATH benchmark. Our results also reveal that supplementing hard task supervision with the corresponding subtask supervision can yield notable performance improvements than simply combining rephrased hard full task supervision, suggesting new avenues for data augmentation. Data and code are released at https://github.com/hexuan21/Weak-to-Strong.