Reasoning-Aware GRPO using Process Mining
This work addresses the need for better reasoning capabilities in AI models, though it appears incremental as it builds on existing GRPO methods.
The paper tackles the problem of outcome-centric reward schemes in RL-based post-training for large reasoning models by proposing PM4GRPO, which uses process mining to add reasoning procedure signals, resulting in significant performance improvements on five benchmarks.
Reinforcement learning (RL)-based post-training has been crucial for enabling multi-step reasoning in large reasoning models (LRMs), yet current reward schemes are typically outcome-centric. We propose PM4GRPO, a reasoning-aware Group Relative Policy Optimization (GRPO) that augments standard answer/format rewards with signals over the reasoning procedure. To this end, process mining techniques are utilized to compute a scalar conformance reward that measures how closely a policy model's reasoning aligns with the pretrained teacher model. The empirical results on five benchmarks demonstrate that PM4GRPO significantly outperforms existing methodologies for GRPO-based post-training. These results highlight that leveraging process mining for reasoning-aware GRPO effectively enhances the reasoning capabilities of policy models.