StepScorer: Accelerating Reinforcement Learning with Step-wise Scoring and Psychological Regret Modeling
This addresses the problem of slow learning in complex environments for applications like robotics, finance, and adaptive education, representing a novel method for a known bottleneck rather than a foundational breakthrough.
The paper tackled the slow convergence problem in reinforcement learning due to sparse rewards by introducing the Psychological Regret Model (PRM), which uses regret-based feedback signals after each step, resulting in stable performance approximately 36% faster than traditional Proximal Policy Optimization in benchmark environments like Lunar Lander.
Reinforcement learning algorithms often suffer from slow convergence due to sparse reward signals, particularly in complex environments where feedback is delayed or infrequent. This paper introduces the Psychological Regret Model (PRM), a novel approach that accelerates learning by incorporating regret-based feedback signals after each decision step. Rather than waiting for terminal rewards, PRM computes a regret signal based on the difference between the expected value of the optimal action and the value of the action taken in each state. This transforms sparse rewards into dense feedback signals through a step-wise scoring framework, enabling faster convergence. We demonstrate that PRM achieves stable performance approximately 36\% faster than traditional Proximal Policy Optimization (PPO) in benchmark environments such as Lunar Lander. Our results indicate that PRM is particularly effective in continuous control tasks and environments with delayed feedback, making it suitable for real-world applications such as robotics, finance, and adaptive education where rapid policy adaptation is critical. The approach formalizes human-inspired counterfactual thinking as a computable regret signal, bridging behavioral economics and reinforcement learning.