The Crucial Role of Problem Formulation in Real-World Reinforcement Learning
This work addresses the gap between RL research and real-world industrial systems, offering incremental improvements through problem formulation design.
The paper tackles the limited real-world adoption of reinforcement learning in industrial cyber-physical systems by showing that well-designed modifications to the RL problem formulation improve performance, stability, and sample efficiency, with experiments on a 1-DoF helicopter testbed validating these results in both simulation and physical hardware.
Reinforcement Learning (RL) offers promising solutions for control tasks in industrial cyber-physical systems (ICPSs), yet its real-world adoption remains limited. This paper demonstrates how seemingly small but well-designed modifications to the RL problem formulation can substantially improve performance, stability, and sample efficiency. We identify and investigate key elements of RL problem formulation and show that these enhance both learning speed and final policy quality. Our experiments use a one-degree-of-freedom (1-DoF) helicopter testbed, the Quanser Aero~2, which features non-linear dynamics representative of many industrial settings. In simulation, the proposed problem design principles yield more reliable and efficient training, and we further validate these results by training the agent directly on physical hardware. The encouraging real-world outcomes highlight the potential of RL for ICPS, especially when careful attention is paid to the design principles of problem formulation. Overall, our study underscores the crucial role of thoughtful problem formulation in bridging the gap between RL research and the demands of real-world industrial systems.