A Validation Tool for Designing Reinforcement Learning Environments
This addresses a largely ignored challenge for RL practitioners in ensuring meaningful problem formulations, though it appears incremental as a tool for validation.
The paper tackles the problem of validating whether a designed Markov Decision Process (MDP) is suitable for reinforcement learning by proposing a heuristic-based feature analysis method, and it shows that the method can identify invalid environment formulations in constructed environments.
Reinforcement learning (RL) has gained increasing attraction in the academia and tech industry with launches to a variety of impactful applications and products. Although research is being actively conducted on many fronts (e.g., offline RL, performance, etc.), many RL practitioners face a challenge that has been largely ignored: determine whether a designed Markov Decision Process (MDP) is valid and meaningful. This study proposes a heuristic-based feature analysis method to validate whether an MDP is well formulated. We believe an MDP suitable for applying RL should contain a set of state features that are both sensitive to actions and predictive in rewards. We tested our method in constructed environments showing that our approach can identify certain invalid environment formulations. As far as we know, performing validity analysis for RL problem formulation is a novel direction. We envision that our tool will serve as a motivational example to help practitioners apply RL in real-world problems more easily.