Gimitest: A Comprehensive Tool for Testing Reinforcement Learning Policies
For RL practitioners, Gimitest provides a unified testing tool that reduces the pain point of evaluating policy reliability across varied scenarios.
The paper introduces Gimitest, an open-source framework for testing single- and multi-agent reinforcement learning policies across diverse environments, addressing the lack of comprehensive automated testing tools. It demonstrates effectiveness on Farama Gymnasium and PettingZoo environments.
Reinforcement learning (RL) policies can be unsafe and vulnerable to attacks. Ensuring their reliability is often a pain point as existing automated testing methods target only selected environments, testing scenarios, and RL algorithms. To address this, we propose a comprehensive framework for testing single- and multi-agent RL policies under varying conditions. Our implementation of this framework, Gimitest, is an open-source tool that supports various gym frameworks and allows for modifications of their integrated components. This article describes the framework and details Gimitest's functionality and architecture. It showcases its effectiveness in testing multiple RL policies in environments such as the official Farama Gymnasium and PettingZoo.