No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL
This addresses the challenge of hyperparameter sensitivity in RL for real-world applications like robotics, where online tuning is prohibitive, though it appears incremental as it builds on existing offline and simulation-based tuning approaches.
The paper tackles the problem of hyperparameter tuning in reinforcement learning (RL) by proposing an offline method that uses logged data to tune hyperparameters before online deployment, avoiding costly real-world testing, and empirically evaluates its effectiveness across various settings.
The performance of reinforcement learning (RL) agents is sensitive to the choice of hyperparameters. In real-world settings like robotics or industrial control systems, however, testing different hyperparameter configurations directly on the environment can be financially prohibitive, dangerous, or time consuming. We propose a new approach to tune hyperparameters from offline logs of data, to fully specify the hyperparameters for an RL agent that learns online in the real world. The approach is conceptually simple: we first learn a model of the environment from the offline data, which we call a calibration model, and then simulate learning in the calibration model to identify promising hyperparameters. We identify several criteria to make this strategy effective, and develop an approach that satisfies these criteria. We empirically investigate the method in a variety of settings to identify when it is effective and when it fails.