Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
Provides empirical guidance for practitioners in scientific and engineering domains on component-level decisions for actor-critic methods in real-world control systems.
This paper analyzes over 33,000 experiments on a real water treatment control task to evaluate how actor-critic design components affect reliability and hyperparameter sensitivity, finding that bounded action distributions with adaptive update schedules are more robust than common defaults like Gaussian distributions with pathwise gradient estimators.
Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variability across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These findings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapting actor-critic methods to new real-world control settings.