Directional Ensemble Aggregation for Actor-Critics
This work addresses a specific bottleneck in actor-critic methods for continuous control tasks, offering an incremental improvement over existing ensemble aggregation techniques.
The paper tackled the problem of overestimation bias in off-policy reinforcement learning for continuous control by proposing Directional Ensemble Aggregation (DEA), which adaptively combines Q-value estimates using learnable parameters, resulting in improved performance over static ensemble strategies across various benchmarks and learning regimes.
Off-policy reinforcement learning in continuous control tasks depends critically on accurate $Q$-value estimates. Conservative aggregation over ensembles, such as taking the minimum, is commonly used to mitigate overestimation bias. However, these static rules are coarse, discard valuable information from the ensemble, and cannot adapt to task-specific needs or different learning regimes. We propose Directional Ensemble Aggregation (DEA), an aggregation method that adaptively combines $Q$-value estimates in actor-critic frameworks. DEA introduces two fully learnable directional parameters: one that modulates critic-side conservatism and another that guides actor-side policy exploration. Both parameters are learned using ensemble disagreement-weighted Bellman errors, which weight each sample solely by the direction of its Bellman error. This directional learning mechanism allows DEA to adjust conservatism and exploration in a data-driven way, adapting aggregation to both uncertainty levels and the phase of training. We evaluate DEA across continuous control benchmarks and learning regimes - from interactive to sample-efficient - and demonstrate its effectiveness over static ensemble strategies.