From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization
This addresses the exploration-exploitation trade-off in blackbox optimization for applications like reinforcement learning, though it appears incremental by combining existing techniques in a novel context.
The paper tackles the problem of optimizing high-dimensional blackbox functions with expensive queries by introducing ASEBO, an algorithm that adapts to the function's geometry and learns optimal sensing directions on-the-fly, resulting in more sample-efficient optimization than state-of-the-art methods, as demonstrated on reinforcement learning tasks and benchmark functions.
We present a new algorithm ASEBO for optimizing high-dimensional blackbox functions. ASEBO adapts to the geometry of the function and learns optimal sets of sensing directions, which are used to probe it, on-the-fly. It addresses the exploration-exploitation trade-off of blackbox optimization with expensive blackbox queries by continuously learning the bias of the lower-dimensional model used to approximate gradients of smoothings of the function via compressed sensing and contextual bandits methods. To obtain this model, it leverages techniques from the emerging theory of active subspaces in the novel ES blackbox optimization context. As a result, ASEBO learns the dynamically changing intrinsic dimensionality of the gradient space and adapts to the hardness of different stages of the optimization without external supervision. Consequently, it leads to more sample-efficient blackbox optimization than state-of-the-art algorithms. We provide theoretical results and test ASEBO advantages over other methods empirically by evaluating it on the set of reinforcement learning policy optimization tasks as well as functions from the recently open-sourced Nevergrad library.