Hierarchical Zero-Order Optimization for Deep Neural Networks
This addresses the efficiency bottleneck for researchers and practitioners using zeroth-order methods in deep learning, representing a significant but incremental improvement over existing approaches.
The paper tackles the computational complexity of zeroth-order optimization in deep neural networks by proposing Hierarchical Zeroth-Order (HZO) optimization, which reduces query complexity from O(ML^2) to O(ML log L) and achieves competitive accuracy on CIFAR-10 and ImageNet compared to backpropagation.
Zeroth-order (ZO) optimization has long been favored for its biological plausibility and its capacity to handle non-differentiable objectives, yet its computational complexity has historically limited its application in deep neural networks. Challenging the conventional paradigm that gradients propagate layer-by-layer, we propose Hierarchical Zeroth-Order (HZO) optimization, a novel divide-and-conquer strategy that decomposes the depth dimension of the network. We prove that HZO reduces the query complexity from $O(ML^2)$ to $O(ML \log L)$ for a network of width $M$ and depth $L$, representing a significant leap over existing ZO methodologies. Furthermore, we provide a detailed error analysis showing that HZO maintains numerical stability by operating near the unitary limit ($L_{lip} \approx 1$). Extensive evaluations on CIFAR-10 and ImageNet demonstrate that HZO achieves competitive accuracy compared to backpropagation.