Problem-Parameter-Free Decentralized Bilevel Optimization
This addresses the problem of reducing manual effort in hyperparameter tuning for large-scale machine learning practitioners, though it is incremental as it builds on existing decentralized bilevel optimization methods.
The paper tackles the challenge of decentralized bilevel optimization by proposing AdaSDBO, a problem-parameter-free algorithm that eliminates the need for manual hyperparameter tuning, achieving a convergence rate of O~(1/T) and demonstrating competitive performance and robustness in experiments.
Decentralized bilevel optimization has garnered significant attention due to its critical role in solving large-scale machine learning problems. However, existing methods often rely on prior knowledge of problem parameters-such as smoothness, convexity, or communication network topologies-to determine appropriate stepsizes. In practice, these problem parameters are typically unavailable, leading to substantial manual effort for hyperparameter tuning. In this paper, we propose AdaSDBO, a fully problem-parameter-free algorithm for decentralized bilevel optimization with a single-loop structure. AdaSDBO leverages adaptive stepsizes based on cumulative gradient norms to update all variables simultaneously, dynamically adjusting its progress and eliminating the need for problem-specific hyperparameter tuning. Through rigorous theoretical analysis, we establish that AdaSDBO achieves a convergence rate of $\widetilde{\mathcal{O}}\left(\frac{1}{T}\right)$, matching the performance of well-tuned state-of-the-art methods up to polylogarithmic factors. Extensive numerical experiments demonstrate that AdaSDBO delivers competitive performance compared to existing decentralized bilevel optimization methods while exhibiting remarkable robustness across diverse stepsize configurations.