ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation
This work aims to improve the robustness and performance of memory-efficient Neural Architecture Search for researchers and practitioners using DARTS-based methods, representing an incremental improvement.
The paper addresses the performance collapse in single-path Differentiable Architecture Search (DARTS) caused by an excess of parameter-free operations. Their proposed ROME algorithm disentangles topology and operation search, and uses Gumbel-Top2 reparameterization and gradient accumulation to robustify optimization, demonstrating effectiveness across 15 benchmarks.
Albeit being a prevalent architecture searching approach, differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. This is where the single-path DARTS comes in, which only chooses a single-path submodel at each step. While being memory-friendly, it also comes with low computational costs. Nonetheless, we discover a critical issue of single-path DARTS that has not been primarily noticed. Namely, it also suffers from severe performance collapse since too many parameter-free operations like skip connections are derived, just like DARTS does. In this paper, we propose a new algorithm called RObustifying Memory-Efficient NAS (ROME) to give a cure. First, we disentangle the topology search from the operation search to make searching and evaluation consistent. We then adopt Gumbel-Top2 reparameterization and gradient accumulation to robustify the unwieldy bi-level optimization. We verify ROME extensively across 15 benchmarks to demonstrate its effectiveness and robustness.