CSCO: Connectivity Search of Convolutional Operators
This addresses the challenge of optimizing connectivity design in ConvNets for computer vision applications, representing an incremental advance over prior neural architecture search techniques.
The paper tackles the problem of discovering effective connectivity patterns for convolutional operators in computer vision, proposing CSCO, a novel paradigm that uses a neural predictor and evolutionary search to achieve a ~0.6% performance improvement on ImageNet over existing methods.
Exploring dense connectivity of convolutional operators establishes critical "synapses" to communicate feature vectors from different levels and enriches the set of transformations on Computer Vision applications. Yet, even with heavy-machinery approaches such as Neural Architecture Search (NAS), discovering effective connectivity patterns requires tremendous efforts due to either constrained connectivity design space or a sub-optimal exploration process induced by an unconstrained search space. In this paper, we propose CSCO, a novel paradigm that fabricates effective connectivity of convolutional operators with minimal utilization of existing design motifs and further utilizes the discovered wiring to construct high-performing ConvNets. CSCO guides the exploration via a neural predictor as a surrogate of the ground-truth performance. We introduce Graph Isomorphism as data augmentation to improve sample efficiency and propose a Metropolis-Hastings Evolutionary Search (MH-ES) to evade locally optimal architectures and advance search quality. Results on ImageNet show ~0.6% performance improvement over hand-crafted and NAS-crafted dense connectivity. Our code is publicly available.