Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contribution
This addresses computational inefficiency and suboptimal partitioning in few-shot NAS for graph neural networks, though it appears incremental by building on existing few-shot NAS methods.
The paper tackles the weight coupling problem in few-shot Neural Architecture Search by analyzing gradient direction conflicts and proposing a Gradient Contribution method for efficient supernet partitioning, achieving state-of-the-art performance in partitioning quality and time efficiency. It also introduces a Unified Graph Neural Architecture Search framework that explores optimal combinations of Message Passing Neural Networks and Graph Transformers, with searched architectures outperforming manually designed GNNs and existing NAS methods.
To address the weight coupling problem, certain studies introduced few-shot Neural Architecture Search (NAS) methods, which partition the supernet into multiple sub-supernets. However, these methods often suffer from computational inefficiency and tend to provide suboptimal partitioning schemes. To address this problem more effectively, we analyze the weight coupling problem from a novel perspective, which primarily stems from distinct modules in succeeding layers imposing conflicting gradient directions on the preceding layer modules. Based on this perspective, we propose the Gradient Contribution (GC) method that efficiently computes the cosine similarity of gradient directions among modules by decomposing the Vector-Jacobian Product during supernet backpropagation. Subsequently, the modules with conflicting gradient directions are allocated to distinct sub-supernets while similar ones are grouped together. To assess the advantages of GC and address the limitations of existing Graph Neural Architecture Search methods, which are limited to searching a single type of Graph Neural Networks (Message Passing Neural Networks (MPNNs) or Graph Transformers (GTs)), we propose the Unified Graph Neural Architecture Search (UGAS) framework, which explores optimal combinations of MPNNs and GTs. The experimental results demonstrate that GC achieves state-of-the-art (SOTA) performance in supernet partitioning quality and time efficiency. In addition, the architectures searched by UGAS+GC outperform both the manually designed GNNs and those obtained by existing NAS methods. Finally, ablation studies further demonstrate the effectiveness of all proposed methods.