Contrastive Embeddings for Neural Architectures
This work addresses the problem of search space parametrization dependency in Neural Architecture Search, which is a significant challenge for researchers and practitioners in the field.
This paper introduces a method to create architecture embeddings that are independent of the search space parametrization by using contrastive learning on data Jacobians. These embeddings enable traditional black-box optimization algorithms to achieve state-of-the-art performance in Neural Architecture Search and facilitate transfer learning between different search spaces.
The performance of algorithms for neural architecture search strongly depends on the parametrization of the search space. We use contrastive learning to identify networks across different initializations based on their data Jacobians, and automatically produce the first architecture embeddings independent from the parametrization of the search space. Using our contrastive embeddings, we show that traditional black-box optimization algorithms, without modification, can reach state-of-the-art performance in Neural Architecture Search. As our method provides a unified embedding space, we perform for the first time transfer learning between search spaces. Finally, we show the evolution of embeddings during training, motivating future studies into using embeddings at different training stages to gain a deeper understanding of the networks in a search space.