Rethinking Neural Operations for Diverse Tasks
This work addresses the challenge for AutoML users in discovering optimal neural operations for under-explored domains, offering a novel approach that is not incremental but provides broad applicability.
The paper tackles the problem of automating neural network design for new tasks by introducing XD-Operations, a search space that includes many known operations and is more expressive than standard convolutions, resulting in lower error across diverse tasks such as PDE solving, protein folding, and music modeling compared to baselines and expert-designed methods.
An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains. Motivated by this goal, we study the problem of enabling users to discover the right neural operations given data from their specific domain. We introduce a search space of operations called XD-Operations that mimic the inductive bias of standard multi-channel convolutions while being much more expressive: we prove that it includes many named operations across multiple application areas. Starting with any standard backbone such as ResNet, we show how to transform it into a search space over XD-operations and how to traverse the space using a simple weight-sharing scheme. On a diverse set of tasks -- solving PDEs, distance prediction for protein folding, and music modeling -- our approach consistently yields models with lower error than baseline networks and often even lower error than expert-designed domain-specific approaches.