TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets
This addresses an under-explored problem in NAS for tabular data, offering a more effective approach for researchers and practitioners dealing with resource-constrained machine learning tasks, though it is incremental as it builds on existing RL-based NAS methods.
The paper tackles the problem of neural architecture search (NAS) for tabular datasets with resource constraints, developing TabNAS, which uses rejection sampling to discard architectures that violate constraints without training them, and demonstrates superiority over previous methods by finding better models that obey constraints.
The best neural architecture for a given machine learning problem depends on many factors: not only the complexity and structure of the dataset, but also on resource constraints including latency, compute, energy consumption, etc. Neural architecture search (NAS) for tabular datasets is an important but under-explored problem. Previous NAS algorithms designed for image search spaces incorporate resource constraints directly into the reinforcement learning (RL) rewards. However, for NAS on tabular datasets, this protocol often discovers suboptimal architectures. This paper develops TabNAS, a new and more effective approach to handle resource constraints in tabular NAS using an RL controller motivated by the idea of rejection sampling. TabNAS immediately discards any architecture that violates the resource constraints without training or learning from that architecture. TabNAS uses a Monte-Carlo-based correction to the RL policy gradient update to account for this extra filtering step. Results on several tabular datasets demonstrate the superiority of TabNAS over previous reward-shaping methods: it finds better models that obey the constraints.