RepNAS: Searching for Efficient Re-parameterizing Blocks
This work addresses the problem of designing efficient neural networks for deployment in resource-constrained environments, though it appears incremental by building on existing re-parameterization techniques.
The authors tackled the challenge of efficiently searching for neural architectures that balance performance and inference speed by proposing RepNAS, a one-stage NAS approach that searches for optimal diverse branch blocks under branch number constraints, achieving results that surpass manually designed blocks with efficient training.
In the past years, significant improvements in the field of neural architecture search(NAS) have been made. However, it is still challenging to search for efficient networks due to the gap between the searched constraint and real inference time exists. To search for a high-performance network with low inference time, several previous works set a computational complexity constraint for the search algorithm. However, many factors affect the speed of inference(e.g., FLOPs, MACs). The correlation between a single indicator and the latency is not strong. Currently, some re-parameterization(Rep) techniques are proposed to convert multi-branch to single-path architecture which is inference-friendly. Nevertheless, multi-branch architectures are still human-defined and inefficient. In this work, we propose a new search space that is suitable for structural re-parameterization techniques. RepNAS, a one-stage NAS approach, is present to efficiently search the optimal diverse branch block(ODBB) for each layer under the branch number constraint. Our experimental results show the searched ODBB can easily surpass the manual diverse branch block(DBB) with efficient training.