NasHD: Efficient ViT Architecture Performance Ranking using Hyperdimensional Computing
This work addresses the computational bottleneck in NAS for vision transformers, offering a faster alternative for researchers and practitioners, though it is incremental as it builds on existing hyperdimensional computing methods.
The paper tackles the high computational cost of Neural Architecture Search (NAS) for vision transformers by proposing NasHD, a hyperdimensional computing-based model that ranks architecture performance efficiently, achieving comparable results to sophisticated models while ranking nearly 100K models in about 1 minute on a benchmark dataset.
Neural Architecture Search (NAS) is an automated architecture engineering method for deep learning design automation, which serves as an alternative to the manual and error-prone process of model development, selection, evaluation and performance estimation. However, one major obstacle of NAS is the extremely demanding computation resource requirements and time-consuming iterations particularly when the dataset scales. In this paper, targeting at the emerging vision transformer (ViT), we present NasHD, a hyperdimensional computing based supervised learning model to rank the performance given the architectures and configurations. Different from other learning based methods, NasHD is faster thanks to the high parallel processing of HDC architecture. We also evaluated two HDC encoding schemes: Gram-based and Record-based of NasHD on their performance and efficiency. On the VIMER-UFO benchmark dataset of 8 applications from a diverse range of domains, NasHD Record can rank the performance of nearly 100K vision transformer models with about 1 minute while still achieving comparable results with sophisticated models.