AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation
This work addresses the need for efficient and scalable architectures for various AI tasks, though it appears incremental as it builds on hybrid approaches.
The authors tackled the problem of designing a versatile neural network architecture by introducing AsCAN, a hybrid model that asymmetrically combines convolutional and transformer blocks, achieving state-of-the-art performance in tasks like recognition, segmentation, and text-to-image generation with faster inference speeds than existing efficient models.
Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN -- a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, even without any computation optimization for transformer blocks, our models still yield faster inference speed than existing works featuring efficient attention mechanisms, highlighting the advantages and the value of our approach.