Can CNNs Be More Robust Than Transformers?
This work addresses the robustness gap in image recognition for researchers and practitioners, showing it is incremental by refining CNN designs rather than introducing a new paradigm.
The paper challenges the belief that Transformers are inherently more robust than CNNs by identifying three simple architectural modifications—patchifying inputs, enlarging kernel size, and reducing activation/normalization layers—that enable CNNs to match or exceed Transformer robustness on out-of-distribution samples.
The recent success of Vision Transformers is shaking the long dominance of Convolutional Neural Networks (CNNs) in image recognition for a decade. Specifically, in terms of robustness on out-of-distribution samples, recent research finds that Transformers are inherently more robust than CNNs, regardless of different training setups. Moreover, it is believed that such superiority of Transformers should largely be credited to their self-attention-like architectures per se. In this paper, we question that belief by closely examining the design of Transformers. Our findings lead to three highly effective architecture designs for boosting robustness, yet simple enough to be implemented in several lines of code, namely a) patchifying input images, b) enlarging kernel size, and c) reducing activation layers and normalization layers. Bringing these components together, we are able to build pure CNN architectures without any attention-like operations that are as robust as, or even more robust than, Transformers. We hope this work can help the community better understand the design of robust neural architectures. The code is publicly available at https://github.com/UCSC-VLAA/RobustCNN.