Systematic Architectural Design of Scale Transformed Attention Condenser DNNs via Multi-Scale Class Representational Response Similarity Analysis
This work addresses efficiency and performance tuning in deep learning for computer vision, offering a systematic method to reduce hyperparameter search, though it is incremental as it builds on existing attention mechanisms.
The authors tackled the problem of hyperparameter tuning in self-attention convolutional neural networks by proposing Multi-Scale Class Representational Response Similarity Analysis (ClassRepSim) to guide architectural design, resulting in the Spatial Transformed Attention Condenser (STAC) module that increased top-1 accuracy by up to 1.6% on ImageNet64x64 compared to ResNet models.
Self-attention mechanisms are commonly included in a convolutional neural networks to achieve an improved efficiency performance balance. However, adding self-attention mechanisms adds additional hyperparameters to tune for the application at hand. In this work we propose a novel type of DNN analysis called Multi-Scale Class Representational Response Similarity Analysis (ClassRepSim) which can be used to identify specific design interventions that lead to more efficient self-attention convolutional neural network architectures. Using insights grained from ClassRepSim we propose the Spatial Transformed Attention Condenser (STAC) module, a novel attention-condenser based self-attention module. We show that adding STAC modules to ResNet style architectures can result in up to a 1.6% increase in top-1 accuracy compared to vanilla ResNet models and up to a 0.5% increase in top-1 accuracy compared to SENet models on the ImageNet64x64 dataset, at the cost of up to 1.7% increase in FLOPs and 2x the number of parameters. In addition, we demonstrate that results from ClassRepSim analysis can be used to select an effective parameterization of the STAC module resulting in competitive performance compared to an extensive parameter search.