Finet: Using Fine-grained Batch Normalization to Train Light-weight Neural Networks
This addresses the need for more efficient inference in resource-constrained environments, offering a novel method for light-weight network design.
The paper tackles the problem of building efficient light-weight neural networks by proposing Fine-grained Batch Normalization (FBN), which normalizes intermediate states instead of final summations, resulting in state-of-the-art performance on ImageNet with 65.706% accuracy at 43M FLOPs and 73.786% accuracy at 303M FLOPs.
To build light-weight network, we propose a new normalization, Fine-grained Batch Normalization (FBN). Different from Batch Normalization (BN), which normalizes the final summation of the weighted inputs, FBN normalizes the intermediate state of the summation. We propose a novel light-weight network based on FBN, called Finet. At training time, the convolutional layer with FBN can be seen as an inverted bottleneck mechanism. FBN can be fused into convolution at inference time. After fusion, Finet uses the standard convolution with equal channel width, thus makes the inference more efficient. On ImageNet classification dataset, Finet achieves the state-of-art performance (65.706% accuracy with 43M FLOPs, and 73.786% accuracy with 303M FLOPs), Moreover, experiments show that Finet is more efficient than other state-of-art light-weight networks.