WeightNet: Revisiting the Design Space of Weight Networks
This work addresses the need for efficient and flexible weight generation in neural networks, offering incremental improvements over existing methods.
The authors tackled the problem of designing weight generating networks by introducing WeightNet, a framework that unifies SENet and CondConv, achieving better Accuracy-FLOPs and Accuracy-Parameter trade-offs on ImageNet and COCO detection tasks.
We present a conceptually simple, flexible and effective framework for weight generating networks. Our approach is general that unifies two current distinct and extremely effective SENet and CondConv into the same framework on weight space. The method, called WeightNet, generalizes the two methods by simply adding one more grouped fully-connected layer to the attention activation layer. We use the WeightNet, composed entirely of (grouped) fully-connected layers, to directly output the convolutional weight. WeightNet is easy and memory-conserving to train, on the kernel space instead of the feature space. Because of the flexibility, our method outperforms existing approaches on both ImageNet and COCO detection tasks, achieving better Accuracy-FLOPs and Accuracy-Parameter trade-offs. The framework on the flexible weight space has the potential to further improve the performance. Code is available at https://github.com/megvii-model/WeightNet.