Adaptive Convolutions with Per-pixel Dynamic Filter Atom
This work addresses a bottleneck in dynamic convolutions for computer vision tasks, offering a plug-and-play solution that is translation-equivariant and parametrically efficient, though it is incremental as it builds on existing dynamic convolution methods.
The paper tackles the challenge of implementing scalable and versatile dynamic convolutions with per-pixel adapted filters by decomposing filters over dynamic atoms generated from local features, achieving comparable or better performance across tasks, particularly in handling significant intra-image variance.
Applying feature dependent network weights have been proved to be effective in many fields. However, in practice, restricted by the enormous size of model parameters and memory footprints, scalable and versatile dynamic convolutions with per-pixel adapted filters are yet to be fully explored. In this paper, we address this challenge by decomposing filters, adapted to each spatial position, over dynamic filter atoms generated by a light-weight network from local features. Adaptive receptive fields can be supported by further representing each filter atom over sets of pre-fixed multi-scale bases. As plug-and-play replacements to convolutional layers, the introduced adaptive convolutions with per-pixel dynamic atoms enable explicit modeling of intra-image variance, while avoiding heavy computation, parameters, and memory cost. Our method preserves the appealing properties of conventional convolutions as being translation-equivariant and parametrically efficient. We present experiments to show that, the proposed method delivers comparable or even better performance across tasks, and are particularly effective on handling tasks with significant intra-image variance.