Deformably-Scaled Transposed Convolution
This work addresses a bottleneck in upsampling for computer vision tasks, offering a broadly applicable improvement for researchers and practitioners in fields like medical imaging and image synthesis.
The authors tackled the under-explored problem of transposed convolution for high-resolution output generation by introducing a novel layer that selectively places information and controls synthesis stroke breadth with minimal parameter overhead. The result is a drop-in replacement layer that substantially improves performance across diverse tasks including instance segmentation, object detection, and generative image modeling, outperforming all existing transposed convolution variants.
Transposed convolution is crucial for generating high-resolution outputs, yet has received little attention compared to convolution layers. In this work we revisit transposed convolution and introduce a novel layer that allows us to place information in the image selectively and choose the `stroke breadth' at which the image is synthesized, whilst incurring a small additional parameter cost. For this we introduce three ideas: firstly, we regress offsets to the positions where the transpose convolution results are placed; secondly we broadcast the offset weight locations over a learnable neighborhood; and thirdly we use a compact parametrization to share weights and restrict offsets. We show that simply substituting upsampling operators with our novel layer produces substantial improvements across tasks as diverse as instance segmentation, object detection, semantic segmentation, generative image modeling, and 3D magnetic resonance image enhancement, while outperforming all existing variants of transposed convolutions. Our novel layer can be used as a drop-in replacement for 2D and 3D upsampling operators and the code will be publicly available.