Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields
This work addresses the challenge of tuning locality in visual tasks for computer vision applications, representing an incremental advancement in network design.
The authors tackled the problem of optimizing and adapting receptive field size and shape in deep networks for semantic segmentation, achieving accuracy improvements of 1-2 points on Cityscapes for strong architectures and up to 10 points for suboptimal designs, with dynamic adaptation matching free-form deformation accuracy while being more efficient.
The visual world is vast and varied, but its variations divide into structured and unstructured factors. We compose free-form filters and structured Gaussian filters, optimized end-to-end, to factorize deep representations and learn both local features and their degree of locality. Our semi-structured composition is strictly more expressive than free-form filtering, and changes in its structured parameters would require changes in free-form architecture. In effect this optimizes over receptive field size and shape, tuning locality to the data and task. Dynamic inference, in which the Gaussian structure varies with the input, adapts receptive field size to compensate for local scale variation. Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs. Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.