Learning Affinity-Aware Upsampling for Deep Image Matting
This work offers a more efficient way to incorporate pairwise interactions in deep networks for detail-sensitive tasks like image matting, benefiting researchers and practitioners seeking compact and high-performing models.
This paper introduces Affinity-Aware Upsampling (A2U), a method that learns pairwise interactions directly within the upsampling process using second-order features. On the Composition-1k matting dataset, A2U achieved a 14% relative improvement in the SAD metric over a strong baseline with less than 0.5% parameter increase, and an 8% higher performance with only 40% model complexity compared to state-of-the-art matting networks.
We show that learning affinity in upsampling provides an effective and efficient approach to exploit pairwise interactions in deep networks. Second-order features are commonly used in dense prediction to build adjacent relations with a learnable module after upsampling such as non-local blocks. Since upsampling is essential, learning affinity in upsampling can avoid additional propagation layers, offering the potential for building compact models. By looking at existing upsampling operators from a unified mathematical perspective, we generalize them into a second-order form and introduce Affinity-Aware Upsampling (A2U) where upsampling kernels are generated using a light-weight lowrank bilinear model and are conditioned on second-order features. Our upsampling operator can also be extended to downsampling. We discuss alternative implementations of A2U and verify their effectiveness on two detail-sensitive tasks: image reconstruction on a toy dataset; and a largescale image matting task where affinity-based ideas constitute mainstream matting approaches. In particular, results on the Composition-1k matting dataset show that A2U achieves a 14% relative improvement in the SAD metric against a strong baseline with negligible increase of parameters (<0.5%). Compared with the state-of-the-art matting network, we achieve 8% higher performance with only 40% model complexity.