Kronecker Attention Networks
This addresses a computational bottleneck for applying attention to high-dimensional data like images and videos, though it is an incremental improvement over existing attention methods.
The paper tackled the computational inefficiency and structural loss from flattening high-order data for attention mechanisms by proposing Kronecker attention operators that assume matrix-variate normal distributions, reducing computational resources by factors of hundreds while maintaining competitive performance.
Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by assuming the data follow matrix-variate normal distributions. Based on this new view, we develop Kronecker attention operators (KAOs) that operate on high-order tensor data directly. More importantly, the proposed KAOs lead to dramatic reductions in computational resources. Experimental results show that our methods reduce the amount of required computational resources by a factor of hundreds, with larger factors for higher-dimensional and higher-order data. Results also show that networks with KAOs outperform models without attention, while achieving competitive performance as those with original attention operators.