Deep spatial context: when attention-based models meet spatial regression
This work addresses the need for quantitative analysis of spatial context in vision models, particularly for applications like medical imaging, but it is incremental as it builds on existing attention and spatial regression techniques.
The study tackled the problem of quantifying spatial context in attention-based vision models by proposing the Deep Spatial Context (DSCon) method, which uses spatial regression and three measures to analyze spatial relationships, revealing that spatial context is more significant in tumor lesion classification than in normal tissues and is largest in feature space.
We propose 'Deep spatial context' (DSCon) method, which serves for investigation of the attention-based vision models using the concept of spatial context. It was inspired by histopathologists, however, the method can be applied to various domains. The DSCon allows for a quantitative measure of the spatial context's role using three Spatial Context Measures: $SCM_{features}$, $SCM_{targets}$, $SCM_{residuals}$ to distinguish whether the spatial context is observable within the features of neighboring regions, their target values (attention scores) or residuals, respectively. It is achieved by integrating spatial regression into the pipeline. The DSCon helps to verify research questions. The experiments reveal that spatial relationships are much bigger in the case of the classification of tumor lesions than normal tissues. Moreover, it turns out that the larger the size of the neighborhood taken into account within spatial regression, the less valuable contextual information is. Furthermore, it is observed that the spatial context measure is the largest when considered within the feature space as opposed to the targets and residuals.