Overcoming Statistical Shortcuts for Open-ended Visual Counting
This addresses the issue of spurious correlations in visual counting tasks, which is important for improving model robustness in real-world applications, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of machine learning models over-relying on statistical shortcuts in open-ended visual counting by proposing a protocol to penalize such shortcuts and a novel network for visual analysis, achieving superior performance on the TallyQA dataset compared to state-of-the-art models.
Machine learning models tend to over-rely on statistical shortcuts. These spurious correlations between parts of the input and the output labels does not hold in real-world settings. We target this issue on the recent open-ended visual counting task which is well suited to study statistical shortcuts. We aim to develop models that learn a proper mechanism of counting regardless of the output label. First, we propose the Modifying Count Distribution (MCD) protocol, which penalizes models that over-rely on statistical shortcuts. It is based on pairs of training and testing sets that do not follow the same count label distribution such as the odd-even sets. Intuitively, models that have learned a proper mechanism of counting on odd numbers should perform well on even numbers. Secondly, we introduce the Spatial Counting Network (SCN), which is dedicated to visual analysis and counting based on natural language questions. Our model selects relevant image regions, scores them with fusion and self-attention mechanisms, and provides a final counting score. We apply our protocol on the recent dataset, TallyQA, and show superior performances compared to state-of-the-art models. We also demonstrate the ability of our model to select the correct instances to count in the image. Code and datasets are available: https://github.com/cdancette/spatial-counting-network