Generalization of CNNs on Relational Reasoning with Bar Charts
This study addresses the problem of robust relational reasoning in visualizations for AI researchers, highlighting incremental insights into CNN limitations compared to human performance.
The paper investigates how convolutional neural networks (CNNs) generalize on relational reasoning tasks with bar charts, finding that CNNs outperform humans only when training and test data share the same visual encodings, but are sensitive to irrelevant visual perturbations, whereas humans focus mainly on bar lengths.
This paper presents a systematic study of the generalization of convolutional neural networks (CNNs) and humans on relational reasoning tasks with bar charts. We first revisit previous experiments on graphical perception and update the benchmark performance of CNNs. We then test the generalization performance of CNNs on a classic relational reasoning task: estimating bar length ratios in a bar chart, by progressively perturbing the standard visualizations. We further conduct a user study to compare the performance of CNNs and humans. Our results show that CNNs outperform humans only when the training and test data have the same visual encodings. Otherwise, they may perform worse. We also find that CNNs are sensitive to perturbations in various visual encodings, regardless of their relevance to the target bars. Yet, humans are mainly influenced by bar lengths. Our study suggests that robust relational reasoning with visualizations is challenging for CNNs. Improving CNNs' generalization performance may require training them to better recognize task-related visual properties.