Evaluating Visual Number Discrimination in Deep Neural Networks
This addresses the problem of understanding basic numerical competence in AI models for researchers in cognitive science and machine learning, but it is incremental as it builds on existing studies.
The study evaluated whether state-of-the-art deep neural networks for vision can discriminate between large and small quantities, finding that models with vision-specific inductive biases had the lowest test errors and psychometric curves resembling humans and animals, but failed in transfer experiments.
The ability to discriminate between large and small quantities is a core aspect of basic numerical competence in both humans and animals. In this work, we examine the extent to which the state-of-the-art neural networks designed for vision exhibit this basic ability. Motivated by studies in animal and infant numerical cognition, we use the numerical bisection procedure to test number discrimination in different families of neural architectures. Our results suggest that vision-specific inductive biases are helpful in numerosity discrimination, as models with such biases have lowest test errors on the task, and often have psychometric curves that qualitatively resemble those of humans and animals performing the task. However, even the strongest models, as measured on standard metrics of performance, fail to discriminate quantities in transfer experiments with differing training and testing conditions, indicating that such inductive biases might not be sufficient.