A Gated Peripheral-Foveal Convolutional Neural Network for Unified Image Aesthetic Prediction
This work addresses image aesthetic prediction for applications like photography and design, but it is incremental as it builds on existing neural network approaches with a biologically inspired twist.
The paper tackled the problem of learning fine-grained details in image aesthetic assessment by proposing a Gated Peripheral-Foveal Convolutional Neural Network (GPF-CNN), which mimics human visual perception with peripheral and foveal subnets and a gated fusion mechanism, achieving effectiveness in aesthetic quality classification, score regression, and distribution prediction on AVA and Photo.net datasets.
Learning fine-grained details is a key issue in image aesthetic assessment. Most of the previous methods extract the fine-grained details via random cropping strategy, which may undermine the integrity of semantic information. Extensive studies show that humans perceive fine-grained details with a mixture of foveal vision and peripheral vision. Fovea has the highest possible visual acuity and is responsible for seeing the details. The peripheral vision is used for perceiving the broad spatial scene and selecting the attended regions for the fovea. Inspired by these observations, we propose a Gated Peripheral-Foveal Convolutional Neural Network (GPF-CNN). It is a dedicated double-subnet neural network, i.e. a peripheral subnet and a foveal subnet. The former aims to mimic the functions of peripheral vision to encode the holistic information and provide the attended regions. The latter aims to extract fine-grained features on these key regions. Considering that the peripheral vision and foveal vision play different roles in processing different visual stimuli, we further employ a gated information fusion (GIF) network to weight their contributions. The weights are determined through the fully connected layers followed by a sigmoid function. We conduct comprehensive experiments on the standard AVA and Photo.net datasets for unified aesthetic prediction tasks: (i) aesthetic quality classification; (ii) aesthetic score regression; and (iii) aesthetic score distribution prediction. The experimental results demonstrate the effectiveness of the proposed method.