ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
This work addresses accessibility by enabling reasoning over statistical charts like bar and pie charts, but it is incremental as it builds on existing MAC-Networks with modifications.
The authors tackled the problem of visual reasoning over statistical charts by formulating it as a classification task using MAC-Networks, with ChartNet outperforming state-of-the-art methods on a custom dataset for both in-vocabulary and out-of-vocabulary answers.
Despite the improvements in perception accuracies brought about via deep learning, developing systems combining accurate visual perception with the ability to reason over the visual percepts remains extremely challenging. A particular application area of interest from an accessibility perspective is that of reasoning over statistical charts such as bar and pie charts. To this end, we formulate the problem of reasoning over statistical charts as a classification task using MAC-Networks to give answers from a predefined vocabulary of generic answers. Additionally, we enhance the capabilities of MAC-Networks to give chart-specific answers to open-ended questions by replacing the classification layer by a regression layer to localize the textual answers present over the images. We call our network ChartNet, and demonstrate its efficacy on predicting both in vocabulary and out of vocabulary answers. To test our methods, we generated our own dataset of statistical chart images and corresponding question answer pairs. Results show that ChartNet consistently outperform other state-of-the-art methods on reasoning over these questions and may be a viable candidate for applications containing images of statistical charts.