Human Attention during Goal-directed Reading Comprehension Relies on Task Optimization
This research addresses the computational principles of attention in complex tasks like reading comprehension, providing insights into human cognition and AI modeling, but it is incremental as it builds on existing DNN and eye-tracking methods.
The study tackled the problem of understanding attention allocation in goal-directed reading by showing that reading times are predicted by transformer-based DNNs optimized for the same task, with eye-tracking revealing separate attention to text features and question-relevant information, and DNNs optimized for word prediction predicting reading times in non-goal-directed scanning.
The computational principles underlying attention allocation in complex goal-directed tasks remain elusive. Goal-directed reading, i.e., reading a passage to answer a question in mind, is a common real-world task that strongly engages attention. Here, we investigate what computational models can explain attention distribution in this complex task. We show that the reading time on each word is predicted by the attention weights in transformer-based deep neural networks (DNNs) optimized to perform the same reading task. Eye-tracking further reveals that readers separately attend to basic text features and question-relevant information during first-pass reading and rereading, respectively. Similarly, text features and question relevance separately modulate attention weights in shallow and deep DNN layers. Furthermore, when readers scan a passage without a question in mind, their reading time is predicted by DNNs optimized for a word prediction task. Therefore, attention during real-world reading can be interpreted as the consequence of task optimization.