Reading Comprehension Ability Test-A Turing Test for Reading Comprehension
This work addresses the need for standardized evaluation of AI reading comprehension, which is incremental as it adapts the Turing test concept to a specific domain.
The authors tackled the problem of evaluating machine reading comprehension by proposing the Comprehension Ability Test (CAT), a multi-level test similar to the Turing test that assesses abilities from fact identification to understanding intent and sentiment, with the result being a framework for comparing human and machine comprehension without specifying concrete numbers.
Reading comprehension is an important ability of human intelligence. Literacy and numeracy are two most essential foundation for people to succeed at study, at work and in life. Reading comprehension ability is a core component of literacy. In most of the education systems, developing reading comprehension ability is compulsory in the curriculum from year one to year 12. It is an indispensable ability in the dissemination of knowledge. With the emerging artificial intelligence, computers start to be able to read and understand like people in some context. They can even read better than human beings for some tasks, but have little clue in other tasks. It will be very beneficial if we can identify the levels of machine comprehension ability, which will direct us on the further improvement. Turing test is a well-known test of the difference between computer intelligence and human intelligence. In order to be able to compare the difference between people reading and machines reading, we proposed a test called (reading) Comprehension Ability Test (CAT).CAT is similar to Turing test, passing of which means we cannot differentiate people from algorithms in term of their comprehension ability. CAT has multiple levels showing the different abilities in reading comprehension, from identifying basic facts, performing inference, to understanding the intent and sentiment.