CommAI: Evaluating the first steps towards a useful general AI
This addresses the lack of standardized evaluation methods for general AI, which is a foundational issue for the AI research community.
The paper tackles the problem of measuring progress towards general AI by proposing a set of desiderata and a testing platform, aiming to provide objective benchmarks for evaluating broad machine intelligence.
With machine learning successfully applied to new daunting problems almost every day, general AI starts looking like an attainable goal. However, most current research focuses instead on important but narrow applications, such as image classification or machine translation. We believe this to be largely due to the lack of objective ways to measure progress towards broad machine intelligence. In order to fill this gap, we propose here a set of concrete desiderata for general AI, together with a platform to test machines on how well they satisfy such desiderata, while keeping all further complexities to a minimum.