What's a Good Prediction? Challenges in evaluating an agent's knowledge
This addresses a foundational challenge in AI for researchers and practitioners by highlighting limitations in current evaluation methods and suggesting a novel direction, though it is incremental as it builds on existing GVF frameworks.
The paper tackles the problem of evaluating an agent's knowledge by showing that accuracy metrics can conflict with usefulness, demonstrated through thought experiments and a Minecraft example using General Value Functions (GVF). It proposes an alternative evaluation approach based on examining internal learning processes, such as feature relevance, in online continual learning settings.
Constructing general knowledge by learning task-independent models of the world can help agents solve challenging problems. However, both constructing and evaluating such models remains an open challenge. The most common approaches to evaluating models is to assess their accuracy with respect to observable values. However, the prevailing reliance on estimator accuracy as a proxy for the usefulness of the knowledge has the potential to lead us astray. We demonstrate the conflict between accuracy and usefulness through a series of illustrative examples including both a thought experiment and empirical example in MineCraft, using the General Value Function framework (GVF). Having identified challenges in assessing an agent's knowledge, we propose an alternate evaluation approach that arises continually in the online continual learning setting we recommend evaluation by examining internal learning processes, specifically the relevance of a GVF's features to the prediction task at hand. This paper contributes a first look into evaluation of predictions through their use, an integral component of predictive knowledge which is as of yet unexplored.