The Difficulty of Novelty Detection in Open-World Physical Domains: An Application to Angry Birds
This addresses the challenge of assessing AI systems' ability to handle novel situations in dynamic environments, though it is incremental as it focuses on a specific domain and validation method.
The paper tackles the problem of evaluating novelty detection difficulty in open-world physical domains by proposing a qualitative physics-based method, applied to Angry Birds, and finds that its calculated difficulties align with human user assessments.
Detecting and responding to novel situations in open-world environments is a key capability of human cognition and is a persistent problem for AI systems. In an open-world, novelties can appear in many different forms and may be easy or hard to detect. Therefore, to accurately evaluate the novelty detection capability of AI systems, it is necessary to investigate how difficult it may be to detect different types of novelty. In this paper, we propose a qualitative physics-based method to quantify the difficulty of novelty detection focusing on open-world physical domains. We apply our method in the popular physics simulation game Angry Birds, and conduct a user study across different novelties to validate our method. Results indicate that our calculated detection difficulties are in line with those of human users.