BOWL: A Deceptively Simple Open World Learner
This addresses the challenge of making neural networks adaptable to real-world, dynamic environments for practical applications, though it appears incremental by leveraging existing batch normalization.
The paper tackles the problem of open world learning, where models must handle dynamic, uncurated inputs, by proposing that batch normalization layers can be used to detect in- and out-of-distribution samples, select data, and enable continuous updates, resulting in more robust models with less forgetting and efficient training.
Traditional machine learning excels on static benchmarks, but the real world is dynamic and seldom as carefully curated as test sets. Practical applications may generally encounter undesired inputs, are required to deal with novel information, and need to ensure operation through their full lifetime - aspects where standard deep models struggle. These three elements may have been researched individually, but their practical conjunction, i.e., open world learning, is much less consolidated. In this paper, we posit that neural networks already contain a powerful catalyst to turn them into open world learners: the batch normalization layer. Leveraging its tracked statistics, we derive effective strategies to detect in- and out-of-distribution samples, select informative data points, and update the model continuously. This, in turn, allows us to demonstrate that existing batch-normalized models can be made more robust, less prone to forgetting over time, and be trained efficiently with less data.