Memory Efficient Experience Replay for Streaming Learning
This addresses the challenge of enabling robots and agents to learn continuously from non-iid data streams without forgetting, though it is incremental as it builds on rehearsal methods.
The paper tackles the problem of catastrophic forgetting in streaming learning for deep neural networks by proposing the ExStream algorithm for memory-efficient rehearsal, finding that full rehearsal can eliminate forgetting and ExStream performs well with significantly reduced memory and computation.
In supervised machine learning, an agent is typically trained once and then deployed. While this works well for static settings, robots often operate in changing environments and must quickly learn new things from data streams. In this paradigm, known as streaming learning, a learner is trained online, in a single pass, from a data stream that cannot be assumed to be independent and identically distributed (iid). Streaming learning will cause conventional deep neural networks (DNNs) to fail for two reasons: 1) they need multiple passes through the entire dataset; and 2) non-iid data will cause catastrophic forgetting. An old fix to both of these issues is rehearsal. To learn a new example, rehearsal mixes it with previous examples, and then this mixture is used to update the DNN. Full rehearsal is slow and memory intensive because it stores all previously observed examples, and its effectiveness for preventing catastrophic forgetting has not been studied in modern DNNs. Here, we describe the ExStream algorithm for memory efficient rehearsal and compare it to alternatives. We find that full rehearsal can eliminate catastrophic forgetting in a variety of streaming learning settings, with ExStream performing well using far less memory and computation.