Learning Representations that Support Extrapolation
This addresses the limitation of neural networks in making inferences beyond interpolation, which is incremental but important for advancing AI towards human-like intelligence.
The paper tackled the problem of enabling neural networks to extrapolate beyond training data by introducing a visual analogy benchmark for graded evaluation and a technique called temporal context normalization. The result was a significant improvement in extrapolation ability, outperforming competitive methods.
Extrapolation -- the ability to make inferences that go beyond the scope of one's experiences -- is a hallmark of human intelligence. By contrast, the generalization exhibited by contemporary neural network algorithms is largely limited to interpolation between data points in their training corpora. In this paper, we consider the challenge of learning representations that support extrapolation. We introduce a novel visual analogy benchmark that allows the graded evaluation of extrapolation as a function of distance from the convex domain defined by the training data. We also introduce a simple technique, temporal context normalization, that encourages representations that emphasize the relations between objects. We find that this technique enables a significant improvement in the ability to extrapolate, considerably outperforming a number of competitive techniques.