Encouraging an Appropriate Representation Simplifies Training of Neural Networks
This work highlights a potential limitation in neural network training for AI practitioners, suggesting that integrating domain knowledge via representation guidance could improve generalization, though it is incremental in scope.
The paper challenges the assumption that neural networks automatically learn appropriate internal representations, showing that state-of-the-art training fails on two simple tasks despite the model's capacity, and that encouraging specific representations enables solving them.
A common assumption about neural networks is that they can learn an appropriate internal representations on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.