Neural Networks Trained to Solve Differential Equations Learn General Representations
This work addresses the need for efficient generality measurement in neural networks for researchers in machine learning and scientific computing, though it is incremental as it builds on existing analysis methods.
The authors tackled the problem of measuring generality in neural network layers across continuously-parametrized tasks, using a technique based on singular vector canonical correlation analysis (SVCCA) applied to networks trained on Poisson partial differential equations, finding that first layers are general and deeper layers are more specific, with validation showing their method is much faster than existing techniques.
We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layer is general, and that deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. We find excellent agreement between the two methods, and note that our method is much faster, particularly for continuously-parametrized problems. Finally, we visualize the general representations of the first layers, and interpret them as generalized coordinates over the input domain.