Implicit Hypothesis Testing and Divergence Preservation in Neural Network Representations
It provides a theoretical explanation for training dynamics in neural networks, potentially aiding in developing better training or regularization strategies, though it appears incremental.
The paper models neural network classification as binary hypothesis testing between class-conditional distributions, showing that well-generalizing networks align with optimal decision rules through monotonic improvements in KL divergence related to error rates.
We study the supervised training dynamics of neural classifiers through the lens of binary hypothesis testing. We model classification as a set of binary tests between class-conditional distributions of representations and empirically show that, along training trajectories, well-generalizing networks increasingly align with Neyman-Pearson optimal decision rules via monotonic improvements in KL divergence that relate to error rate exponents. We finally discuss how this yields an explanation and possible training or regularization strategies for different classes of neural networks.