An overview of condensation phenomenon in deep learning
It provides insights into neural network training dynamics for researchers, but is incremental as it overviews an existing phenomenon.
The paper investigates the condensation phenomenon in neural networks, where neurons in the same layer group into similar outputs during training, and finds that this process is influenced by factors like small weight initializations and Dropout, correlating with improved generalization and reasoning abilities in models.
In this paper, we provide an overview of a common phenomenon, condensation, observed during the nonlinear training of neural networks: During the nonlinear training of neural networks, neurons in the same layer tend to condense into groups with similar outputs. Empirical observations suggest that the number of condensed clusters of neurons in the same layer typically increases monotonically as training progresses. Neural networks with small weight initializations or Dropout optimization can facilitate this condensation process. We also examine the underlying mechanisms of condensation from the perspectives of training dynamics and the structure of the loss landscape. The condensation phenomenon offers valuable insights into the generalization abilities of neural networks and correlates to stronger reasoning abilities in transformer-based language models.