The Forward-Forward Algorithm: Characterizing Training Behavior
This provides insights into the internal behavior of a novel learning algorithm, which could aid in developing more efficient or interpretable AI systems, though it is incremental as it focuses on characterization rather than new applications.
The paper investigates the training dynamics of Forward-Forward networks, an alternative to backpropagation, finding that deeper layers experience delayed accuracy improvement and shallower layers strongly correlate with overall model accuracy.
The Forward-Forward algorithm is an alternative learning method which consists of two forward passes rather than a forward and backward pass employed by backpropagation. Forward-Forward networks employ layer local loss functions which are optimized based on the layer activation for each forward pass rather than a single global objective function. This work explores the dynamics of model and layer accuracy changes in Forward-Forward networks as training progresses in pursuit of a mechanistic understanding of their internal behavior. Treatments to various system characteristics are applied to investigate changes in layer and overall model accuracy as training progresses, how accuracy is impacted by layer depth, and how strongly individual layer accuracy is correlated with overall model accuracy. The empirical results presented suggest that layers deeper within Forward-Forward networks experience a delay in accuracy improvement relative to shallower layers and that shallower layer accuracy is strongly correlated with overall model accuracy.