A theory for the sparsity emerged in the Forward Forward algorithm
This provides a theoretical foundation for sparsity in a specific algorithm, which is incremental to existing work.
The paper tackles the problem of explaining the high sparsity phenomenon in the Forward-Forward algorithm by proposing two theorems that predict sparsity changes in activations, with experimental validation on the MNIST dataset.
This report explores the theory that explains the high sparsity phenomenon \citep{tosato2023emergent} observed in the forward-forward algorithm \citep{hinton2022forward}. The two theorems proposed predict the sparsity changes of a single data point's activation in two cases: Theorem \ref{theorem:1}: Decrease the goodness of the whole batch. Theorem \ref{theorem:2}: Apply the complete forward forward algorithm to decrease the goodness for negative data and increase the goodness for positive data. The theory aligns well with the experiments tested on the MNIST dataset.