Diffusion Models are Kelly Gamblers
This work provides theoretical insights into diffusion models, linking them to information theory and quantum mechanics, which could help improve model design and understanding, though it appears incremental as it builds on existing concepts.
The paper connects diffusion models to the Kelly criterion in betting games, showing that conditional diffusion models store mutual information between signal and conditioning, and that classifier-free guidance boosts this mutual information at sampling time, especially beneficial for image models due to low mutual information between images and labels.
We draw a connection between diffusion models and the Kelly criterion for maximizing returns in betting games. We find that conditional diffusion models store additional information to bind the signal $X$ with the conditioning information $Y$, equal to the mutual information between them. Classifier-free guidance effectively boosts the mutual information between $X$ and $Y$ at sampling time. This is especially helpful in image models, since the mutual information between images and their labels is low, a fact which is intimately connected to the manifold hypothesis. Finally, we point out some nuances in the popular perspective that diffusion models are infinitely deep autoencoders. In doing so, we relate the denoising loss to the Fermi Golden Rule from quantum mechanics.