A Survey of Learn-to-Compute Paradigms for Rate-Distortion-Type Problems
For researchers in information theory and machine learning, this survey provides a structured overview of neural methods for RD-type problems, but it is an incremental survey without new results.
This survey reviews neural approaches for computing rate-distortion-type objectives, including variational inference, neural mutual-information estimation, and dual-form optimization, showing they transform classical problems into scalable differentiable objectives, though challenges remain in large-scale applications.
Rate-distortion (RD) theory and its related formulations play a central role in understanding efficient information representation, but computing these quantities remains challenging in high-dimensional settings. Classical iterative methods such as the Blahut-Arimoto algorithm become impractical in high-dimensional domains due to the curse of dimensionality and the intractability of mutual-information terms. Recent advances in neural modeling and differentiable optimization offer a promising alternative through a learn-to-compute paradigm, in which probability distributions and objective functionals are represented by flexible neural parameterizations. This survey presents an overview of neural approaches for evaluating the RD-type objectives. We present three representative families of methods: variational inference, neural mutual-information estimation, and dual-form optimization. By reviewing their theoretical principles, algorithmic techniques, and consistency properties, we elucidate how these methods collectively transform classical RD-type problems into scalable differentiable objectives suitable for deep learning, though challenges remain in large-scale applications. Together, these perspectives offer promising avenues for scaling information-theoretic computation to complex, high-dimensional machine learning systems.