Stable and Convexified Information Bottleneck Optimization via Symbolic Continuation and Entropy-Regularized Trajectories
This work addresses the instability issue in Information Bottleneck optimization for researchers and practitioners in representation learning, though it is incremental as it builds on existing IB methods.
The paper tackles the problem of unstable optimization in the Information Bottleneck method by introducing a novel approach using symbolic continuation and entropy-regularized trajectories, resulting in analytically proven convexity and uniqueness of the solution path with stable representation learning across a wide range of beta values, supported by extensive sensitivity analyses and 95% confidence intervals.
The Information Bottleneck (IB) method frequently suffers from unstable optimization, characterized by abrupt representation shifts near critical points of the IB trade-off parameter, beta. In this paper, I introduce a novel approach to achieve stable and convex IB optimization through symbolic continuation and entropy-regularized trajectories. I analytically prove convexity and uniqueness of the IB solution path when an entropy regularization term is included, and demonstrate how this stabilizes representation learning across a wide range of \b{eta} values. Additionally, I provide extensive sensitivity analyses around critical points (beta) with statistically robust uncertainty quantification (95% confidence intervals). The open-source implementation, experimental results, and reproducibility framework included in this work offer a clear path for practical deployment and future extension of my proposed method.