Structured IB: Improving Information Bottleneck with Structured Feature Learning
This work addresses a problem for researchers and practitioners in machine learning by improving the Information Bottleneck principle, though it appears incremental as it builds on existing IB frameworks.
The paper tackled the limitation of Information Bottleneck (IB) methods, which depend on design quality and are error-prone, by introducing Structured IB to extract more informative features, resulting in superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size.
The Information Bottleneck (IB) principle has emerged as a promising approach for enhancing the generalization, robustness, and interpretability of deep neural networks, demonstrating efficacy across image segmentation, document clustering, and semantic communication. Among IB implementations, the IB Lagrangian method, employing Lagrangian multipliers, is widely adopted. While numerous methods for the optimizations of IB Lagrangian based on variational bounds and neural estimators are feasible, their performance is highly dependent on the quality of their design, which is inherently prone to errors. To address this limitation, we introduce Structured IB, a framework for investigating potential structured features. By incorporating auxiliary encoders to extract missing informative features, we generate more informative representations. Our experiments demonstrate superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size.