Discovering Common Information in Multi-view Data
This work addresses the challenge of multi-view data analysis for researchers and practitioners in machine learning, offering a novel approach with theoretical guarantees, though it is incremental in building on existing information theory concepts.
The paper tackles the problem of extracting common and unique information from multi-view data by introducing a mathematically rigorous definition and a supervised learning framework that minimizes total correlation to ensure independence, achieving superior performance on synthetic and seven benchmark datasets compared to state-of-the-art methods.
We introduce an innovative and mathematically rigorous definition for computing common information from multi-view data, drawing inspiration from Gács-Körner common information in information theory. Leveraging this definition, we develop a novel supervised multi-view learning framework to capture both common and unique information. By explicitly minimizing a total correlation term, the extracted common information and the unique information from each view are forced to be independent of each other, which, in turn, theoretically guarantees the effectiveness of our framework. To estimate information-theoretic quantities, our framework employs matrix-based R{é}nyi's $α$-order entropy functional, which forgoes the need for variational approximation and distributional estimation in high-dimensional space. Theoretical proof is provided that our framework can faithfully discover both common and unique information from multi-view data. Experiments on synthetic and seven benchmark real-world datasets demonstrate the superior performance of our proposed framework over state-of-the-art approaches.