Information theoretic learning of robust deep representations
This work addresses the challenge of robustness in deep learning for data representation, though it appears incremental as it builds on existing information-theoretic concepts without demonstrating broad SOTA impact.
The authors tackled the problem of learning robust deep representations by proposing a novel information-theoretic objective function that maximizes mutual information between feature subsets and supervision, aiming to conserve information against noise or missing features, with preliminary experiments showing promising results.
We propose a novel objective function for learning robust deep representations of data based on information theory. Data is projected into a feature-vector space such that the mutual information of all subsets of features relative to the supervising signal is maximized. This objective function gives rise to robust representations by conserving available information relative to supervision in the face of noisy or unavailable features. Although the objective function is not directly tractable, we are able to derive a surrogate objective function. Minimizing this surrogate loss encourages features to be non-redundant and conditionally independent relative to the supervising signal. To evaluate the quality of obtained solutions, we have performed a set of preliminary experiments that show promising results.