Deep Information Networks
This work proposes a novel classifier architecture that could benefit machine learning practitioners by improving modularity and reducing complexity, though it appears incremental as it builds on existing information bottleneck concepts.
The authors tackled the problem of designing a modular and less complex classifier by introducing a tree-structured Information Network based on information theory, which achieved good accuracy results.
We describe a novel classifier with a tree structure, designed using information theory concepts. This Information Network is made of information nodes, that compress the input data, and multiplexers, that connect two or more input nodes to an output node. Each information node is trained, independently of the others, to minimize a local cost function that minimizes the mutual information between its input and output with the constraint of keeping a given mutual information between its output and the target (information bottleneck). We show that the system is able to provide good results in terms of accuracy, while it shows many advantages in terms of modularity and reduced complexity.