Conditional Information Gain Networks
This work addresses the challenge of deploying deep models on mobile devices by introducing a method to reduce parameters and computation, though it is incremental as it builds on existing conditional computing approaches.
The paper tackles the problem of reducing computational requirements in deep neural networks for resource-limited environments by proposing Conditional Information Gain Networks, which enable conditional execution to skip parts of the model based on sample-specific decisions, achieving better or comparable classification results on MNIST and Fashion MNIST datasets using significantly fewer parameters.
Deep neural network models owe their representational power to the high number of learnable parameters. It is often infeasible to run these largely parametrized deep models in limited resource environments, like mobile phones. Network models employing conditional computing are able to reduce computational requirements while achieving high representational power, with their ability to model hierarchies. We propose Conditional Information Gain Networks, which allow the feed forward deep neural networks to execute conditionally, skipping parts of the model based on the sample and the decision mechanisms inserted in the architecture. These decision mechanisms are trained using cost functions based on differentiable Information Gain, inspired by the training procedures of decision trees. These information gain based decision mechanisms are differentiable and can be trained end-to-end using a unified framework with a general cost function, covering both classification and decision losses. We test the effectiveness of the proposed method on MNIST and recently introduced Fashion MNIST datasets and show that our information gain based conditional execution approach can achieve better or comparable classification results using significantly fewer parameters, compared to standard convolutional neural network baselines.