Diversity Networks: Neural Network Compression Using Determinantal Point Processes
This addresses the problem of reducing memory footprints in neural networks for AI practitioners, though it is incremental as it builds on existing compression techniques.
The paper tackles neural network compression by introducing Divnet, a method that uses Determinantal Point Processes to select diverse neurons and fuse redundant ones, resulting in smaller networks without performance loss, as shown by superior pruning results compared to competing approaches.
We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to select a subset of diverse neurons and subsequently fuses the redundant neurons into the selected ones. Compared with previous approaches, Divnet offers a more principled, flexible technique for capturing neuronal diversity and thus implicitly enforcing regularization. This enables effective auto-tuning of network architecture and leads to smaller network sizes without hurting performance. Moreover, through its focus on diversity and neuron fusing, Divnet remains compatible with other procedures that seek to reduce memory footprints of networks. We present experimental results to corroborate our claims: for pruning neural networks, Divnet is seen to be notably superior to competing approaches.