Efficient Neural Networks with Discrete Cosine Transform Activations
This work addresses the need for compact and interpretable neural networks in machine learning, offering incremental improvements over previous ENN methods.
The paper tackles the problem of designing efficient and interpretable neural networks by proposing an extension of Expressive Neural Networks (ENNs) with Discrete Cosine Transform (DCT) activations, achieving state-of-the-art accuracy with low parameters and enabling pruning of up to 40% of activation coefficients without performance loss.
In this paper, we extend our previous work on the Expressive Neural Network (ENN), a multilayer perceptron with adaptive activation functions parametrized using the Discrete Cosine Transform (DCT). Building upon previous work that demonstrated the strong expressiveness of ENNs with compact architectures, we now emphasize their efficiency, interpretability and pruning capabilities. The DCT-based parameterization provides a structured and decorrelated representation that reveals the functional role of each neuron and allows direct identification of redundant components. Leveraging this property, we propose an efficient pruning strategy that removes unnecessary DCT coefficients with negligible or no loss in performance. Experimental results across classification and implicit neural representation tasks confirm that ENNs achieve state-of-the-art accuracy while maintaining a low number of parameters. Furthermore, up to 40% of the activation coefficients can be safely pruned, thanks to the orthogonality and bounded nature of the DCT basis. Overall, these findings demonstrate that the ENN framework offers a principled integration of signal processing concepts into neural network design, achieving a balanced trade-off between expressiveness, compactness, and interpretability.