Ray-Tracing for Conditionally Activated Neural Networks
This work addresses efficiency in neural network inference for AI applications, though it appears incremental as it builds on existing Mixture of Experts methods.
The paper tackles the problem of reducing parameter count in neural networks by introducing a conditionally activated architecture that dynamically unfolds based on input complexity, achieving competitive accuracy with significantly fewer parameters for inference.
In this paper, we introduce a novel architecture for conditionally activated neural networks combining a hierarchical construction of multiple Mixture of Experts (MoEs) layers with a sampling mechanism that progressively converges to an optimized configuration of expert activation. This methodology enables the dynamic unfolding of the network's architecture, facilitating efficient path-specific training. Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines while significantly reducing the parameter count required for inference. Notably, this parameter reduction correlates with the complexity of the input patterns, a property naturally emerging from the network's operational dynamics without necessitating explicit auxiliary penalty functions.