ORXE: Orchestrating Experts for Dynamically Configurable Efficiency
This provides a scalable solution for efficient AI deployment in diverse real-world scenarios, though it is incremental as it builds on existing expert-based methods.
The paper tackles the problem of achieving real-time configurable efficiency in AI models by introducing ORXE, a framework that dynamically adjusts inference pathways using pre-trained experts, and it demonstrates superior performance compared to individual experts and other dynamic models in image classification tasks.
This paper presents ORXE, a modular and adaptable framework for achieving real-time configurable efficiency in AI models. By leveraging a collection of pre-trained experts with diverse computational costs and performance levels, ORXE dynamically adjusts inference pathways based on the complexity of input samples. Unlike conventional approaches that require complex metamodel training, ORXE achieves high efficiency and flexibility without complicating the development process. The proposed system utilizes a confidence-based gating mechanism to allocate appropriate computational resources for each input. ORXE also supports adjustments to the preference between inference cost and prediction performance across a wide range during runtime. We implemented a training-free ORXE system for image classification tasks, evaluating its efficiency and accuracy across various devices. The results demonstrate that ORXE achieves superior performance compared to individual experts and other dynamic models in most cases. This approach can be extended to other applications, providing a scalable solution for diverse real-world deployment scenarios.