Impact of Inference Accelerators on hardware selection
This work addresses hardware selection challenges for AI deployment in healthcare, offering practical insights for cost-effective model inference, though it is incremental in nature.
The study tackled the problem of selecting optimal hardware for deploying AI models in healthcare by analyzing cost-performance trade-offs under domain-specific constraints, finding that CPU execution can be preferable to GPU accelerators in certain realistic scenarios.
As opportunities for AI-assisted healthcare grow steadily, model deployment faces challenges due to the specific characteristics of the industry. The configuration choice for a production device can impact model performance while influencing operational costs. Moreover, in healthcare some situations might require fast, but not real time, inference. We study different configurations and conduct a cost-performance analysis to determine the optimized hardware for the deployment of a model subject to healthcare domain constraints. We observe that a naive performance comparison may not lead to an optimal configuration selection. In fact, given realistic domain constraints, CPU execution might be preferable to GPU accelerators. Hence, defining beforehand precise expectations for model deployment is crucial.