FlexServe: Deployment of PyTorch Models as Flexible REST Endpoints
This addresses a domain-specific need for operational environments with strict security requirements, offering an incremental improvement over existing tools like TensorFlow Serving.
The paper tackles the problem of deploying PyTorch models as REST endpoints without error-prone transformations, developing FlexServe to enable rapid deployments with flexible batching.
The integration of artificial intelligence capabilities into modern software systems is increasingly being simplified through the use of cloud-based machine learning services and representational state transfer architecture design. However, insufficient information regarding underlying model provenance and the lack of control over model evolution serve as an impediment to the more widespread adoption of these services in many operational environments which have strict security requirements. Furthermore, tools such as TensorFlow Serving allow models to be deployed as RESTful endpoints, but require error-prone transformations for PyTorch models as these dynamic computational graphs. This is in contrast to the static computational graphs of TensorFlow. To enable rapid deployments of PyTorch models without intermediate transformations we have developed FlexServe, a simple library to deploy multi-model ensembles with flexible batching.