Frustrated with Replicating Claims of a Shared Model? A Solution
It addresses a practical issue for model owners and evaluators in ML/DL, though it is incremental as it builds on existing evaluation challenges.
The paper tackles the problem of replicating deep learning model evaluations due to lack of standard systems, proposing MLModelScope to specify and provision experiments, which facilitates rapid adoption of innovations.
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that model owners and evaluators are hard-pressed analyzing and studying them. This is exacerbated by the complicated procedures for evaluation. The lack of standard systems and efficient techniques for specifying and provisioning ML/DL evaluation is the main cause of this "pain point". This work discusses common pitfalls for replicating DL model evaluation, and shows that these subtle pitfalls can affect both accuracy and performance. It then proposes a solution to remedy these pitfalls called MLModelScope, a specification for repeatable model evaluation and a runtime to provision and measure experiments. We show that by easing the model specification and evaluation process, MLModelScope facilitates rapid adoption of ML/DL innovations.