Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution
This addresses the need for trustworthy AI in cloud services by offering a reliable interpretation method for users and developers, though it is incremental as it builds on existing probing techniques but with guarantees.
The paper tackles the problem of interpreting predictions from AI models hidden behind APIs without access to internal parameters, proposing a closed-form solution named OpenAPI that provides exact and consistent interpretations for Piecewise Linear Models, with experiments demonstrating its effectiveness.
More and more AI services are provided through APIs on cloud where predictive models are hidden behind APIs. To build trust with users and reduce potential application risk, it is important to interpret how such predictive models hidden behind APIs make their decisions. The biggest challenge of interpreting such predictions is that no access to model parameters or training data is available. Existing works interpret the predictions of a model hidden behind an API by heuristically probing the response of the API with perturbed input instances. However, these methods do not provide any guarantee on the exactness and consistency of their interpretations. In this paper, we propose an elegant closed form solution named OpenAPI to compute exact and consistent interpretations for the family of Piecewise Linear Models (PLM), which includes many popular classification models. The major idea is to first construct a set of overdetermined linear equation systems with a small set of perturbed instances and the predictions made by the model on those instances. Then, we solve the equation systems to identify the decision features that are responsible for the prediction on an input instance. Our extensive experiments clearly demonstrate the exactness and consistency of our method.