A Visual Technique to Analyze Flow of Information in a Machine Learning System
This provides a diagnostic tool for developers and researchers working on large-scale ML systems, though it is incremental as it adapts an existing visualization method to a new domain.
The paper tackles the problem of understanding complex information flow in machine learning systems by proposing a visual technique using Sankey diagrams, demonstrating its application to analyze training data, features, and classifier performance in an e-commerce product categorization task.
Machine learning (ML) algorithms and machine learning based software systems implicitly or explicitly involve complex flow of information between various entities such as training data, feature space, validation set and results. Understanding the statistical distribution of such information and how they flow from one entity to another influence the operation and correctness of such systems, especially in large-scale applications that perform classification or prediction in real time. In this paper, we propose a visual approach to understand and analyze flow of information during model training and serving phases. We build the visualizations using a technique called Sankey Diagram - conventionally used to understand data flow among sets - to address various use cases of in a machine learning system. We demonstrate how the proposed technique, tweaked and twisted to suit a classification problem, can play a critical role in better understanding of the training data, the features, and the classifier performance. We also discuss how this technique enables diagnostic analysis of model predictions and comparative analysis of predictions from multiple classifiers. The proposed concept is illustrated with the example of categorization of millions of products in the e-commerce domain - a multi-class hierarchical classification problem.