Analyzing Hypersensitive AI: Instability in Corporate-Scale Machine Learning
This addresses instability issues in corporate-scale machine learning, such as recommender systems, but appears incremental as it focuses on measurement and analysis rather than a new solution.
The paper tackles the problem of instability and variance in neural predictive algorithms on large datasets by presenting an approach to measure changes in geometric models in terms of output consistency and topological stability, using a word2vec-based recommender system as an example to analyze factors like data points and parameters, with findings aimed at stabilizing models and detecting data value differences at scale.
Predictive geometric models deliver excellent results for many Machine Learning use cases. Despite their undoubted performance, neural predictive algorithms can show unexpected degrees of instability and variance, particularly when applied to large datasets. We present an approach to measure changes in geometric models with respect to both output consistency and topological stability. Considering the example of a recommender system using word2vec, we analyze the influence of single data points, approximation methods and parameter settings. Our findings can help to stabilize models where needed and to detect differences in informational value of data points on a large scale.