Grammar Based Directed Testing of Machine Learning Systems
This addresses the need for systematic and scalable validation methods for machine-learning systems, particularly in NLP, though it appears incremental as it builds on existing robustness properties.
The paper tackles the problem of validating machine-learning systems by introducing OGMA, a grammar-based directed testing approach that automatically discovers erroneous behaviors in classifiers and uses them to improve models, finding thousands of errors in three NLP classifiers and showing up to 489% greater effectiveness than random testing.
The massive progress of machine learning has seen its application over a variety of domains in the past decade. But how do we develop a systematic, scalable and modular strategy to validate machine-learning systems? We present, to the best of our knowledge, the first approach, which provides a systematic test framework for machine-learning systems that accepts grammar-based inputs. Our OGMA approach automatically discovers erroneous behaviours in classifiers and leverages these erroneous behaviours to improve the respective models. OGMA leverages inherent robustness properties present in any well trained machine-learning model to direct test generation and thus, implementing a scalable test generation methodology. To evaluate our OGMA approach, we have tested it on three real world natural language processing (NLP) classifiers. We have found thousands of erroneous behaviours in these systems. We also compare OGMA with a random test generation approach and observe that OGMA is more effective than such random test generation by up to 489%.