Synthetic Ground Truth Generation for Evaluating Generative Policy Models
This addresses the need for realistic evaluation scenarios for GPMs in domains like military or autonomous systems, though it is incremental as it builds on existing frameworks for data generation.
The paper tackles the problem of evaluating Generative Policy-based Models (GPMs) in complex environments by presenting a method using an agile knowledge representation framework to generate synthetic ground truth data, and it releases conceptual models and datasets to support this evaluation.
Generative Policy-based Models aim to enable a coalition of systems, be they devices or services to adapt according to contextual changes such as environmental factors, user preferences and different tasks whilst adhering to various constraints and regulations as directed by a managing party or the collective vision of the coalition. Recent developments have proposed new architectures to realize the potential of GPMs but as the complexity of systems and their associated requirements increases, there is an emerging requirement to have scenarios and associated datasets to realistically evaluate GPMs with respect to the properties of the operating environment, be it the future battlespace or an autonomous organization. In order to address this requirement, in this paper, we present a method of applying an agile knowledge representation framework to model requirements, both individualistic and collective that enables synthetic generation of ground truth data such that advanced GPMs can be evaluated robustly in complex environments. We also release conceptual models, annotated datasets, as well as means to extend the data generation approach so that similar datasets can be developed for varying complexities and different situations.