Adversarial Structured Prediction for Multivariate Measures
This addresses the inconsistency problem in structured prediction for applications like NLP, offering a more direct optimization method, though it appears incremental as it builds on existing adversarial and structured prediction frameworks.
The paper tackles the mismatch between surrogate losses and exact multivariate performance measures like F-score and AER in structured prediction by introducing an adversarial approach that optimizes these exact measures while approximating training data. It demonstrates this method on word alignment and named entity recognition tasks, showing competitive results.
Many predicted structured objects (e.g., sequences, matchings, trees) are evaluated using the F-score, alignment error rate (AER), or other multivariate performance measures. Since inductively optimizing these measures using training data is typically computationally difficult, empirical risk minimization of surrogate losses is employed, using, e.g., the hinge loss for (structured) support vector machines. These approximations often introduce a mismatch between the learner's objective and the desired application performance, leading to inconsistency. We take a different approach: adversarially approximate training data while optimizing the exact F-score or AER. Structured predictions under this formulation result from solving zero-sum games between a predictor seeking the best performance and an adversary seeking the worst while required to (approximately) match certain structured properties of the training data. We explore this approach for word alignment (AER evaluation) and named entity recognition (F-score evaluation) with linear-chain constraints.