RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations
This addresses robustness issues in Table QA for users relying on accurate data interpretation, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of robustness in Table QA models by introducing RobuT, a benchmark with human-annotated adversarial perturbations, and finds that state-of-the-art models falter under these conditions; it proposes using large language models to generate adversarial examples for training, which significantly improves robustness.
Despite significant progress having been made in question answering on tabular data (Table QA), it's unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called RobuT, which builds upon existing Table QA datasets (WTQ, WikiSQL-Weak, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models. Our data and code is publicly available at https://github.com/yilunzhao/RobuT.