TARGA: Targeted Synthetic Data Generation for Practical Reasoning over Structured Data
This addresses the problem of limited generalization and high annotation costs in knowledge base question answering for AI researchers and practitioners, offering a practical incremental improvement.
The paper tackles the challenges of semantic parsing for reasoning over structured data by proposing TARGA, a framework that generates targeted synthetic data without manual annotation, resulting in substantial F1 score improvements on KBQA datasets, such as +7.7 on GrailQA and +12.2 on KBQA-Agent, using only a 7B-parameter model.
Semantic parsing, which converts natural language questions into logic forms, plays a crucial role in reasoning within structured environments. However, existing methods encounter two significant challenges: reliance on extensive manually annotated datasets and limited generalization capability to unseen examples. To tackle these issues, we propose Targeted Synthetic Data Generation (TARGA), a practical framework that dynamically generates high-relevance synthetic data without manual annotation. Starting from the pertinent entities and relations of a given question, we probe for the potential relevant queries through layer-wise expansion and cross-layer combination. Then we generate corresponding natural language questions for these constructed queries to jointly serve as the synthetic demonstrations for in-context learning. Experiments on multiple knowledge base question answering (KBQA) datasets demonstrate that TARGA, using only a 7B-parameter model, substantially outperforms existing non-fine-tuned methods that utilize close-sourced model, achieving notable improvements in F1 scores on GrailQA(+7.7) and KBQA-Agent(+12.2). Furthermore, TARGA also exhibits superior sample efficiency, robustness, and generalization capabilities under non-I.I.D. settings.