Inroads to a Structured Data Natural Language Bijection and the role of LLM annotation
This is an incremental study for researchers in natural language processing, focusing on optimization strategies for sequence-to-sequence models.
This work tackled the problem of improving performance on structured data natural language tasks using multi-task learning and LLM annotation, finding that a multi-task T5-small model increased F1 score from 0.692 to 0.771, but adding synthetic LLM-annotated data did not substantially change performance.
This work finds limited evidence supporting the theory that using multiple tasks with sequence-to-sequence transformer language models can improve performance on some metrics. In particular, the multi-task generalist t5-small outperforms the specialist t5-small with a $F_1$ of $0.771$ up from $0.692$, which may point to underlying cross-task knowledge generalization. This further suggests that even with the same network, "re-using" the same data in a different way may lead to higher performance in some metrics. However, the inverse task alone is likely only an optimization strategy, since it does not yield a significant general improvement at the model sizes explored in this work. Also, adding $\approx 4500$ LLM annotated records (interlaced with the $12800$ WebNLG training records) does not substantially change automatic metric performance compared to the same t5-small model without the synthetic data. This may be due to a learning capacity bottleneck on account of model size, and decreases observed may be due to distributional differences in the corpora. Future research using larger models or human evaluation is required to more fully explain the mechanisms contributing to performance on these tasks.