Structural Transfer Learning in NL-to-Bash Semantic Parsers
This provides insights into transfer learning design for semantic parsing tasks, though it appears incremental in analyzing existing methods.
The researchers investigated how structural overlap between machine translation tasks affects transfer learning for natural language to Bash semantic parsing, finding it's largely reducible to lexical alignment and showing strong overlap with SQL parsing, while increased pre-training compute doesn't always improve transfer.
Large-scale pre-training has made progress in many fields of natural language processing, though little is understood about the design of pre-training datasets. We propose a methodology for obtaining a quantitative understanding of structural overlap between machine translation tasks. We apply our methodology to the natural language to Bash semantic parsing task (NLBash) and show that it is largely reducible to lexical alignment. We also find that there is strong structural overlap between NLBash and natural language to SQL. Additionally, we perform a study varying compute expended during pre-training on the English to German machine translation task and find that more compute expended during pre-training does not always correspond semantic representations with stronger transfer to NLBash.