Instilling Type Knowledge in Language Models via Multi-Task QA
This addresses the challenge of noisy and incomplete type labels in language models for tasks like dialog state tracking and entity type inference, though it appears incremental as it builds on existing pre-training methods with new data.
The paper tackled the problem of learning fine-grained entity types by introducing a method that uses text-to-text pre-training on type-centric questions with knowledge base documents and graphs, resulting in state-of-the-art zero-shot performance on dialog state tracking benchmarks and accurate type inference in Wikipedia articles.
Understanding human language often necessitates understanding entities and their place in a taxonomy of knowledge -- their types. Previous methods to learn entity types rely on training classifiers on datasets with coarse, noisy, and incomplete labels. We introduce a method to instill fine-grained type knowledge in language models with text-to-text pre-training on type-centric questions leveraging knowledge base documents and knowledge graphs. We create the WikiWiki dataset: entities and passages from 10M Wikipedia articles linked to the Wikidata knowledge graph with 41K types. Models trained on WikiWiki achieve state-of-the-art performance in zero-shot dialog state tracking benchmarks, accurately infer entity types in Wikipedia articles, and can discover new types deemed useful by human judges.