Entities as Experts: Sparse Memory Access with Entity Supervision
This addresses the challenge of integrating entity knowledge into language models for tasks like question answering, offering a more efficient and effective approach compared to previous methods.
The paper tackles the problem of capturing declarative knowledge about entities in language models by introducing Entities as Experts (EAE), which accesses distinct entity memories learned directly from text, resulting in outperforming a 10x larger Transformer on TriviaQA and containing more factual knowledge than BERT on LAMA probes.
We focus on the problem of capturing declarative knowledge about entities in the learned parameters of a language model. We introduce a new model - Entities as Experts (EAE) - that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EAE's entity representations are learned directly from text. We show that EAE's learned representations capture sufficient knowledge to answer TriviaQA questions such as "Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?", outperforming an encoder-generator Transformer model with 10x the parameters. According to the LAMA knowledge probes, EAE contains more factual knowledge than a similarly sized BERT, as well as previous approaches that integrate external sources of entity knowledge. Because EAE associates parameters with specific entities, it only needs to access a fraction of its parameters at inference time, and we show that the correct identification and representation of entities is essential to EAE's performance.