Can Language Model Understand Word Semantics as A Chatbot? An Empirical Study of Language Model Internal External Mismatch
This addresses the problem of ensuring language models' internal knowledge aligns with user interactions for developers and researchers, but it is incremental as it builds on prior work on sentence-level discrepancies.
The paper investigated the discrepancy between language models' internal knowledge and external prompts for word semantics understanding, finding mismatches across Encoder-only, Decoder-only, and Encoder-Decoder pre-trained models.
Current common interactions with language models is through full inference. This approach may not necessarily align with the model's internal knowledge. Studies show discrepancies between prompts and internal representations. Most focus on sentence understanding. We study the discrepancy of word semantics understanding in internal and external mismatch across Encoder-only, Decoder-only, and Encoder-Decoder pre-trained language models.