Breaking Down Word Semantics from Pre-trained Language Models through Layer-wise Dimension Selection
This work addresses the challenge of interpreting hidden semantic information in embeddings for natural language processing researchers, but it is incremental as it builds on existing disentanglement and layer-wise knowledge concepts.
The paper tackled the problem of interpreting semantic aspects in pre-trained language model embeddings by disentangling semantic sense from BERT using a binary mask on middle outputs across layers without updating parameters, resulting in improved performance in binary classification tasks for word meaning similarity.
Contextual word embeddings obtained from pre-trained language model (PLM) have proven effective for various natural language processing tasks at the word level. However, interpreting the hidden aspects within embeddings, such as syntax and semantics, remains challenging. Disentangled representation learning has emerged as a promising approach, which separates specific aspects into distinct embeddings. Furthermore, different linguistic knowledge is believed to be stored in different layers of PLM. This paper aims to disentangle semantic sense from BERT by applying a binary mask to middle outputs across the layers, without updating pre-trained parameters. The disentangled embeddings are evaluated through binary classification to determine if the target word in two different sentences has the same meaning. Experiments with cased BERT$_{\texttt{base}}$ show that leveraging layer-wise information is effective and disentangling semantic sense further improve performance.