Community-Specific Slang and Entity Detection via Semantic Shift in Fine-Tuned Language Models
This work addresses the problem of identifying domain-specific vocabulary in online communities without labeled data, offering a scalable approach for social media analysis.
The paper proposes an unsupervised method to detect community-specific slang and entities by measuring semantic shift in fine-tuned language models. Using DistilRoBERTa on Reddit data, they show that words in the bottom 10% of cosine similarity between base and fine-tuned models correspond to community-specific terms.
We propose an unsupervised method of resolving slang, unique entities, and folklore from online communities by isolating words in the lexicon that have the highest magnitude of semantic shift. Semantic shift is defined as the evolution of a word's encoded representation as a result of fine-tuning a pretrained Large Language Model (LLM) on a community-specific text corpus. This value is inversely proportional to the cosine similarity between the base model's encoded representation of a word, and a fine-tuned model's encoded representation. We fine-tune the DistilRoBERTa model on text corpora collected from 3 Reddit subreddits (r/Technology, r/Gaming, r/WorldofWarcraft), model a distribution of cosine similarity over the lexicon, and show that one can successfully resolve words that have unique significance to the community by pulling data in the bottom 10-percentile. In contrast, we show that data in the top 10-percentile consist of words that carry relatively universal semantics.