Diagnosing Representation Dynamics in NER Model Extension
This work provides a mechanistic diagnosis for adapting NER models to new entities, addressing a common need in processing noisy spoken data, but it is incremental as it builds on existing BERT-based methods.
The study tackled the problem of extending Named Entity Recognition (NER) models to new PII entities in noisy spoken-language data, finding that joint fine-tuning on standard and new entities causes minimal degradation for original classes, with LOC uniquely vulnerable due to representation overlap and a 'reverse O-tag drift' that blocks learning until the 'O' tag classifier is unfrozen.
Extending Named Entity Recognition (NER) models to new PII entities in noisy spoken-language data is a common need. We find that jointly fine-tuning a BERT model on standard semantic entities (PER, LOC, ORG) and new pattern-based PII (EMAIL, PHONE) results in minimal degradation for original classes. We investigate this "peaceful coexistence," hypothesizing that the model uses independent semantic vs. morphological feature mechanisms. Using an incremental learning setup as a diagnostic tool, we measure semantic drift and find two key insights. First, the LOC (location) entity is uniquely vulnerable due to a representation overlap with new PII, as it shares pattern-like features (e.g., postal codes). Second, we identify a "reverse O-tag representation drift." The model, initially trained to map PII patterns to 'O', blocks new learning. This is resolved only by unfreezing the 'O' tag's classifier, allowing the background class to adapt and "release" these patterns. This work provides a mechanistic diagnosis of NER model adaptation, highlighting feature independence, representation overlap, and 'O' tag plasticity. Work done based on data gathered by https://www.papernest.com