Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers
This work addresses a bottleneck in applying transformers to token-level tasks without labeled data, offering a domain-specific solution for NLP.
The paper tackled the problem of adapting sentence-level transformers for zero-shot token-level sequence labeling without supervision, achieving significant performance improvements over existing methods.
We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.