Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models
This work addresses the need for more effective fine-tuning methods in natural language processing, though it is incremental as it builds on existing masking techniques.
The paper tackles the problem of improving pre-trained language models for specific downstream tasks by proposing Typhoon, a task-specific masking strategy based on token input gradients, which achieves competitive performance with whole-word masking on the MRPC dataset.
Through exploiting a high level of parallelism enabled by graphics processing units, transformer architectures have enabled tremendous strides forward in the field of natural language processing. In a traditional masked language model, special MASK tokens are used to prompt our model to gather contextual information from surrounding words to restore originally hidden information. In this paper, we explore a task-specific masking framework for pre-trained large language models that enables superior performance on particular downstream tasks on the datasets in the GLUE benchmark. We develop our own masking algorithm, Typhoon, based on token input gradients, and compare this with other standard baselines. We find that Typhoon offers performance competitive with whole-word masking on the MRPC dataset. Our implementation can be found in a public Github Repository.