Meta-learning for downstream aware and agnostic pretraining
This addresses a bottleneck in pretraining for NLP applications, but it is incremental as it builds on existing pretraining methods.
The paper tackles the inefficiency in neural network pretraining by proposing a meta-learning method to select informative tasks during pretraining, aiming to improve computational and memory efficiency while maintaining performance.
Neural network pretraining is gaining attention due to its outstanding performance in natural language processing applications. However, pretraining usually leverages predefined task sequences to learn general linguistic clues. The lack of mechanisms in choosing proper tasks during pretraining makes the learning and knowledge encoding inefficient. We thus propose using meta-learning to select tasks that provide the most informative learning signals in each episode of pretraining. With the proposed method, we aim to achieve better efficiency in computation and memory usage for the pretraining process and resulting networks while maintaining the performance. In this preliminary work, we discuss the algorithm of the method and its two variants, downstream-aware and downstream-agnostic pretraining. Our experiment plan is also summarized, while empirical results will be shared in our future works.