Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning
This addresses interference issues in sentence representation learning for NLP applications, but it is incremental as it builds on existing multi-task instruction tuning methods.
The paper tackles interference problems across tasks and instances in multi-task instruction tuning for sentence representation learning by proposing Data-CUBE, a data curriculum method that optimizes task order and instance difficulty ordering, resulting in performance boosts on MTEB evaluation tasks.
Recently, multi-task instruction tuning has been applied into sentence representation learning, which endows the capability of generating specific representations with the guidance of task instruction, exhibiting strong generalization ability on new tasks. However, these methods mostly neglect the potential interference problems across different tasks and instances, which may affect the training and convergence of the model. To address it, we propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training, to minimize the interference risks from the two views. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk, which is exactly the traveling salesman problem, hence we utilize a simulated annealing algorithm to find its solution. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training. Experiments on MTEB sentence representation evaluation tasks show that our approach can boost the performance of state-of-the-art methods. Our code and data are publicly available at the link: \url{https://github.com/RUCAIBox/Data-CUBE}.