Divide et Impera: Multi-Transformer Architectures for Complex NLP-Tasks
This addresses the challenge of controlling transformer outputs for complex tasks like bias reduction in NLP, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of fine-tuning transformer models for complex NLP tasks by dividing them into simpler subtasks, each handled by a dedicated model, which simplifies dataset compilation and increases controllability. It demonstrates this approach by reducing gender bias, showing it outperforms using a single model.
The growing capabilities of transformer models pave the way for solving increasingly complex NLP tasks. A key to supporting application-specific requirements is the ability to fine-tune. However, compiling a fine-tuning dataset tailored to complex tasks is tedious and results in large datasets, limiting the ability to control transformer output. We present an approach in which complex tasks are divided into simpler subtasks. Multiple transformer models are fine-tuned to one subtask each, and lined up to accomplish the complex task. This simplifies the compilation of fine-tuning datasets and increases overall controllability. Using the example of reducing gender bias as a complex task, we demonstrate our approach and show that it performs better than using a single model.