Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve
This work addresses robustness issues in AI models for language processing, offering insights that could enhance reliability in multilingual applications, though it appears incremental in connecting multitasking to existing robustness concepts.
The paper tackles the problem of neural network robustness to neuron failures by showing that bilingual language models retain higher performance under various neuron perturbations compared to monolingual ones, with concrete improvements demonstrated in experiments.
We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise compared to equivalent monolingual ones. We provide a theoretical justification for this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. Our analysis connects robustness to spectral properties of the learned representation and proves that multitasking leads to higher robustness for diverse task vectors. We open-source our code and models: https://github.com/giannisdaras/multilingual_robustness