BERT & Family Eat Word Salad: Experiments with Text Understanding
This work highlights a critical vulnerability in current large language models, demonstrating their lack of true natural language understanding, which is a problem for anyone relying on these models for robust text processing.
This paper investigates the robustness of BERT-family models to incoherent inputs generated by simple heuristics. It finds that state-of-the-art models fail to recognize these inputs as ill-formed, producing high-confidence but incorrect predictions, and that models trained on randomly permuted word order perform similarly to state-of-the-art models. The authors demonstrate that explicitly training models to recognize invalid inputs can improve robustness without performance degradation.
In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.