DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text
This work addresses the need for more reliable and generalizable depression detection tools in mental health applications, though it is incremental as it builds on existing BERT models with domain-specific enhancements.
The paper tackled the problem of performance inconsistencies and poor generalization in BERT-based models for detecting depression from text by introducing DECK, a set of depression-specific behavioral tests, which improved out-of-distribution F1-scores by up to 53.93%.
Models that accurately detect depression from text are important tools for addressing the post-pandemic mental health crisis. BERT-based classifiers' promising performance and the off-the-shelf availability make them great candidates for this task. However, these models are known to suffer from performance inconsistencies and poor generalization. In this paper, we introduce the DECK (DEpression ChecKlist), depression-specific model behavioural tests that allow better interpretability and improve generalizability of BERT classifiers in depression domain. We create 23 tests to evaluate BERT, RoBERTa and ALBERT depression classifiers on three datasets, two Twitter-based and one clinical interview-based. Our evaluation shows that these models: 1) are robust to certain gender-sensitive variations in text; 2) rely on the important depressive language marker of the increased use of first person pronouns; 3) fail to detect some other depression symptoms like suicidal ideation. We also demonstrate that DECK tests can be used to incorporate symptom-specific information in the training data and consistently improve generalizability of all three BERT models, with an out-of-distribution F1-score increase of up to 53.93%.