Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Norms
This work addresses the problem of improving annotator modeling for subjective tasks like social norm judgments, but it is incremental as it builds on prior research on using personal information.
The study investigated how different types of self-disclosure information affect the prediction of annotator labels for social norms, finding that only a small number of comments related to the original post are needed and that sampling from a larger pool without filtering yields the best performance.
Recent work has explored the use of personal information in the form of persona sentences or self-disclosures to improve modeling of individual characteristics and prediction of annotator labels for subjective tasks. The volume of personal information has historically been restricted and thus little exploration has gone into understanding what kind of information is most informative for predicting annotator labels. In this work, we categorize self-disclosures and use them to build annotator models for predicting judgments of social norms. We perform several ablations and analyses to examine the impact of the type of information on our ability to predict annotation patterns. Contrary to previous work, only a small number of comments related to the original post are needed. Lastly, a more diverse sample of annotator self-disclosures did not lead to the best performance. Sampling from a larger pool of comments without filtering still yields the best performance, suggesting that there is still much to uncover in terms of what information about an annotator is most useful for verdict prediction.