The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP
This addresses the problem of inconsistent reporting in NLP research, particularly for human evaluations, by providing a standardized framework, though it is incremental as it builds on existing datasheet concepts.
The paper tackles the lack of standardized documentation for human evaluation experiments in NLP by introducing the Human Evaluation Datasheet 1.0, a template designed to record details to support comparability, meta-evaluation, and reproducibility.
This paper introduces the Human Evaluation Datasheet, a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP). Originally taking inspiration from seminal papers by Bender and Friedman (2018), Mitchell et al. (2019), and Gebru et al. (2020), the Human Evaluation Datasheet is intended to facilitate the recording of properties of human evaluations in sufficient detail, and with sufficient standardisation, to support comparability, meta-evaluation, and reproducibility tests.