Reproducibility Beyond the Research Community: Experience from NLP Beginners
This addresses the problem of making NLP research more accessible to beginners, which is incremental in highlighting practical barriers beyond technical skills.
The study investigated whether beginners can easily apply recent NLP developments by having 93 students in an introductory course reproduce results from papers, finding that technical skill had limited impact on effort, while author accessibility efforts like documentation and data access were crucial.
As NLP research attracts public attention and excitement, it becomes increasingly important for it to be accessible to a broad audience. As the research community works to democratize NLP, it remains unclear whether beginners to the field can easily apply the latest developments. To understand their needs, we conducted a study with 93 students in an introductory NLP course, where students reproduced results of recent NLP papers. Surprisingly, our results suggest that their technical skill (i.e., programming experience) has limited impact on their effort spent completing the exercise. Instead, we find accessibility efforts by research authors to be key to a successful experience, including thorough documentation and easy access to required models and datasets.