What do writing features tell us about AI papers?
This work addresses the challenge of scalable quality assessment for AI papers, which is incremental as it builds on existing prediction tasks with interpretable features.
The authors tackled the problem of automatically assessing AI paper quality by analyzing interpretable writing features, achieving F1 scores of 60-90 for predicting conference vs. workshop publication and sometimes outperforming content-based methods like tf-idf and RoBERTa.
As the numbers of submissions to conferences grow quickly, the task of assessing the quality of academic papers automatically, convincingly, and with high accuracy attracts increasing attention. We argue that studying interpretable dimensions of these submissions could lead to scalable solutions. We extract a collection of writing features, and construct a suite of prediction tasks to assess the usefulness of these features in predicting citation counts and the publication of AI-related papers. Depending on the venues, the writing features can predict the conference vs. workshop appearance with F1 scores up to 60-90, sometimes even outperforming the content-based tf-idf features and RoBERTa. We show that the features describe writing style more than content. To further understand the results, we estimate the causal impact of the most indicative features. Our analysis on writing features provides a perspective to assessing and refining the writing of academic articles at scale.