Learning representations of learning representations
This provides a benchmark for evaluating representation learning methods in NLP, though it's incremental as it applies existing methods to new data.
The authors created the ICLR dataset of 24,000 paper abstracts from 2017-2024 and found that simple bag-of-words representations outperform most sentence transformers in kNN classification accuracy, with top language models barely beating TF-IDF, challenging the NLP community.
The ICLR conference is unique among the top machine learning conferences in that all submitted papers are openly available. Here we present the ICLR dataset consisting of abstracts of all 24 thousand ICLR submissions from 2017-2024 with meta-data, decision scores, and custom keyword-based labels. We find that on this dataset, bag-of-words representation outperforms most dedicated sentence transformer models in terms of $k$NN classification accuracy, and the top performing language models barely outperform TF-IDF. We see this as a challenge for the NLP community. Furthermore, we use the ICLR dataset to study how the field of machine learning has changed over the last seven years, finding some improvement in gender balance. Using a 2D embedding of the abstracts' texts, we describe a shift in research topics from 2017 to 2024 and identify hedgehogs and foxes among the authors with the highest number of ICLR submissions.