Everything You Always Wanted to Know About TREC RTS* (*But Were Afraid to Ask)
This work addresses evaluation issues in real-time summarization for researchers, but it is incremental as it critiques existing methods without introducing new ones.
The authors analyzed the TREC Real-Time Summarization framework from 2016 and 2017, identifying limitations in its metrics and providing recommendations for fair collection reuse.
The TREC Real-Time Summarization (RTS) track provides a framework for evaluating systems monitoring the Twitter stream and pushing tweets to users according to given profiles. It includes metrics, files, settings and hypothesis provided by the organizers. In this work, we perform a thorough analysis of each component of the framework used in 2016 and 2017 and found some limitations for the Scenario A of this track. Our main findings point out the weakness of the metrics and give clear recommendations to fairly reuse the collection.