A Web Scraping Methodology for Bypassing Twitter API Restrictions
This addresses a bottleneck for researchers in fields like NLP and sentiment analysis who rely on historical social media data, though it is an incremental improvement over existing methods.
The paper tackles the problem of collecting historical tweets beyond the three-week limit imposed by the Twitter API by developing a web scraping methodology, enabling data retrieval for any date range.
Retrieving information from social networks is the first and primordial step many data analysis fields such as Natural Language Processing, Sentiment Analysis and Machine Learning. Important data science tasks relay on historical data gathering for further predictive results. Most of the recent works use Twitter API, a public platform for collecting public streams of information, which allows querying chronological tweets for no more than three weeks old. In this paper, we present a new methodology for collecting historical tweets within any date range using web scraping techniques bypassing for Twitter API restrictions.