Fighting Redundancy and Model Decay with Embeddings
This addresses model maintenance and efficiency issues for teams at Twitter handling large-scale, dynamic social media data, but is incremental as it builds on existing embedding techniques.
The paper tackles the problem of model decay and redundancy in Twitter's data pipeline due to covariate shift and stale features, by developing tools and pipelines to generate and share up-to-date embedding models across teams, resulting in maintained performance and increased productivity.
Every day, hundreds of millions of new Tweets containing over 40 languages of ever-shifting vernacular flow through Twitter. Models that attempt to extract insight from this firehose of information must face the torrential covariate shift that is endemic to the Twitter platform. While regularly-retrained algorithms can maintain performance in the face of this shift, fixed model features that fail to represent new trends and tokens can quickly become stale, resulting in performance degradation. To mitigate this problem we employ learned features, or embedding models, that can efficiently represent the most relevant aspects of a data distribution. Sharing these embedding models across teams can also reduce redundancy and multiplicatively increase cross-team modeling productivity. In this paper, we detail the commoditized tools, algorithms and pipelines that we have developed and are developing at Twitter to regularly generate high quality, up-to-date embeddings and share them broadly across the company.