Look Ahead Text Understanding and LLM Stitching
This addresses a problem in generative AI and social media for tasks like sentiment classification, but it is incremental as it builds on existing transformer-based LLMs.
The paper tackles the look ahead text understanding problem, exemplified by look ahead section identification (LASI), and shows that it is more challenging than classic section identification, with their approach outperforming established models, especially in noisy text conditions.
This paper proposes a look ahead text understanding problem with look ahead section identification (LASI) as an example. This problem may appear in generative AI as well as human interactions, where we want to understand the direction of a developing text or conversation. We tackle the problem using transformer-based LLMs. We show that LASI is more challenging than classic section identification (SI). We argue that both bidirectional contextual information (e.g., BERT) and unidirectional predictive ability (e.g., GPT) will benefit the task. We propose two approaches to stitch together BERT and GPT. Experiments show that our approach outperforms the established models, especially when there is noise in the text (which is often the case for developing text in generative AI). Our paper sheds light on other look ahead text understanding tasks that are important to social media, such as look ahead sentiment classification, and points out the opportunities to leverage pre-trained LLMs through stitching.