Next Word Suggestion using Graph Neural Network
This addresses the high computational cost of large language models for next word suggestion, but it is incremental as it builds on existing GNN and LSTM methods.
The paper tackled the problem of context embedding for next word prediction by proposing a Graph Convolutional Network combined with LSTMs, achieving fair performance on a custom Wikipedia corpus with limited resources.
Language Modeling is a prevalent task in Natural Language Processing. The currently existing most recent and most successful language models often tend to build a massive model with billions of parameters, feed in a tremendous amount of text data, and train with enormous computation resources which require millions of dollars. In this project, we aim to address an important sub-task in language modeling, i.e., context embedding. We propose an approach to exploit the Graph Convolution operation in GNNs to encode the context and use it in coalition with LSTMs to predict the next word given a local context of preceding words. We test this on the custom Wikipedia text corpus using a very limited amount of resources and show that this approach works fairly well to predict the next word.