GROWN+UP: A Graph Representation Of a Webpage Network Utilizing Pre-training
This addresses a gap in web information retrieval for tasks like content extraction, though it is incremental in applying pre-training to webpage structures.
The paper tackles the lack of flexible pre-trained models for web information retrieval by introducing a graph neural network feature extractor that pre-trains on webpage structures, achieving state-of-the-art results on benchmarks like boilerplate removal and genre classification.
Large pre-trained neural networks are ubiquitous and critical to the success of many downstream tasks in natural language processing and computer vision. However, within the field of web information retrieval, there is a stark contrast in the lack of similarly flexible and powerful pre-trained models that can properly parse webpages. Consequently, we believe that common machine learning tasks like content extraction and information mining from webpages have low-hanging gains that yet remain untapped. We aim to close the gap by introducing an agnostic deep graph neural network feature extractor that can ingest webpage structures, pre-train self-supervised on massive unlabeled data, and fine-tune to arbitrary tasks on webpages effectually. Finally, we show that our pre-trained model achieves state-of-the-art results using multiple datasets on two very different benchmarks: webpage boilerplate removal and genre classification, thus lending support to its potential application in diverse downstream tasks.