An FNet based Auto Encoder for Long Sequence News Story Generation
This is an incremental improvement for text generation efficiency in news applications.
The paper tackles news story generation from the C4 dataset using an autoencoder based on Google's FNet architecture, which trains 80% faster on GPUs and 70% faster on TPUs compared to BERT-based encoders, and analyzes outputs across epochs and seed texts.
In this paper, we design an auto encoder based off of Google's FNet Architecture in order to generate text from a subset of news stories contained in Google's C4 dataset. We discuss previous attempts and methods to generate text from autoencoders and non LLM Models. FNET poses multiple advantages to BERT based encoders in the realm of efficiency which train 80% faster on GPUs and 70% faster on TPUs. We then compare outputs of how this autencoder perfroms on different epochs. Finally, we analyze what outputs the encoder produces with different seed text.