Introduction to Sequence Modeling with Transformers
It addresses the difficulty for machine learning engineers in grasping transformer details, but is incremental as it builds on existing knowledge without introducing new methods.
This paper tackles the challenge of understanding the transformer architecture by focusing on its programming components like tokenization and embedding, using simple binary sequences to incrementally build and test each component's role.
Understanding the transformer architecture and its workings is essential for machine learning (ML) engineers. However, truly understanding the transformer architecture can be demanding, even if you have a solid background in machine learning or deep learning. The main working horse is attention, which yields to the transformer encoder-decoder structure. However, putting attention aside leaves several programming components that are easy to implement but whose role for the whole is unclear. These components are 'tokenization', 'embedding' ('un-embedding'), 'masking', 'positional encoding', and 'padding'. The focus of this work is on understanding them. To keep things simple, the understanding is built incrementally by adding components one by one, and after each step investigating what is doable and what is undoable with the current model. Simple sequences of zeros (0) and ones (1) are used to study the workings of each step.