A Survey of Deep Learning: From Activations to Transformers
It offers a holistic overview for researchers with basic deep learning knowledge, but it is incremental as it synthesizes existing works without introducing new methods.
This paper provides a comprehensive survey of recent deep learning advancements, covering architectures, layers, objectives, and optimization techniques, with the goal of helping researchers form new connections between diverse areas.
Deep learning has made tremendous progress in the last decade. A key success factor is the large amount of architectures, layers, objectives, and optimization techniques. They include a myriad of variants related to attention, normalization, skip connections, transformers and self-supervised learning schemes -- to name a few. We provide a comprehensive overview of the most important, recent works in these areas to those who already have a basic understanding of deep learning. We hope that a holistic and unified treatment of influential, recent works helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for many of the successful innovations over the last decade as well as works that can be seen as rising stars. We also include a discussion on recent commercially built, closed-source models such as OpenAI's GPT-4 and Google's PaLM 2.