LLM Architecture, Scaling Laws, and Economics: A Quick Summary
This provides a concise reference for researchers and practitioners working with LLMs, though it explicitly states it contains no new findings.
This paper summarizes the standard QKV self-attention architecture of Large Language Models (LLMs) and provides scaling laws for compute and memory with 2025 cost estimates for various model scales, including a discussion of DeepSeek as a potential special case.
The current standard architecture of Large Language Models (LLMs) with QKV self-attention is briefly summarized, including the architecture of a typical Transformer. Scaling laws for compute (flops) and memory (parameters plus data) are given, along with their present (2025) rough cost estimates for the parameters of present LLMs of various scales, including discussion of whether DeepSeek should be viewed as a special case. Nothing here is new, but this material seems not otherwise readily available in summary form.