Speculative decoding

EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

13 critique · 28 beaten on benchmarks

EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

10 critique · 21 beaten on benchmarks

EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

19 critique · 12 beaten on benchmarks

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

19 critique · 9 beaten on benchmarks

Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy

8 critique · 13 beaten on benchmarks

PLD+: Accelerating LLM inference by leveraging Language Model Artifacts

7 critique · 13 beaten on benchmarks

REST: Retrieval-Based Speculative Decoding

6 critique · 9 beaten on benchmarks

SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

10 critique · 4 beaten on benchmarks

Speculative Sampling for Parametric Temporal Point Processes

2 critique · 10 beaten on benchmarks

DFlash: Block Diffusion for Flash Speculative Decoding

4 critique · 6 beaten on benchmarks

LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

7 critique · 3 beaten on benchmarks

FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling

4 critique · 5 beaten on benchmarks