Accelerating Autoregressive Generation
Autoregressive language generation is memory-bandwidth bound: each token generation requires loading billions of model parameters from HBM into SRAM to process a single forward pass.
1. The Draft and Verify Paradigm
Speculative decoding pairs a lightweight draft model (e.g., 1B-3B) with a high-capacity target model (e.g., 70B). The draft model speculatively generates $K$ candidate tokens, which the target model verifies in parallel within a single forward pass.
2. Mathematical Acceptance Criteria
Using modified rejection sampling, candidate tokens are accepted or rejected to ensure the output distribution remains mathematically identical to sampling directly from the target model.
3. Tree-Based Draft Verification (Medusa & EAGLE)
Rather than simple linear draft sequences, tree-based verification drafts multiple branching candidates simultaneously, driving average acceptance rates above 80% and yielding 2.2x to 3.0x speedups in production APIs.
